Grammarly-Based Automated Writing Evaluation Lifts EFL Writing Scores in a 190-Student RCT in Tangshan, China

Researchers at Tangshan Normal University's School of Foreign Languages ran a 12-week randomised controlled trial with 190 intermediate-level Chinese English-as-a-Foreign-Language (EFL) students, comparing a commercial AI-based automated writing evaluation (AWE) tool against standard instruction.

To test whether Grammarly-based automated writing evaluation improves EFL writing quality - task achievement, coherence and cohesion, lexical resource and grammatical accuracy - compared with standard instruction alone.

190 students were randomly assigned to an experimental group (95 students, using Grammarly alongside their regular coursework) or a control group (95 students, standard instruction only) for a 12-week extracurricular, not-for-credit writing programme. Writing quality was scored by independent raters (Cohen's kappa = 0.82) against four IELTS-aligned dimensions before and after the intervention.

The Grammarly group showed statistically significant gains over the control group on every dimension: task achievement (B=0.38, p=.044), coherence and cohesion (B=0.46, p=.036), lexical resource (B=0.55, p=.009) and grammatical accuracy (B=0.74, p=.003), with explained variance (R-squared) ranging from 0.28 to 0.46. The authors report the largest improvements among learners who started at lower proficiency levels, and writing self-efficacy was also a significant predictor of gains.

The trial provides strong, single-institution RCT evidence that a commercial AWE tool can improve EFL writing quality, especially for weaker writers, but it is limited to one institution and an extracurricular (not-for-credit) course, evaluates a third-party commercial tool rather than an institution-built system, and does not report on data-protection safeguards or any teacher-training component.