Grammarly-Based Automated Writing Evaluation Lifts EFL Writing Scores in a 190-Student RCT in Tangshan, China

Researchers at Tangshan Normal University's School of Foreign Languages randomly assigned 190 intermediate-level Chinese English-as-a-Foreign-Language (EFL) students (95 experimental, 95 control) to a 12-week extracurricular writing programme. The experimental group used Grammarly, a commercial AI-based automated writing evaluation (AWE) tool, alongside their regular coursework; the control group received standard instruction only.
Writing quality was scored by independent raters (Cohen's kappa = 0.82) against four IELTS-aligned dimensions before and after the intervention. The Grammarly group showed statistically significant gains over the control group on every dimension: task achievement (B=0.38, p=.044), coherence and cohesion (B=0.46, p=.036), lexical resource (B=0.55, p=.009) and grammatical accuracy (B=0.74, p=.003), with explained variance (R²) ranging from 0.28 to 0.46. Writing self-efficacy was also identified as a significant predictor of gains, and the authors report the largest improvements among learners who started at lower proficiency levels.
The trial is limited to a single institution and an extracurricular (not for-credit) writing course, and it evaluates a commercial third-party tool rather than an institution-built system — the study does not report on data-protection safeguards for the AI platform or on any teacher-training component, since the intervention was entirely student-facing.

Read the full analysis: https://public-pages-files-2025.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2023.1249991/pdf