The GPT Surprise — Stanford's Code in Place Randomised Trial of GPT-4 Chat for 5,831 Learners in 146 Countries

Code in Place is Stanford's free online introductory Python course, with small-group sections led by volunteers. In a Learning @ Scale 2025 paper, Allen Nie, Chris Piech and colleagues report a randomised trial in a cohort of 8,762 enrolled students in 146 countries. The experiment pool was the 5,831 students still active after week 1, assigned 60% to treatment (n=3,581) and 40% to control (n=2,250). Treated students were given a course-specific interface to GPT-4.
On average, advertising and offering the tool significantly reduced exam participation (the access estimate reported is about −4.3 percentage points) and also lowered homework participation and section attendance. Students who actually adopted the tool, however, showed positive benefits on exam performance. For students from low-human-development-index countries, access increased exam participation on average, pointing to unequal effects by context.
The authors stress that longer-term effects on student success remain unclear, and that adopter comparisons are not purely randomised. The case is valuable because it combines a very large randomised sample with a clear tension: LLM chat can help the students who use it while reducing overall engagement.

Read the full analysis: https://web.stanford.edu/~cpiech/bio/papers/gptsurprise.pdf