Middlebury College Proctored Experiment — Allowing Generative AI Raised Test Scores 0.27 SD, but Only Augmentation Use Built Lasting Skills

Economists Zara Contractor and Germán Reyes (Middlebury College) ran a proctored randomised experiment with 211 undergraduates over two sessions about a week apart (IZA Discussion Paper 18792, 2026). In session one, students learned about an unfamiliar technical topic and wrote an analytical essay with or without access to off-the-shelf generative AI (ChatGPT, with conversation logs collected); in session two, all students completed unaided assessments.
About 68% of students in the AI-allowed condition used AI and 88% of users found it helpful for the task. AI access raised immediate knowledge-test scores by 0.27 standard deviations, and the gain persisted one week later (about 5.1 percentage points on the retention test). Essay quality changed little while AI was available but improved in style and relevance a week later, when students wrote unaided.
Usage mattered: delayed gains were larger for 'augmentation' users, who used AI to explain concepts (the most common use, by 57% of treated students), whereas 'automation' users' short-run quality gains vanished once AI was removed. The authors identify two mechanisms: students shifted time from drafting text towards reading and searching for information, and reported higher learning enjoyment.
Limits: a single elite liberal-arts college, one short task, a one-week horizon, and a lab-style setting rather than a full course; the paper is a discussion paper, not yet peer-reviewed. The study was approved by the Middlebury IRB. The practical lesson for AI literacy is to teach students to use AI to understand, not to outsource production.

Read the full analysis: https://www.iza.org/publications/dp/18792/experimental-evidence-on-the-learning-impact-of-generative-ai