Course-Integrated AI Tutor — Randomised University Trial Finds Lower Final Grades (Maryland EDSI Research Team)

In fall 2025 a research team at the University of Maryland's Education Policy and Data Science centre (EDSI) ran a randomised controlled trial of a course-integrated generative-AI tutor, the 'Virtual Study Assistant', embedded in a large U.S. public university's learning platform. The tool was powered by GPT-4o and used retrieval-augmented generation over course materials selected by instructors. The working paper describes the site only as 'a large U.S. public university'; the research team is based at the University of Maryland, College Park.
Access was randomised across sections: 2,379 undergraduates and 30 instructors from several disciplines took part. Instructors could choose a direct-instruction mode or a guided-tutoring mode, and most kept the default direct-instruction mode.
Results: being offered the tutor reduced final course grades by 0.37 standard deviations in the within-course comparison (about 4 percentage points; the paper reports a range of 0.27-0.37 SD) and reduced recorded learning-platform participation by 0.90 SD. Only about 15% of students offered the tutor used it; about 74% of requests sought information, explanations or solutions, about 11% concerned practice or test preparation, and under 1% asked for feedback on the student's own attempt. Estimated losses were larger for first-generation students, though the authors caution against over-reading subgroup results.
Interpretation and limits: the authors suggest the tool may displace engagement with existing resources (such as office hours and course materials) rather than add to it. They stress that the trial measures the effect of offering access, not of actual use; that course grades rather than an independent test were the outcome; that students already had other generative-AI tools; and that the results do not show purpose-built, pedagogically designed tutors are harmful. The study is an EdWorkingPapers preprint (October 2026) and has not been peer reviewed.
Why it matters: it is one of the few large randomised trials of an institution-provided AI tutor, and it shows that making a tool available is not the same as improving learning; design (tutoring vs direct-answer mode), instructor integration and uptake need to be evaluated before scaling.

Read the full analysis: https://edsi.umd.edu/publications/effects-course-integrated-ai-tutoring-student-performance-and-engagement-randomized