Personalized Problem Sequencing Lifts Python Exam Scores — A 10-School RCT in Taipei

Researchers from the Wharton School (University of Pennsylvania), Carnegie Mellon University and National Taiwan University ran a randomized controlled trial embedded in an after-school online Python certification course across 10 high schools in Taipei, Taiwan, involving roughly 800 students over five months. Every student had identical access to the same generative-AI chatbot and course materials; the only experimental difference was how practice problems were sequenced. The control group worked through a fixed, easy-to-hard problem order, while the treatment group received a personalized sequence in which an algorithm adjusted problem difficulty based on each student's performance and interactions with the AI tutor.
Students in the personalized-sequencing group scored significantly higher on final exams, an improvement the researchers estimate at 0.15 standard deviations — described, with an explicit caveat that it is 'not a perfect estimate,' as roughly equivalent to six to nine months of additional learning. Treatment-group students also engaged more deeply, spending about an hour per module compared to roughly 30 minutes or less for the control group. Gains were larger among students new to Python and those from less academically elite high schools than among students with prior programming experience or from top schools.
The study's design meant no student was denied access to the AI tutor itself and no additional teacher time or workload was required — only the problem sequencing changed. The authors are explicit that the paper, 'Effective Personalized AI Tutors via LLM-Guided Reinforcement Learning,' was posted as a draft in March 2026 and has not yet completed peer review, and that the sample of enrolled students skewed toward highly motivated teenagers with educated parents, which may limit how the results generalize to less self-selected populations.

Read the full analysis: https://hechingerreport.org/proof-points-ai-tutor-python/