Georgia Tech's Jill Watson, an AI teaching assistant since 2016, answers routine course-logistics questions online. A 2024 peer-reviewed study found 75-97% answer accuracy versus ~30% for bare ChatGPT, though deployment remains limited to two institutions.
75-97 %
Jill Watson answer accuracy (2024-2025)
~30 %
Bare ChatGPT/OpenAI Assistant answer accuracy (2024-2025)
2.7-5.7 %
Jill Watson harmful-error rate (2024-2025)
14.4-16.5 %
Bare assistant harmful-error rate (2024-2025)
66 %
A grades among students with Jill Watson access
62 %
A grades among students without Jill Watson access
3 %
C grades among students with Jill Watson access
7 %
C grades among students without Jill Watson access
40,000 postings
Historical forum postings used for training (2016)
97 %
Answer certainty reached within the term (2016)
300 students
Students enrolled in original 2016 course (2016)
600+ students
Students in current OMSCS AI course deployment
Details
Maturity
Established
Promoter
Georgia Institute of Technology (Design Intelligence Lab)
Period
2016–present
Keywords
higher education, virtual teaching assistant, online learning, natural language processing
Context
Georgia Tech's Jill Watson is an AI virtual teaching assistant developed since 2016 by Ashok Goel's Design Intelligence Lab, initially built on IBM Watson to answer routine student questions on course discussion forums. It evolved through Google's BERT model (2019) and, since 2023-2025, into a retrieval-augmented-generation architecture built on ChatGPT. It is currently deployed in Georgia Tech's OMSCS AI course and an English course at Wiregrass Georgia Technical College, funded by the US National Science Foundation's AI-ALOE institute and the Gates Foundation.
Objectives
To automate routine course-logistics question answering for instructors and teaching assistants while maintaining transparency by disclosing the AI's identity to students, and to test whether AI-assisted answering improves accuracy and safety compared with generic ChatGPT-based assistants in real classrooms.
Activities
The system was trained on roughly 40,000 historical forum postings from Georgia Tech's Knowledge-Based AI course, and its answer certainty was monitored before its identity was revealed to students in April 2016. Since 2023, the team rebuilt Jill Watson on a RAG architecture built on ChatGPT and benchmarked it against a bare OpenAI Assistant, evaluating accuracy and harmful-error rates across two university domains, and compared grade outcomes between course cohorts with and without access to the tool.
Results
A 2024 peer-reviewed AIED conference paper and a September 2025 Georgia Tech research study, corroborated across two university domains, report Jill Watson achieving 75-97% answer accuracy versus roughly 30% for a bare OpenAI Assistant, with harmful-error rates of 2.7-5.7% versus 14.4-16.5% for the baseline assistant. A classroom comparison found students with access to Jill Watson earned more A grades (66% vs 62%) and fewer C grades (3% vs 7%) than those without.
Conclusions
Jill Watson has operated continuously since 2016 with full transparency about its AI identity, but confirmed deployment remains limited to two institutions; a planned partnership with Wiley & Sons intended to scale it to more classrooms had not launched as of mid-2026.
Implementation
Indicative cost
Medium (€50k–€500k)
Time to results
Long (> 3 years)
Staffing & skills
Academic research team led by Ashok Goel's Design Intelligence Lab at the Georgia Institute of Technology, Course instructors and teaching assistants who integrate the assistant into course discussion forums
Conditions for success
Sustained multi-year research funding (US National Science Foundation's AI-ALOE institute, Gates Foundation)
Full disclosure of the AI's identity to students to maintain trust
A large corpus of historical course Q&A data to train and ground the system
Institutional willingness to migrate underlying model architectures over time (IBM Watson to BERT to a ChatGPT-based RAG system)
Common failure modes
Confirmed deployment has remained limited to only two institutions for years
Broader scale-up via the planned Wiley & Sons partnership had not launched as of mid-2026
Where it fits
Governance type
university research lab / higher-education institution
Scale
course-level, confirmed at two institutions (courses ranging from about 300 to 600+ students)
Income level
high-income (United States)
Data sources
Where this practice's information was retrieved from, and when.
IDB/Elige Educar WhatsApp AI chatbot for career guidance to Chilean secondary students. Pre-registered RCT (43,000+ students, 2024–25): AI raised initiation …
