evidoria

← Back to browse

Good practice Imported

GenAI 24x7 Tutor — University of Wollongong's Human-Simulated Benchmark of ChatGPT, Wolfram GPT and Tutor Me GPT for Engineering and Maths

Australia · Wollongong · See the Australia profile

Evidence: Descriptive / self-reported Top 95% 20/100 · Ask Evidence Copilot about this practice

University of Wollongong researchers had assistants role-play average and struggling students across 35 engineering/maths topics to benchmark ChatGPT-4, ChatGPT-4o, Wolfram GPT and Tutor Me GPT as tutors, finding frequent errors and near-zero accuracy on some mechanical topics.

~280 35 topics x 4 tools x 2 student profiles
Tutoring sessions evaluated
3 of 8
Water-pipe pressure-drop question correct attempts
2.97 out of 4
Lowest model tutor-experience average
GenAI 24x7 Tutor — University of Wollongong's Human-Simulated Benchmark of ChatGPT, Wolfram GPT and Tutor Me GPT for Engineering and Maths

Details

Promoter
University of Wollongong, Australasian Artificial Intelligence in Engineering Education Centre (AAIEEC)
Period
Second half of 2024 (data collection); published 21 January 2026
Keywords
engineering education, mathematics education, GenAI tutoring, accuracy benchmarking, higher education

Context

Researchers at the University of Wollongong's Faculty of Engineering and Information Sciences, working through the Australasian Artificial Intelligence in Engineering Education Centre (AAIEEC), tested whether ChatGPT-4, ChatGPT-4o, Wolfram GPT and Tutor Me GPT (Khanmigo Lite) could function as accurate, pedagogically sound 24/7 tutors for engineering and mathematics.

Objectives

Because the university's human research ethics committee would not approve trials with real students without prior evidence on risk, the team used three experienced research assistants to role-play an average student and a consistently struggling student across 7 subjects and 35 topics in electrical engineering, mechanical engineering and mathematics.

Activities

Each of the roughly 280 sessions (35 topics x 4 tools x 2 student profiles) lasted at least 20 minutes and was scored 0-4 on accuracy and on six pedagogical-experience dimensions, using a rubric adapted from Merrill et al.'s human-vs-computer tutoring framework.

Results

Accuracy was strong in electrical engineering and mathematics (generally 3-4) but weak in mechanical engineering, where average scores fell below 3 and, in one thermodynamics topic, dropped to 0-1.5; a water-pipe pressure-drop question was answered correctly in only 3 of 8 attempts. A Friedman test found no statistically significant difference in accuracy between the four tools. Tutor-experience scores were more consistently positive (lowest model average 2.97/4), with ChatGPT-4 rated the strongest overall tutor.

Conclusions

The authors are explicit that this was a simulation, not a trial with real students, and that it 'offers no direct evidence of GenAI's actual effectiveness in supporting learning.' They recommend against broader classroom implementation until accuracy improves and a supervised student trial is run.

Implementation
Indicative cost
Low (< €50k) — Research study using existing commercial AI tools; no infrastructure build cost.
Time to results
Short (< 1 year) — Data collection second half of 2024; published 21 January 2026.
Staffing & skills
University of Wollongong, Faculty of Engineering and Information Sciences, Australasian Artificial Intelligence in Engineering Education Centre (AAIEEC), three research assistants (two PhD candidates, one doctorate holder)

Conditions for success

  • standardised rubric (adapted from Merrill et al.) scoring both accuracy and pedagogical-experience dimensions
  • role-play simulation used specifically because ethics approval for a real-student trial required prior risk evidence

Common failure modes

  • accuracy dropped to near zero on some mechanical-engineering/thermodynamics topics
  • study authors explicitly recommend against classroom use until accuracy improves and a real-student trial is run

Where it fits

Governance type
university research centre
Scale
single research study
Income level
high income (Australia)

Commonly funded by

National / regional programmes

Indicative funding routes for practices of this type — always check each programme's current calls and eligibility rules.

Replication kit

Reusable artefacts from this practice — as published by their sources.

Do you run this practice? Claim it — verified implementers get a public contact pathway and can propose corrections.

Data sources

Where this practice's information was retrieved from, and when.

Attachments

Similar practices you may find useful