evidoria

← Back to browse

Good practice Imported

URAG — Ho Chi Minh City University of Technology's Hybrid RAG Admission Chatbot Outperforms GPT-4o on Real Applicant Questions

Vietnam · Ho Chi Minh City · See the Vietnam profile · See the Ho Chi Minh City profile

Evidence: Quasi-experimental Top 81% 33/100 · Ask Evidence Copilot about this practice

HCMUT built URAG, a two-tier retrieval-and-rules chatbot for admissions questions, that scored 62.8% accuracy on 500 real applicant queries versus 54.4% for GPT-4o, 50.2% for Claude 3.5 Sonnet and 44.0% for Gemini 1.5 Pro.

62.8%
URAG accuracy on 500 real applicant questions
54.4%
GPT-4o accuracy (same 500 questions)
50.2%
Claude 3.5 Sonnet accuracy (same 500 questions)
44.0%
Gemini 1.5 Pro accuracy (same 500 questions)
218 of 268
Correct FAQ-style answers
URAG — Ho Chi Minh City University of Technology's Hybrid RAG Admission Chatbot Outperforms GPT-4o on Real Applicant Questions

Details

Promoter
Ho Chi Minh City University of Technology (HCMUT) — URA Research Group
Period
2024-2025
Keywords
higher education, university admissions, generative AI, retrieval-augmented generation

Context

Ho Chi Minh City University of Technology (HCMUT) built URAG, a two-tier hybrid chatbot for university admissions, to address the risk of large language models hallucinating on sensitive, time-bound questions such as deadlines and requirements.

Objectives

URAG first checks a curated FAQ bank built from official admissions documents, and falls back to retrieval-augmented generation over semantically chunked source documents only when no FAQ match is found.

Activities

The chatbot was deployed live at ura.hcmut.edu.vn for a four-month admissions cycle, with usage peaking as Vietnamese high-school students approached exam and enrolment deadlines, and was recognised at Ho Chi Minh City's WHISE 2024.

Results

Benchmarked against 500 real admission questions from high-school applicants, URAG answered 62.8% correctly, ahead of GPT-4o (54.4%), Claude 3.5 Sonnet (50.2%) and Gemini 1.5 Pro (44.0%); its advantage was concentrated in factual, FAQ-style questions (218 of 268 correct), while it underperformed the general-purpose models on multi-step reasoning.

Conclusions

With an accuracy ceiling around 63%, roughly one in three answers was still wrong, so the tool is best read as a triage layer that reduces routine load rather than a full replacement for human admissions staff.

Implementation
Indicative cost
Low (< €50k)
Time to results
Short (< 1 year)
Staffing & skills
HCMUT URA Research Group engineering team, University admissions office (FAQ content source and coordination)

Conditions for success

  • Curated FAQ bank kept current from official admissions documents (URAG-F)
  • Retrieval-augmented fallback restricted to semantically chunked, vetted source documents (URAG-D)
  • Framing the tool as a triage layer for routine questions rather than a replacement for admissions staff

Commonly funded by

National / regional programmes Own resources / municipal budget

Indicative funding routes for practices of this type — always check each programme's current calls and eligibility rules.

Do you run this practice? Claim it — verified implementers get a public contact pathway and can propose corrections.

Data sources

Where this practice's information was retrieved from, and when.

Attachments

Similar practices you may find useful