UTMACH built a generative-AI vocational-guidance chatbot for Ecuadorian students; in a quasi-experimental trial with 351 students, its picks matched the government's SENESCYT vocational test 80.1% of the time, and 76% rated the experience positively.
351
Students in evaluation
80.1%
Concordance with official SENESCYT test
76%
Students rating interaction/recommendations positively
Details
Maturity
Pilot
Promoter
Universidad Técnica de Machala (UTMACH)
Period
2025
Keywords
secondary education, vocational guidance, generative AI
Context
Ecuador's secondary schools face a shortage of trained vocational-guidance counsellors and limited access to guidance technology. Researchers at Universidad Tecnica de Machala (UTMACH) built a chatbot powered by the DeepSeek language model, using prompt-engineering techniques to interpret students' open-ended answers and match them to a suggested academic path drawn from UTMACH's own programme offerings.
Results
The team evaluated it with a quasi-experimental design combining a treatment group and a control group, involving 351 third-year secondary students. Students completed the government's official SENESCYT vocational aptitude test, then interacted with the AI chatbot, then completed a Likert-scale perception survey. The chatbot's suggestions matched the SENESCYT test results in 80.1% of cases, and 76% of students rated the conversational interaction, question clarity and relevance of recommendations positively.
Conclusions
The 80.1% concordance is a meaningful validation against an external government benchmark, but it also means roughly one in five recommendations diverged from the official test, and the chatbot's suggested pathways were drawn specifically from UTMACH's own course catalogue -- a caveat prospective adopters elsewhere should weigh before reusing the tool unmodified.
Implementation
Indicative cost
Low (< €50k)
Time to results
Short (< 1 year)
Staffing & skills
Universidad Tecnica de Machala (UTMACH) researchers
Conditions for success
validation against an external government benchmark test (SENESCYT) rather than self-report alone
Common failure modes
suggested pathways drawn only from UTMACH's own course catalogue rather than the full national landscape; ~20% of recommendations diverged from the official test