A team from Universidad de La Sabana taught a NAO humanoid robot to teach and check basic Colombian Sign Language vocabulary, hitting 91.6-93.8% sign-recognition accuracy in lab tests — a promising prototype the authors admit has not yet been tried with deaf students.
91.6 %
Sign-recognition F1 score (webcam) (lab testing)
93.8 %
Sign-recognition F1 score (NAO camera) (lab testing)
85 milliseconds
Response time (webcam) (lab testing)
2.3 seconds
Response time (NAO camera) (lab testing)
20 signers
Signers used to train the model (n/a)
Details
Maturity
Pilot
Promoter
Universidad de La Sabana (CAPSAB Research Group, Faculty of Engineering)
A team from Universidad de La Sabana's CAPSAB control-and-automation research group in Chia, Colombia, programmed a NAO H25 humanoid robot to teach and check basic Colombian Sign Language (CSL) vocabulary, starting with 11 colour signs, in response to a shortage of qualified CSL teachers - fewer than 0.2% of Colombian educational institutions offer bilingual instruction for deaf pupils.
Objectives
Build and technically validate an AI-driven sign-language tutoring system before any classroom or learner testing.
Activities
The system extracts hand and torso coordinates from video using MediaPipe and classifies signs with a recurrent (LSTM) neural network trained on 20 signers, tested via either a webcam or the robot's own camera communicating over TCP/IP.
Results
In technical testing, the system reached an F1 score of 91.6% (webcam) and 93.8% (NAO camera), with response times of 85 milliseconds and 2.3 seconds respectively; the sign for 'purple' was recognised with 100% accuracy.
Conclusions
This is early-stage, technology-readiness evidence rather than classroom evidence: the paper reports sign-recognition performance only, and the authors state plainly that the system has not yet been tested with deaf learners - an educational-effectiveness trial is still needed before any claim of learning impact would be warranted.
Implementation
Indicative cost
Low (< €50k) — No cost figure stated; low band is a conservative inferred estimate for a university research prototype built on an existing NAO robot and open-source tools.
Time to results
Short (< 1 year) — Research and testing phase reported for 2023-2024.
Staffing & skills
Universidad de La Sabana CAPSAB research group, Faculty of Engineering
Conditions for success
Built on established open tools (MediaPipe for pose extraction) and a purpose-trained LSTM network using data from 20 real signers
Common failure modes
System has not been tested with deaf learners at all, only technically validated
Vocabulary limited to 11 colour signs
Added latency over the wireless NAO-camera connection (2.3s vs 85ms for webcam)
Where it fits
Governance type
university research group
Scale
lab prototype, single research team
Income level
upper-middle-income
Replication kit
Reusable artefacts from this practice — as published by their sources.
WeSign gives deaf and hard-of-hearing Hanoi primary pupils real-time, AI-scored feedback on fingerspelling and sign formation via webcam. A 30-participant …