Accuracy of AI Speech-to-Text Captioning for Deaf and Hard-of-Hearing Students — York University's Eight-Tool Benchmark

A study by Pam Millett (Faculty of Education, York University, Toronto) tested eight speech-to-text apps and platforms as real-time captioning for deaf and hard-of-hearing (DHH) postsecondary students, under deliberately controlled acoustic conditions: Interact Streamer, Ava, Otter, Google Slides, Microsoft Stream, Microsoft Translator, Camtasia Studio and YouTube.
Accuracy was scored two ways: Total Accuracy (share of words correct) and Meaning Accuracy (errors that change the message). In the live-lecture condition, Otter scored 99.7%, Google Slides 97.7%, Interact Streamer 93.6% and Ava 87.2%, against 99.8% for a human CART captionist. For pre-recorded video, YouTube scored 98% and Microsoft Stream 98.7% (human post-production captioning: 99.9%). In a three-student live conversation, Microsoft Translator scored 94.1% and Ava 92.7%.
Meaning Accuracy was almost identical to Total Accuracy, so the errors were not trivial: some produced unrecognisable text (e.g. 'babies get mauled up' for 'babies get mobile'). The author notes that punctuation errors can mislead and that no research defines when accuracy is 'good enough'.
Cautions: audio was optimised (minimal noise, distance and reverberation), the lecture was short and read from a script, and the author predicts significantly lower accuracy in real classrooms. This is an accuracy benchmark, not a learning-outcome trial. Independent work (Kafle & Huenerfauth, ASSETS 2017, 30 DHH participants) shows word-error-rate alone poorly predicts caption usability for DHH readers, supporting the study's use of a meaning-based score. The practice is an evidence-informed procurement benchmark: test captioning tools for meaning accuracy before adopting them as an accommodation, and keep human CART where accuracy matters most.

Read the full analysis: https://www.edaud.org/assets/docs/1-article-21.pdf