Vanderbilt University's Evidence-Based Rejection of Turnitin's AI-Writing Detector

In April 2023 Vanderbilt University's Brightspace/Center for Teaching enabled Turnitin's newly launched AI-writing detection feature for faculty use campus-wide, even though Turnitin publicly claimed only a roughly 1% false-positive rate. With about 75,000 papers submitted through Turnitin in 2022, Vanderbilt staff calculated that even this low claimed rate implied around 750 papers a year could be wrongly flagged as AI-generated. That same summer, a Stanford University research team published a peer-reviewed study in the Cell Press journal Patterns testing whether commercial AI-text detectors were biased against non-native English writers.

Vanderbilt's stated aim was to assess whether Turnitin's AI-writing detector was accurate and fair enough to underpin high-stakes academic-integrity decisions, weighing the tool's claimed false-positive rate against the number of papers it screened and the transparency of its methodology.

Over several months, Vanderbilt's Brightspace/Center for Teaching ran internal testing of the AI-writing detector and consulted with other universities and with AI vendors, including Turnitin, before deciding on its future use. In parallel, Stanford researchers (Liang, Yuksekgonul, Mao, Wu and Zou) ran 91 genuine TOEFL essays written by non-native English speakers, and a comparison set of US eighth-grade essays by native speakers, through seven widely used commercial GPT detectors, and additionally tested how ChatGPT-based rewording of the essays changed detector outcomes.

Vanderbilt disabled Turnitin's AI-writing detector campus-wide on 16 August 2023, citing both the false-positive risk and Turnitin's refusal to disclose the detailed methodology behind its AI-probability scores. The Stanford study found the seven detectors misclassified an average of 61.3% of non-native TOEFL essays as AI-written, with one detector flagging 97.8% and all seven unanimously misjudging 19.8% of the set, while native-English eighth-grade essays were correctly classified; using ChatGPT to enhance the non-native essays' word choice reduced the false-positive rate to 11.6%, whereas simplifying the native essays' vocabulary increased their misclassification.

Together the two cases show a rare feedback loop between an institution's internal cost-benefit reasoning and independent academic research reaching the same conclusion: commercial AI-text detectors were not yet reliable or fair enough for high-stakes academic-integrity decisions, particularly for international and non-native-English-speaking students, prompting Vanderbilt and other US universities to disable AI detection rather than risk false accusations.