DfT's Consultation Analysis Tool (CAT) — LLM Thematic Analysis of 200,000 Responses, Benchmarked Against Human Coders

The Department for Transport runs about 55 consultations a year, some with over 100,000 free-text responses. With the Alan Turing Institute it built the Consultation Analysis Tool (CAT): an ensemble of large language models generates themes (including rare "golden insights"), human analysts review and validate them, and the models then map every response to the validated themes.
By the time of the December 2025 v1.0 evaluation the CAT had been piloted on live consultations covering 200,000 responses (over 8 million words). The published evaluation used a blind design (CAT against independent human-coded datasets) and a non-blind live-pilot design (CAT output against the human-adjusted result).
Results: without human oversight the CAT recovered about 75% of human-generated themes in blind tests, and 90% recall in live pilots after human review. Theme mapping scored F1 0.75 (blind) and 0.93 (live), with over 92% raw agreement with human experts and no evidence of systematic accuracy differences across demographic groups. DfT estimates it saves 50–70% of the cost and time of a medium-sized consultation; across four completed consultations about £0.5m and roughly 15,000 staff hours, and £1.5–4m a year if applied to the full portfolio (explicitly opportunity-cost savings, not cash).
Caveats stated by the authors: accuracy varies by dataset, a perfect score is effectively unachievable because human reference coding is itself subjective, prevalence figures must be read as estimates with confidence intervals, and human theme review (1–5 hours per question) remains the core safeguard. The savings are DfT's own estimates, and the evaluation was run by the developing team rather than an independent body.

Read the full analysis: https://www.highwaysmagazine.co.uk/news/dft/dft-eyes-annual-savings-ai-consultation-analysis-tool