ONS's AI Push in Official Statistics — From SIC/SOC Autocoding Research to ScannerAI's Receipt-Reading Prototype

The UK's Office for National Statistics assigns free-text survey responses to Standard Industrial Classification (SIC) and Standard Occupational Classification (SOC) codes across major surveys, including the 2021 Census (500,000 records), the Labour Force Survey (300,000 records), the COVID-19 Infection Survey (70,000 records) and the Annual Survey of Hours and Earnings. Its existing in-house rule-based tool, built on knowledge-base fuzzy matching, achieves match rates of roughly 50–62% depending on the survey.

The Data Science Campus prototyped an alternative using logistic regression over uni-gram/bi-gram/tri-gram text features (a deliberately 'basic' approach, with the team noting that more advanced methods such as BERT could do better). Published results show the ML approach reaching 73% match / 84% overall accuracy on Census SOC data, 70% match / 85% accuracy on the Labour Force Survey, and 66–67% match / 69–76% accuracy on two smaller surveys — a marked improvement on the rule-based baseline. Performance dropped sharply on a fifth survey (DLHE, to a 60% match rate versus 84% on Census), which the team attributes to the model not transferring cleanly to a different respondent population, and it flags that gold-coded training data does not yet exist for SIC in sufficient volume, and that the model will learn and repeat any systematic errors already present in past manual coding. The code is open-sourced (ACE2, on GitHub) and the project remains in a research phase, with ONS stating a further comparison study is needed before any deployment decision.

In parallel, the Campus is building ScannerAI, a pipeline to automatically extract itemised data from photographs of receipts submitted for the Living Costs and Food Survey, which collected around 60,000 receipts from close to 4,000 households in a recent year. The pipeline chains image pre-processing, OCR (Tesseract and Google Vision, with the Gemini Pro multimodal model currently preferred), text restructuring via Gemini Pro, and a choice of Random Forest or a generative classification microservice to assign spending categories. ONS's own documentation describes the tool as an early-phase prototype still undergoing accuracy testing against human-coded receipts, not yet in production. Independent reporting in August 2026 situated both projects within a wider AI push reaching the large majority of ONS's roughly 6,000 staff, undertaken while the agency was absorbing a real-terms budget cut — a reminder that these efficiency projects are, in part, a response to resource pressure rather than a purely technology-led initiative.

Read the full analysis: https://datasciencecampus.ons.gov.uk/projects/automated-coding-of-standard-industrial-and-occupational-classifications-sic-soc/