Charlie — Canada Revenue Agency's Tax Chatbot, Audited at 33% Accuracy by the Auditor General

The Canada Revenue Agency (CRA) runs a virtual chatbot, Charlie, which gives automated answers to frequently asked tax questions. Reporting on parliamentary documents indicates the CRA has spent more than $18 million on Charlie since the 2018-19 fiscal year. Charlie is described as a non-generative, rule-based chatbot.
In a performance audit tabled on 21 October 2025, the Auditor General of Canada, Karen Hogan, tested CRA contact channels over four months (February to May 2025). Agents answered just 17% of individual tax questions accurately and 54% of business tax or benefits questions accurately (with completeness just over 30% for the latter). Charlie, tested with six tax questions, was right 33% of the time; the Auditor General noted that was slightly better than reaching an agent on personal tax, but that answers were brief with limited context, and that public web-based conversational AI tools answered 5 of 6 correctly.
The audit also found incentives that undermine accuracy: less than 9% of an agent's performance score related to the accuracy and completeness of information given to callers, versus 45% tied to schedule adherence and handling time.
The CRA responded by extending about 850 agent contracts, adding training, and expanding the questions its chatbot (and generative AI) can answer. Evidence limits: the test sample was small (six chatbot questions), the CRA contests aspects of the findings, and no post-audit accuracy figures are yet published. The value of the case is that an independent audit measured a citizen-facing bot against humans and against open web tools, and made the result public.

Read the full analysis: https://www.cp24.com/news/canada/2025/10/22/get-bad-advice-from-the-canada-revenue-agency-youre-out-of-luck-taxpayer/