CDA-AMC's Evaluation of AI Search Tools for Health-Technology-Assessment Evidence Synthesis

Canada's Drug Agency (CDA-AMC, formerly CADTH) tasked its Research Information Services team with evaluating three AI-assisted literature-search and evidence-synthesis tools — Lens.org, SpiderCite and Microsoft Copilot (GPT-4) — against standard systematic-review search methods, for the Health Technology Assessment (HTA) work underpinning Canada's drug and technology reimbursement recommendations.

To assess whether AI-assisted search tools can match manual systematic-review search performance for HTA evidence synthesis.

A retrospective comparison across 7 completed agency HTA projects (August-November 2024) benchmarked each AI tool's search recall and search time against manual reference search. Findings were published as a peer-reviewed comparative study and incorporated into an official CDA-AMC position statement on AI use in HTA.

Manual reference search achieved a recall/sensitivity of 0.986, versus 0.676 for Lens.org, roughly 0.23-0.26 for SpiderCite, and 0.244 for Copilot. Search time fell from an average of 2.88 hours for manual search to as little as 0.96 hours for Copilot.

The agency's conclusion is cautionary rather than promotional: each tool was judged 'fit for purpose' only in narrow supporting roles — for example, Copilot for search-strategy brainstorming — and explicitly not as a replacement for standard systematic search on complex HTA topics.