JudgeGPT — a Nationwide Randomized Trial of Generative AI for Pakistan's Trial-Court Judges

Pakistan's trial courts faced 2.26 million pending cases at the end of 2024, with fewer than two judges per 100,000 residents compared with 22 in the EU, prompting researchers from ETH Zurich, Imperial College London and the New Economic School to build JudgeGPT, a GPT-4-based retrieval-augmented assistant drawing on 129,235 documents (128,292 court rulings and 943 Pakistani laws).

The team designed a randomized controlled trial covering 1,559 judges across 118 courts - roughly half of Pakistan's trial-court bench - randomised into a JudgeGPT-plus-targeted-training arm, a JudgeGPT-plus-general-seminar arm, and a seminar-only control arm.

Trained judges used the tool far more intensively than the comparison arms, logging around 60 logins and more than 200 prompts each, versus about 20 logins and under 50 prompts in the other arms.

Over the 40 weeks following the intervention, districts with more trained judges resolved about 1,848 additional cases per year - a 6.3% increase - with the largest effect, about 616 extra cases, in the lowest-performing quartile of districts; blind quality assessments rated trained judges' rulings as better in 59% of pairwise comparisons, up from a 42% baseline, and the study found no evidence that AI assistance increased gender or religious bias in judicial language.

The research team's own cost-benefit estimate puts returns at roughly $10-38.50 per dollar invested, though that figure comes from the study's authors rather than an independent government audit, and the programme remains a large-scale research trial rather than a codified permanent institution, with Pakistan's Supreme Court signalling openness to broader judicial AI use while noting the absence of a comprehensive legal framework governing it.