evidoria

← Back to browse

Good practice Imported

Smart Check — Amsterdam Built a 'Fair' Welfare-Fraud AI, Then Shelved It When Bias Reappeared in Live Testing

Netherlands · Amsterdam · See the Netherlands profile · See the Amsterdam profile

Evidence: Quasi-experimental Top 24% 73/100 · Ask Evidence Copilot about this practice

Amsterdam spent about five years and roughly €535,000 building Smart Check, a welfare-fraud screening algorithm engineered to avoid discrimination. A live 2023 pilot of about 1,600 applications was no more accurate than caseworkers and still showed bias, so the city shelved it.

€535,000
Project cost
~1,600
Live welfare applications scored, 2023 pilot
Smart Check — Amsterdam Built a 'Fair' Welfare-Fraud AI, Then Shelved It When Bias Reappeared in Live Testing

Details

Maturity
Discontinued
Promoter
City of Amsterdam, Department of Work, Income and Participation
Period
2019–2023
Keywords
social welfare, algorithmic fairness, fraud detection, public administration

Context

After a Dutch court struck down the national SyRI welfare-fraud algorithm in 2020 for violating human rights, Amsterdam set out to build a welfare-fraud screening algorithm engineered to avoid discrimination.

Activities

The city's Department of Work, Income and Participation built Smart Check, an explainable boosting machine scoring about 15 characteristics while excluding gender, nationality, age and proxy variables such as postal code. Officials commissioned bias audits, human-rights assessments and welfare-recipient feedback, reweighting training data after early bias was found.

Results

Deployed on roughly 1,600 live applications in 2023, Smart Check proved no more accurate than human caseworkers, and bias resurfaced in a different form (over-flagging Dutch nationals, women and applicants with children).

Conclusions

Amsterdam shelved the project in November 2023 and published the code, model documentation and audit data, producing an unusually transparent case study of why fairness engineering alone did not guarantee a non-discriminatory outcome.

Implementation
Indicative cost
Medium (€50k–€500k)
Time to results
Long (> 3 years)
Staffing & skills
Amsterdam Department of Work, Income and Participation, Deloitte (€35,000 contract), academic and welfare-recipient consultees

Conditions for success

  • excluding protected/proxy variables from the model
  • independent bias audits before and during deployment

Common failure modes

  • bias resurfaced in a different form after reweighting
  • no more accurate than existing caseworker judgement
  • project ultimately shelved

Commonly funded by

Own resources / municipal budget

Indicative funding routes for practices of this type — always check each programme's current calls and eligibility rules.

Do you run this practice? Claim it — verified implementers get a public contact pathway and can propose corrections.

Data sources

Where this practice's information was retrieved from, and when.

Attachments

Similar practices you may find useful