XAI-ED Consequential Assessment Framework — Auditing AI-Governance Documentation at 13 Italian Universities Under the EU AI Act

The EU AI Act classifies AI systems used to assess learning outcomes as high-risk under Annex III but does not specify how institutions should evaluate their own compliance. Researchers at Italy's National Research Council (CNR), with the University of Bologna and University of Bari 'Aldo Moro', built the XAI-ED Consequential Assessment Framework (CAF), a 27-item, four-point checklist operationalising the EU's ALTAI trustworthy-AI requirements and grounded in established educational-evaluation theory (Messick's validity framework, Kirkpatrick's four-level model, Stufflebeam's CIPP model).

To give universities and regulators a replicable, low-cost way to self-audit AI-assessment governance documentation ahead of the EU AI Act's high-risk provisions taking effect from August 2026.

Four independent coders applied the 27-item checklist to the publicly published AI policies of all 13 Italian universities that had formally adopted one by September 2025 (about 13% of Italy's roughly 100 universities, including Bologna, Politecnico di Milano and Florence), plus one institution without a policy as a baseline.

Intercoder reliability was strong (Fleiss's kappa=0.626; Krippendorff's alpha=0.838). Institutional governance scores ranged from 0 to 60.32 out of 100 (mean 33.66); scores were highest for 'societal well-being' commitments (mean 52.38) and lowest for 'technical robustness' (mean 19.64), showing universities document aspirational commitments more than operational safeguards.

The framework gives a replicable, low-cost self-audit tool, but the low average governance scores indicate most studied institutions are not yet operationally ready for the EU AI Act's high-risk provisions taking effect from August 2026.