Course Signals — Purdue's Predictive Early-Warning System and Its Debunked Retention Claim

Course Signals was a predictive early-warning system built at Purdue University from 2007, led by John Campbell, Kimberly Arnold and Matthew Pistilli. It combined student demographic data, prior academic history and current in-course performance (grades and engagement recorded in the learning-management system) into a single traffic-light indicator — red, yellow or green — shown to both the student and the instructor, who could then trigger personalised outreach such as emails or referrals to support services.

In a 2012 paper presented at the ACM Learning Analytics and Knowledge conference, Arnold and Pistilli reported that students who took two Signals-enabled courses saw their six-year graduation rate rise by 21.48 percentage points compared with students who took none. On the strength of that claim, Course Signals was expanded from pilot courses to institution-wide deployment and was widely cited across US higher education as a flagship example of learning analytics improving retention.

In 2013, independent analyses by Michael Caulfield (Washington State University Vancouver) and Alfred Essa (McGraw-Hill Education) challenged the finding. Caulfield showed that once Signals expanded beyond the original pilot, the retention advantage disappeared: students were taking more Signals-enabled courses because they were already persisting, not persisting because of Signals. Essa built a simulation substituting random 'chocolate' assignment for Signals participation and reproduced the same statistical pattern, demonstrating the original correlation was very likely an artifact of course-taking behaviour rather than a causal effect of the intervention. Purdue continued to operate Course Signals for several more years, but the system was ultimately decommissioned by its vendor, with institutional access ending on 16 December 2017.

Read the full analysis: https://eliterate.us/purdue-course-signals-data-issue-explainer/