BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

CASE-24 · Case Study — COMPAS: The Risk Score That Wasn't Fair

What happened

COMPAS is a proprietary tool that scores a defendant's risk of re-offending, used across US courts to inform bail, sentencing and parole. In 2016 ProPublica obtained the scores for more than 7,000 arrestees in Broward County, Florida, waited two years to see who actually re-offended, and compared the predictions to reality. The model was roughly as accurate for Black and white defendants — but its errors fell in opposite directions. Among people who did not go on to re-offend, Black defendants were nearly twice as likely to have been labelled high-risk (a false positive). Among people who did re-offend, white defendants were far more likely to have been labelled low-risk (a false negative). Same accuracy, wildly different harm.

See it — one model, two very different error profiles

flowchart LR
  A["Historical arrest<br/>data (biased)"] --> B["COMPAS risk<br/>score 1-10"]
  B --> C["Black, did NOT reoffend<br/>→ often HIGH risk (FP)"]
  B --> D["White, DID reoffend<br/>→ often LOW risk (FN)"]
  C --> E["Same overall accuracy,<br/>opposite harms"]
  D --> E

The root cause

The model never used race as an input — and that is exactly the trap. It used proxies: prior arrests, neighbourhood, employment, age at first contact with police. In a world where policing itself is unevenly distributed, "prior arrests" already encodes race, so a race-blind model happily reconstructs a race-aware decision. Deeper still, the fight that followed exposed a mathematical fact, not a coding bug: Northpointe was right that COMPAS was calibrated (a score of 7 meant the same re-offend rate for everyone), and ProPublica was right that the false-positive rates differed by race. When the underlying base rates of two groups differ, a score cannot satisfy both calibration and equal error rates at once — you must choose which notion of fairness to honour. COMPAS chose one silently, and never told the courts which.

The fix that would have prevented it

There is no toggle that makes this "fair"; the fix is to make the trade-off explicit and owned. Pick a fairness definition on the record — calibration, equalized odds, or equal false-positive rates — justify it for the stakes (a false positive here can cost someone their freedom), and audit against it before deployment, not two years later. Interrogate the proxies: measure how strongly features like prior-arrest reconstruct race, and drop or reweight the worst offenders. And treat a risk score as advice to a human, never a verdict — surface the confidence interval and the base rate so a judge sees "this is a weak signal", not "the computer said 7".

The blast radius

The scores had already shaped bail and sentencing for hundreds of thousands of defendants before anyone audited the error asymmetry. Because COMPAS was a trade secret, defendants could not inspect the logic used against them — a due-process problem the Wisconsin Supreme Court flagged in State v. Loomis. The episode became the canonical example in every fairness curriculum since, and hardened a rule regulators now repeat: "we didn't use the protected attribute" is not a defence.

The lesson

Accuracy is not fairness, and "race-blind" is not race-neutral. A model trained on biased history will launder that bias through proxy features while looking objective. When group base rates differ, fairness is a menu, not a checkbox — you cannot have every desirable property at once, so someone accountable must choose which error to minimize and say so out loud.

Where it rhymes