BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

CASE-27 · Case Study — The Health Algorithm That Under-Served Black Patients

What happened

A widely used commercial algorithm — applied to an estimated 200 million Americans — helped hospitals decide which patients were sick enough to enrol in high-touch "care management" programs. In 2019 researchers publishing in Science showed it was racially biased: at any given risk score, Black patients were considerably sicker than white patients with the same score. Because the score gated access to extra care, the bias meant Black patients had to be far sicker to receive the same help. The authors calculated that correcting the bias would more than double the share of Black patients automatically flagged for the program, from about 18% to 47%.

See it — the wrong target variable

flowchart LR
  A["Goal: predict who is<br/>SICKEST"] --> B["Proxy used: who will<br/>COST the most"]
  B --> C["Less spent on Black patients<br/>(access, not health)"]
  C --> D["Model: 'lower cost'<br/>= 'lower need'"]
  D --> E["Sicker Black patients<br/>scored as low-risk"]

The root cause

There was no malicious feature and, again, no race input. The flaw was the label. Predicting "health need" is hard, so the vendor trained the model to predict future healthcare cost as a stand-in — a reasonable-looking proxy, because sicker people usually cost more. But in the US, less money is spent on Black patients at equal levels of sickness, due to reduced access and structural inequities in care. So "cost" was a biased proxy for "need": the model learned that Black patients cost less and therefore concluded they needed less, when in fact they were being treated less. The optimiser did its job perfectly — on the wrong objective.

The fix that would have prevented it

The researchers showed the fix is often upstream of the model, in the choice of target variable. Re-specifying the label from "cost" to a direct measure of health — number of active chronic conditions, or predicted avoidable illness — shrank the bias by roughly 84% without touching the features or the algorithm. The general practice: interrogate every proxy target by asking "does the thing I'm predicting diverge from the thing I actually care about, differently across groups?" Audit outcomes by protected group before deployment for a high-stakes allocation decision, and keep a human clinician in the loop with the authority to override an automated triage.

The blast radius

The algorithm was already steering care for a substantial share of the US population when the bias was quantified, meaning years of under-referral had likely accumulated. The finding triggered a New York regulatory inquiry, forced the vendor to work with the researchers on remediation, and reframed the fairness conversation: the most consequential bias often lives not in the model but in the innocent-looking label an engineer chose because it was easy to measure.

The lesson

Choosing the target variable is an ethical decision, not a technical convenience. When the label you can measure (cost) drifts from the label you mean (need) unevenly across groups, an accurate model will encode that gap as a policy. Debiasing frequently means fixing the objective, not the algorithm — and it is nearly free when you catch it early and impossible to see if you never audit outcomes by group.

Where it rhymes