BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

CASE-25 · Case Study — Amazon's Recruiting AI That Taught Itself Sexism

What happened

Starting around 2014, Amazon built an experimental engine to rank résumés the way its warehouse recommender ranked products — feed in a CV, get back a one-to-five-star hireability score. It was trained on ten years of résumés the company had actually received and the hiring outcomes attached to them. By 2015 engineers noticed the model was systematically downgrading women. It penalised résumés containing the word "women's" (as in "women's chess club captain") and marked down graduates of two all-women's colleges. Amazon tried to patch the specific terms, could not convince itself the model wasn't finding new proxies, and quietly killed the project.

See it — the loop that encodes the past

flowchart LR
  A["10 yrs of résumés,<br/>mostly men hired"] --> B["Model learns:<br/>'looks like a hire' = male-coded"]
  B --> C["Penalises 'women's',<br/>all-women colleges"]
  C --> D["Patch the words →<br/>finds new proxies"]
  D --> E["Project abandoned"]

The root cause

The labels were the problem. The model was asked "who looks like the people we hired?" — and in a male-dominated field, the honest answer was "men." It faithfully learned the pattern in the data, which was a decade of human bias, and reproduced it with the confidence of mathematics. Because it never saw gender as a field, it manufactured proxies: verbs, hobbies, colleges, phrasing that correlated with gender. Every time engineers scrubbed one proxy, the optimiser was free to lean on the next. You cannot debias a model whose target variable is itself the biased outcome you are trying to escape.

The fix that would have prevented it

Start upstream: audit the training labels for the bias you're about to automate, and recognise that "past hiring decisions" is a poisoned target when your goal is fairer hiring. Measure adverse impact — the four-fifths rule and selection-rate parity across groups — as a first-class metric during training, not a post-mortem. Adversarially probe for proxies by testing whether a second model can predict gender from the "gender-blind" features; if it can, the bias is still in there. And keep hiring decisions human-owned with an audit trail, using models only to widen the funnel (surface overlooked candidates), never to silently filter it.

The blast radius

Amazon says the tool never became the sole basis for a hire, and it was scrapped before wide rollout — so the direct damage was contained. The lasting impact was reputational and instructive: it became the textbook proof that a well-meaning, race- and gender-blind model can still discriminate, and it stiffened the emerging regulation of hiring algorithms (New York City's Local Law 144 now mandates bias audits for exactly this class of tool).

The lesson

A supervised model is a mirror of its labels. If the outcome you're predicting was produced by biased humans, an accurate model will reproduce the bias precisely — and hide it behind features that look neutral. "We removed the protected attribute" solves nothing when a dozen proxies remain. Sometimes the correct engineering decision is the one Amazon eventually made: do not ship it.

Where it rhymes