AIM-01-010-P · Primer — Data Leakage

The idea in one sentence

Data leakage is when information the model would never have at prediction time sneaks into training — so it scores brilliantly on your test set and collapses in the real world.

Why it exists

Imagine a student who studies with the answer key taped inside the textbook: they ace the practice exam and learn nothing, because the exam stopped measuring understanding the moment the answers were in reach. Leakage does exactly this to a model — it hands over a shortcut to the correct answer, so the reported score measures the leak, not the skill. It is insidious precisely because nothing errors out; the pipeline runs clean, the metric looks great, and everyone celebrates a model that has quietly learned to cheat.

How it works

There are two dominant forms. Target leakage is a feature that is a consequence of the label rather than a cause available beforehand: in a fraud model, an "account status" column where fraudulent accounts get frozen the day after fraud is confirmed lets the model learn "frozen equals fraud" and hit 99.9% accuracy — but a brand-new transaction has no frozen flag yet, so in production the feature is blank and the model is useless. Train-test contamination is subtler: if you scale or impute using the whole dataset and only then split into train and test, the test rows have already influenced the mean and the standard deviation, so the model peeked at the test set through the back door and your score is inflated. Cousins include temporal leakage (training on next week's rows to predict this week) and duplicate/group leakage (the same customer or patient appearing in both train and test, so the model just recognizes a row it has already seen).

See it — leaky order vs. leak-proof order

flowchart TD
    D["raw dataset"] --> W["WRONG:<br/>fit scaler on<br/>ALL rows"]
    D --> R["RIGHT:<br/>split FIRST"]
    W --> WS["then split<br/>train / test"]
    WS --> WB["test stats<br/>leaked into fit<br/>= score inflated"]
    R --> RT["fit scaler on<br/>TRAIN only"]
    RT --> RP["transform test<br/>with train stats<br/>= honest score"]

The one gotcha

A suspiciously high score is not a trophy — it is the first symptom of leakage. Beginners see 99% accuracy and celebrate; practitioners get nervous and immediately hunt for the feature that is "too good to be true," asking of every column: would I actually have this value before the outcome happens? The rule that prevents almost all of it is boring and absolute: split first, then fit every transformation on the training data only.

See it (visualization plan)

Open with a CompareScene/TilesScene contrast — a giant BigNumberScene counting up to 99.9% test accuracy, then the same model at ~52% in production. Use a ChainScene to trace target leakage: fraud happens → account frozen next day → "status = frozen" feature → model reads the answer. Then a PipelineScene/LayersScene showing the contamination order: the wrong way (scale on all data → split) versus the right way (split → fit scaler on train only → transform test), with the leaking path drawn dashed and red.

The one-line takeaway

If it looks too good to be true, a feature is leaking the answer.

Where you'll meet it

You meet leakage the first time a scikit-learn model scores 99% and you can't reproduce it live — which is the whole reason Pipeline exists, so preprocessing is fit inside each cross-validation fold and never on the held-out data. Reach for TimeSeriesSplit on temporal problems, group-aware splitters (GroupKFold) when the same entity recurs, and always call train_test_split before you touch a scaler or imputer. And when a leak runs undetected long enough, you get the war-story version — resilience Drill 10, The Contaminated Year, where a shared feature cache leaked evaluation rows into training for a full year and left every shipped metric suspect at once.

Named concepts: data leakage, target leakage, train-test contamination