Drill 95 — The Test Set It Had Seen

Theme: Evaluation & Validation · Format: 20-min drill · Difficulty: core

The story

A consumer lender's data-science team spends six weeks rebuilding the loan-default model. The pipeline is clean, modern, and reviewed: two years of applications pulled into one dataframe, features scaled, a gradient-boosted model swept across a few hundred hyperparameter configurations, the best one reported on a held-out test set. The number is the best anyone has seen on this problem — AUC 0.94, up from the incumbent's 0.88. The readout deck lands, risk signs off, and the challenger goes live for new underwriting decisions.

Two months in, the model-monitoring dashboard tells a different story. On live loans, the challenger's discrimination has collapsed to AUC 0.71 — worse than the model it replaced. Nobody shipped a bug. The features are computed the same way in production as in training; there is no train-serve skew to find. The scoring code is byte-identical to the offline pipeline. The model is doing exactly what it was built to do; it was just never really tested.

The team re-runs the offline evaluation to sanity-check it. It reproduces perfectly: 0.94, every time. The holdout still says the model is excellent. Two numbers, both computed correctly, that cannot both be true about the same model.

⏸ Stop here and ask the room: the offline AUC of 0.94 reproduces exactly and there's no train-serve skew. So how can the same model score 0.94 on the holdout and 0.71 in production?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

The holdout reproduces at 0.94. What would make that number a lie even though it's computed correctly?

• Listen for: data leakage — information from the test set influenced training, so the "held-out" score measures memorization, not generalization. Concretely: rows the model trained on also appear in test (the same customer has five loans split across both sides), so the model has effectively seen the answers. The metric is real; the holdout is not independent.
• Weak signal: "the holdout was too small" or "it overfit" — overfitting shows up as a train/test gap; here the test score is inflated too, which is the signature of leakage, not ordinary variance.

The features are scaled before the split, and hyperparameters were chosen by test AUC. Why does each of those leak?

• Listen for: preprocessing leakage — fitting the scaler / imputer / target-encoder on the full dataset lets the test set's statistics (means, category rates, quantiles) bleed into the training features; the transform has already "seen" the test distribution. And tuning on the test set — picking the config with the best test AUC across hundreds of tries turns the test set into a training signal for the hyperparameters; you optimized directly against the thing that was supposed to be an independent check.
• Weak signal: "scaling is harmless, it's just normalization" — any transform fit on data outside the training fold is leakage; harmless-looking steps are the most common culprits precisely because nobody flags them.

How should this have been split and evaluated so the number couldn't lie?

• Listen for: split by entity (customer / account) before anything is fit, so no customer straddles train and test; fit every transform inside each cross-validation fold (a pipeline, not a pre-scaled matrix); tune hyperparameters with nested cross-validation so the outer fold never informs the inner search; and keep a final holdout that is opened exactly once, at the very end, to produce the number you actually report.
• Weak signal: "add more cross-validation folds" — more folds over a leaky split just averages the same inflated number; the fix is what you split on and when you fit, not how many folds.

The reveal

The holdout was never held out. Three leaks stacked: (1) the random row-level split put the same customers on both sides — roughly 9% of test rows shared a customer with a train row, and repeat borrowers behave alike, so the model recognized them; (2) the StandardScaler and target-encoder were fit on the full dataframe before the split, so test-set statistics were baked into the training features; (3) the hyperparameter sweep selected the config with the highest test AUC, quietly fitting the model's knobs to the test set across a few hundred trials. Each leak alone inflates the score a little; together they manufactured a 0.94 that described a test set the model had already, in effect, seen. Production was the first truly unseen data it ever met — and there it scored 0.71. The team rebuilt the split around the customer, moved every transform inside the fold, added nested CV for tuning, and locked a final holdout opened once. The honest offline number came back 0.79 — lower, but it matched production, and for the first time "it's better" meant the same thing in both places.

Why this happens — the mechanics

A test set only measures generalization if the model learned nothing from it — directly or indirectly. "Indirectly" is where competent teams get caught, because every convenient shortcut in a modern pipeline is a channel for test information to leak into training. Fitting a transform on the whole dataset before splitting leaks distributional information (the test set's means and category frequencies). A random train_test_split on rows leaks identity information when the natural unit of the problem is an entity that owns many rows — the same customer, device, or account appears on both sides, and the model recognizes the entity rather than predicting the outcome. Selecting hyperparameters by test-set score leaks optimization information: with enough trials, the test set becomes a surface you are climbing, and its AUC stops being an estimate of anything. In every case the offline number is computed correctly; it is just answering a question — "how well does it do on data related to what it trained on?" — that nobody wanted the answer to.

The organizational reason it survives review is that leakage is invisible in the code diff. scaler.fit(X) then train_test_split(X) looks tidy and passes tests; the two-line reorder that makes it correct looks like a no-op. The metric is precise, green, and reproducible, so it reads as trustworthy — and the only thing that could have contradicted it, an untouched final holdout, was spent early or never set aside. The discipline is to treat the pipeline order and the split key as first-class parts of the evaluation, and to keep one slice of data the model provably never influenced.

Diagrams

Problem — how a "held-out" score gets built from data the model already touched:

timeline
    title The holdout that wasn't
    Week 1 : Load 2 years of loans<br/>one row per loan
    Week 2 : Fit scaler + encoder<br/>on FULL data (silent leak)
    Week 3 : Random split by ROW<br/>same customer both sides (silent)
    Week 4 : Sweep 300 configs<br/>keep best TEST AUC (silent leak)
    Week 6 : Report AUC 0.94<br/>ship the challenger
    Month 2 : Live AUC 0.71<br/>first truly unseen data

Solution — an evaluation the model provably never saw:

flowchart TD
    A["Raw loans<br/>(one entity, many rows)"] --> B["Split by CUSTOMER first<br/>before any fitting"]
    B --> C["Cross-validation folds"]
    C --> D["Fit transforms INSIDE<br/>each fold (pipeline)"]
    D --> E["Nested CV for<br/>hyperparameter search"]
    B --> F["Locked final holdout<br/>(opened once)"]
    G["Near-duplicate / shared-entity<br/>scan across the split"] -.-> B
    E -.-> H["Offline gap alarm<br/>(offline vs live > 0.05 AUC)"]
    F -.-> H
    H --> I["Report the number the<br/>model never influenced"]

What good looks like

• Split by entity before you fit anything: group the split on the natural key (customer_id / account_id) with GroupKFold or StratifiedGroupKFold, so zero customers straddle train and test — assert it in code, and fail the run if the intersection is non-empty.
• Fit every transform inside the fold: wrap scaling, imputation, and target/category encoding in a Pipeline that is .fit only on the training portion of each fold — never .fit on the full matrix. A pre-scaled feature table handed to CV is a leak by construction.
• Tune with nested cross-validation: the hyperparameter search runs in an inner loop; the outer fold that reports performance never participates in selection. Report the outer-loop score, not the best inner-loop score.
• A final holdout opened exactly once: carve off a time-forward slice (the most recent N months, split by customer) at project start, and open it a single time to produce the number you report. If you looked at it twice, it's a validation set now, not a holdout.
• Near-duplicate and shared-entity scan across the split: before training, scan for exact and near-duplicate rows (and shared entities) that cross the train/test boundary; alert if leaked pairs exceed 0% for exact duplicates or a small tolerance for near-duplicates.
• An offline-vs-online gap tripwire: track reported-offline metric minus realized-live metric for every shipped model; a persistent gap beyond 0.05 AUC pages the owner and blocks the next promotion — the gap itself is the leak detector of last resort.

The lesson

A test set measures generalization only if the model learned nothing from it — not through duplicated entities, not through a transform fit before the split, not through a hyperparameter chosen by its score. Split by the entity, fit inside the fold, tune in a nested loop, and keep one slice the model provably never touched — then the offline number will survive contact with production.

Named concepts: data leakage, train/test contamination, preprocessing leakage, group/entity-aware splitting, nested cross-validation.

Real-world echoes

• "Leakage and the reproducibility crisis in ML-based science" (Kapoor & Narayanan, 2023). A survey across 17 fields found hundreds of published studies whose reported accuracy was inflated by data leakage — most commonly no clean train/test separation, preprocessing fit on the full dataset, and the same subjects appearing on both sides of the split. When the leaks were closed, the impressive numbers shrank or vanished. The same failure this drill dramatizes, at the scale of an entire literature: a metric computed correctly on a holdout that was never actually held out.

Transfer

For your most important model, can you prove the reported metric came from data the model never influenced — no shared entity across the split, no transform fit before the split, no hyperparameter chosen by the test score? "We used a holdout" is not proof; name the split key and the point in the pipeline where each transform is fit.

Make it a ticket: for the highest-stakes model, switch the split to group-by-entity, move all preprocessing inside a CV pipeline, add a cross-split duplicate/shared-entity scan that fails the build above zero exact leaks, and reserve a final holdout opened once — then compare the new offline number against live.

Related

• Drill 93 — The Metric That Lied — the sibling failure: the holdout was clean but the metric measured the wrong thing; here the metric is right but the holdout is contaminated.
• Drill 87 — Train-Serve Skew — the lifecycle cousin: there the served features differed from training; here training and serving agree, and the evaluation is what lied.
• Drill 4 — The Stale Score — another gap between an offline score and the world it claims to describe.
• Pattern #132 — SLO / Error-Budget Policy — the same reflex: decide on a number the system can't quietly game.