Drill 94 — The Leak From the Future

Theme: Evaluation & Validation · Format: 20-min drill · Difficulty: advanced

The story

A bank's servicing team builds a model to flag loans that are about to default, so collections can intervene a cycle early. The training data is a clean warehouse join: two years of loans, a few hundred features per account pulled from the feature store, and a binary label — did this loan charge off within twelve months. The team does everything by the book. They hold out a random 20% test set, tune on the rest, and report the number. The AUC is 0.97.

Nobody on the team has seen a risk model score 0.97 before. The usual ceiling for this kind of early-warning problem is somewhere in the low 0.80s, and the honest instinct in the room is unease. But the pipeline is audited, the split is stratified, there is no obvious duplication between train and test, and the calibration plot looks reasonable. The lead reasons that the new feature store simply exposes richer behavioral signals than the old models had. The deck says 0.97, the model passes review, and it ships to score the live book.

Six weeks later the collections team complains that the flags are useless — no better than the old rules. The MLOps dashboard confirms it in numbers nobody wants to see: the live AUC, measured against loans that have since resolved, is 0.61. The model that was near-perfect offline is barely better than a coin-weighted guess in production. No code changed between the two measurements. The training pipeline and the serving pipeline run the same model artifact on what everyone believes are the same features.

⏸ Stop here and ask the room: the holdout was random, stratified, and de-duplicated, and the 0.97 was real on that test set. How does a model measurably rank risk near-perfectly offline and then collapse to near-random the moment it makes live decisions?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

A 0.97 on a problem that tops out in the low 0.80s — what is that number actually telling you?

• Listen for: a score that is too good for the problem is a smell, not a triumph. The most common cause is target leakage — a feature that is a proxy for, or is populated from, the label itself. The first move is a leakage audit: rank features by importance and ask of the top few, "would this value exist, with this meaning, at the moment we actually make the decision?" A single feature that alone can score >0.90 AUC is a red flag, not a lucky break.
• Weak signal: "the feature store just has better signal now," or celebrating the number and shipping. Treating an implausible metric as good news rather than a hypothesis to falsify is the whole failure.

The train and serve pipelines run the same features — so why do the numbers disagree?

• Listen for: same column name is not same value. A feature can be available at training time but not at decision time: the offline warehouse row is assembled long after the loan resolved, so a field written by a resolution-time process is fully populated and meaningful in training, while at serving time — scoring an active loan — that field is null, stale, or carries a different meaning. This is a point-in-time correctness failure: the training row saw the future.
• Weak signal: "it must be drift" or "the holdout was too small." Neither explains a same-artifact, same-name collapse; the values, not the world, differ between train and serve.

The split was random and stratified. Why didn't that catch it?

• Listen for: a random split cannot detect leakage, because it smears future-derived information evenly across train and test — the leaky feature is just as predictive on the held-out rows, so the test set confirms the leak instead of exposing it. The evaluation has to respect time: temporal (out-of-time) validation — train on an earlier window, test on a strictly later one — and a train/serve availability check that verifies each feature is populated at decision time in production, not just in the warehouse.
• Weak signal: "add cross-validation" or "get a bigger holdout." More folds of the same random, non-temporal split reproduce the same leak with tighter confidence intervals.

The reveal

The leaking feature is days_since_last_payment. In the servicing warehouse, that column is not computed live — it is written by a nightly batch that stamps the value as of resolution for closed loans. A loan that charged off gets a large days_since_last_payment recorded at close; a healthy loan gets a small one. In the training snapshot, the field therefore separates defaulters from non-defaulters almost perfectly — because it was written after the outcome was known. It is the label wearing a feature's name.

At serving time the model scores active loans, where days_since_last_payment is the small, current, honest value — carrying none of the resolution-time signal it had in training. The model leans its entire weight on a column that means one thing in the past and another at the decision point, so it collapses to 0.61. The team rebuilds the feature with a strict point-in-time (as-of) join — only data timestamped at or before the decision moment — adds a leakage audit that flags any single feature scoring above 0.90 AUC, adds a train/serve availability check per feature, and replaces the random split with out-of-time validation. Rebuilt honestly, the model scores 0.83 offline and 0.82 live — because "predictive" finally means the same thing in both places.

Why this happens — the mechanics

Target leakage is a mismatch between the information a model has when it learns and the information it has when it decides. A supervised model will happily exploit any column correlated with the label, and it has no way to know that a particular correlation only exists because the column was written after the outcome. The offense is almost never a deliberate copy of the label; it is a field assembled by a downstream process — a resolution stamp, a collections flag, a days_since_last_payment recomputed at close — that already knows how the story ended. In the frozen training warehouse, every row is a completed history, so future-derived fields look like ordinary features. In production, the row is a live, half-written story, and those fields are empty or mean something else.

A random train/test split is powerless against this, and that is the trap that fools careful teams. Splitting rows at random keeps the leaky feature equally predictive in train and test, so the holdout — the thing everyone trusts to catch mistakes — rewards the leak with a near-perfect score. The only split that exposes a leak from the future is one that respects the arrow of time: train on the past, test on a strictly later period, and check that every feature the model relies on is actually populated, with the same meaning, at the moment of decision. The discipline is to treat an implausibly high offline metric as a hypothesis to falsify, not a result to ship — and to build features under a point-in-time contract so that the training row can never see anything the serving row won't.

Diagrams

Problem — how the leak from the future unfolds:

sequenceDiagram
    autonumber
    participant DS as Data Scientist
    participant FS as Feature Store
    participant V as Validation
    participant P as Production
    DS->>FS: pull training features
    FS-->>DS: days_since_last_pay<br/>(written post-resolution)
    Note over DS: feature encodes<br/>the outcome
    DS->>V: random split, evaluate
    V-->>DS: AUC 0.97 — ship it
    Note over V: leak sits in train<br/>AND test alike
    DS->>P: deploy same artifact
    P--xP: feature null at<br/>decision time
    Note over P: AUC collapses to 0.61

Solution — where the guardrails go:

flowchart TD
    A["New feature set"] --> G1["Point-in-time join<br/>(only data at decision time)"]
    A --> G2["Leakage audit on<br/>top features (single-feat AUC)"]
    A --> G3["Train/serve availability<br/>check per feature"]
    A --> G4["Temporal held-out<br/>validation (not random)"]
    G1 -.-> AL["Leak alarm —<br/>block promotion"]
    G2 -.-> AL
    G3 -.-> AL
    G4 -.-> AL
    AL --> H["Promote only if it holds<br/>on future-dated data"]

What good looks like

• Point-in-time feature construction: every feature is built with an as-of join that admits only data timestamped at or before the decision moment. No field written by a resolution-time or downstream process may enter a training row unless it existed, with the same value, at scoring time.
• A leakage audit on the top features: rank features by importance and compute each top feature's standalone AUC; any single feature above 0.90 AUC, or that alone explains a majority of model gain, is flagged and manually cleared before promotion. Suspiciously predictive is a stop, not a win.
• A train/serve availability check per feature: compare population/coverage in the offline warehouse against production at decision time. If a feature is ~100% populated in training but < 95% populated (or differently distributed) at serving time, the model is blocked until it is dropped or fixed.
• Temporal (out-of-time) validation, not random: train on an earlier window, test on a strictly later one. Treat an offline-to-temporal AUC drop > 0.10 as a leak signal to investigate, and never promote on a random split alone.
• An implausibility tripwire: pin the plausible ceiling for the problem class (here, low-0.80s AUC). Any model that beats it by a wide margin auto-triggers the leakage audit before anyone reads the deck.
• Track offline-vs-online AUC delta for every shipped model: a persistent gap is itself a bug in the evaluation, and a sudden near-random live score pages the owning team.

The lesson

A feature you only have after the outcome is not a feature — it is the answer, and a random split will happily grade it as genius. Build features under a point-in-time contract, audit the suspiciously predictive ones, and validate across time — so that "predictive" means the same thing when you train and when you decide.

Named concepts: target leakage, point-in-time feature construction, temporal (out-of-time) validation, train/serve feature availability, leakage audit.

Real-world echoes

• Kapoor & Narayanan, "Leakage and the Reproducibility Crisis in ML-based Science" (2023). The authors catalog dozens of published studies across fields whose headline accuracy came from data leakage — most often a feature or split that let information from the outcome bleed into training — and show the results evaporate once the leak is closed. The same-class lesson: an implausibly strong metric is far more often a leak than a breakthrough, and only time-respecting, point-in-time evaluation tells the difference.

Transfer

Look at your highest-stakes model and list its top five features by importance. For each, ask the point-in-time question out loud: would this value exist, unchanged, at the exact moment we make the decision in production? If you cannot answer yes for every one, you may be shipping a leak from the future.

Make it a ticket: for the top model, compute each top feature's standalone AUC and its train-vs-serve coverage, re-run validation on an out-of-time split, and block promotion on any single-feature AUC > 0.90 or an offline-to-temporal drop > 0.10.

Related

• Drill 93 — The Metric That Lied — the sibling failure: the offline number was real but measured the wrong thing on the wrong population; here it measures a leaked feature.
• Drill 87 — Train-Serve Skew — the lifecycle cousin: the model was fine, the served features weren't; leakage is that skew created at feature-build time.
• Drill 10 — The Contaminated Year — data-side leakage: outcome information contaminating the training window, caught only by respecting time.
• Pattern #132 — SLO / Error-Budget Policy — the analog of "decide on the metric that matters, in production, not the flattering offline one."