Drill 10 — The Contaminated Year

Theme: Data Disasters · Format: 20-min drill · Difficulty: advanced

The story

An internal audit finds that a shared feature cache leaked evaluation-period rows into training data — for a year. Every model shipped in that window, every reported metric, every "we improved fraud catch by 12%" claim is now suspect. Three of those models are making production decisions right now.

⏸ Stop here and ask the room: you found it. What are the first three things you do, and who do you tell?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Do you turn the models off?

Listen for: A risk-based answer — for each contaminated model, compare it against the realistic fallback (the previous model, a rules engine, human review); contamination usually means overrated, not worthless, so the call is per-model, made against clean re-evaluation numbers, with monitoring tightened in the meantime. Weak signal: A reflexive "kill them all" or "leave them running" — either answer given without asking what would replace the models or what their clean-data performance actually is.

How do you quantify the real damage?

Listen for: Re-evaluate every affected model on clean out-of-time data, recompute each headline metric on that basis, and diff the decisions the contaminated models actually made against what the clean baseline would have decided — that diff is the damage. Weak signal: "Re-run the numbers" without specifying clean, held-out, post-period data — re-running against the same contaminated cache just reproduces the lie with more confidence.

What do you say to leadership — and to the client whose numbers you reported?

Listen for: Incident discipline — declare a severity, notify with the known/unknown/next-steps structure, commit to re-stated metrics by a date; the client conversation is treated as owed, not optional, because the reported 12% was an input to their decisions. Weak signal: "Wait until we know everything before saying anything" — the delay that quietly becomes the coverup.

What does the quiet-fix instinct cost?

Listen for: The insight that discovery is survivable and concealment isn't — a contamination found and disclosed is an engineering incident; a contamination found, hidden, and later discovered is an integrity scandal that voids trust in every number the team has ever reported. Plus the internal cost: engineers watching a quiet fix learn that surfacing problems is punished. Weak signal: Framing the quiet fix as pragmatic — "we fixed it, no harm done" — without pricing the asymmetry between correcting a metric and being caught having known.

The reveal

(Discussion, not a twist.) The honest path: treat it as a full incident — severity, comms, timeline — re-evaluate on clean out-of-time data, and re-state the metrics. Usually the models are less good but not worthless; the coverup, not the contamination, is what destroys trust.

Why this happens — the mechanics

Shared feature caches exist for good reasons: computing features once and reusing them everywhere saves enormous compute and keeps teams consistent. But train/eval separation is a property of the training process, not of the data — the cache has no idea which rows will later serve as someone's evaluation period. The moment a cache is shared across teams and time, every consumer inherits whatever leakage its construction allowed, and one subtle flaw contaminates a year of downstream work without any single team doing anything wrong.

The organizational mechanics are what stretch it to a year. Inflated metrics look like success, and success is not a symptom anyone investigates — the failure mode actively rewards the people inside it. Reported numbers ratchet: once "fraud catch up 12%" has gone into a leadership deck and a client report, it becomes a baseline with constituents, and re-opening it has a political price. Meanwhile the checking cadence is annual (audit) rather than continuous (CI — Continuous Integration, the automated pipeline that tests every change), so the detection latency is built in.

It's invisible for the same reason it's seductive: nothing is broken. Pipelines run, models ship, metrics climb. The only observable symptom — offline numbers persistently rosier than production reality — is routinely absorbed as "drift" or "the market." That's why the fix is structural: contamination checks on every training run, so the gap between train and eval is a machine-enforced invariant rather than an annual archaeology project.

Diagrams

Problem — how the failure unfolds:

timeline
    title A year of contamination
    section Quarter 1
        Cache built : Shared feature cache ships : Eval rows leak in (silent)
    section Quarters 2-3
        Models shipped : Every model trains on the future : Three make production decisions
    section Quarter 4
        Metrics reported : Inflated wins go to leadership : 12% fraud-catch claim to client
    section One year in
        Audit finds the leak : Every shipped model suspect : Every reported metric suspect

Solution — where the guardrails go:

flowchart TD
    A["Shared feature cache feeds<br/>training run"] --> B["Contamination check in CI<br/>on every run"]
    B --> C["Train and eval row overlap<br/>detected"]
    C --> D["Build blocked before<br/>training starts"]
    C -.-> E["Alarm fires: leaked rows<br/>identified same day"]
    D --> F["Re-evaluate on clean out-<br/>of-time data"]
    F --> G["Metrics re-stated,<br/>incident disclosed"]

What good looks like

A contamination check in CI on every training run: hash-based row-overlap between training and evaluation sets, with the build failing on any nonzero overlap — not a quarterly script, a merge gate.
Out-of-time evaluation enforced by the framework, not by convention: the evaluation window must end-to-end postdate the training window, and the tooling refuses configurations that don't.
Feature caches carry split and timestamp metadata; training code refuses any feature value computed later than the label's decision time.
Metric provenance on every reported number: run identifier, dataset hashes, evaluation window — so "which numbers are affected?" is a query, not a month of forensics.
A correctness-incident runbook that predates the incident: severity levels, comms templates for leadership and clients, and a decision tree for whether each affected model keeps running.
A scheduled red-team exercise: deliberately plant a leaked row in a staging training set once a quarter and verify the checks catch it — an unverified contamination check is unverified code.
A blameless-disclosure norm made visible: the person who finds contamination gets public credit, so the next finder comes forward in a day, not a year.

The lesson

Correctness failures deserve the same incident machinery as outages: declared severity, honest disclosure, remediation, postmortem. And eval hygiene belongs in CI — contamination checks on every training run, not annual audits.

Named concepts: correctness incidents, blameless disclosure, eval-as-CI, out-of-time validation.

Real-world echoes

Corporate versions of this incident essentially never go public under their own names — they surface, if at all, as quietly revised numbers. The closest public analogue is academic: researchers studying machine-learning-based science have documented leakage as a leading cause of results that later had to be walked back across dozens of fields. The corporate silence is the point — you will not get to learn this one from someone else's postmortem.

Transfer

If our headline metric were wrong for a year, how would we find out? What contamination check runs automatically today?

Make it a ticket: add a train/eval row-overlap check to the CI of our most important model's training pipeline, failing the build on overlap — and dry-run the correctness-incident comms template against a hypothetical while nothing is on fire.

Drill 8 — The Backfill That Rewrote History — the other route to training on the future, via edited history instead of a leaky cache.
Theme 1, Drill 5 — The Twice-a-Year Bug: the cultural failure is the same — an unexamined anomaly (there, a dip; here, a suspiciously good number) left uninvestigated.