Drill 35 — The Model That Ate Its Own Tail

Theme: Feedback Loops & Adversaries · Format: 20-min drill · Difficulty: core

The story

Fraud model v2 was trained on the outcomes of cases v1 flagged. V3 trained on v2's. Each generation gets sharper on known fraud patterns and blinder everywhere else — the training data is, by construction, only what previous models chose to look at. A new fraud pattern grows for eight months in the blind spot. It's finally discovered through external data — network-level chargeback reports — not by anything the model or its metrics surfaced.

⏸ Stop here and ask the room: every generation's offline metrics improved. Explain how that's compatible with eight months of growing losses.

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Read the story aloud, pause at the blockquote, and let the room sit with the apparent contradiction — improving metrics, growing losses — before opening the questions.

"What population are the metrics computed on?"

Listen for: the observation that precision and recall were computed on investigated cases — cases a previous model chose to flag. The metric population and the transaction population diverged generations ago; the metrics are honest about an ever-narrower corridor.

Weak signal: answers that jump to "the model overfit" or "it needs more data." Overfitting is a modeling story; this is a data-generation story — more of the same data makes the corridor deeper, not wider.

"What data could ever reveal the blind spot?"

Listen for: data whose existence doesn't depend on the model's decisions — a score-blind random audit slice, external chargeback feeds, network-level reports, customer complaints. The load-bearing phrase is outside the loop.

Weak signal: proposals to slice or dashboard the existing labels harder. Every cut of corridor data is still corridor data — no view of it contains the blind spot.

"What would you permanently change about the training loop?"

Listen for: a standing exploration/audit budget — a fixed share of cases investigated regardless of score — plus headline metrics recomputed on that audited sample. Bonus points if someone prices the audit slice as a sensor cost rather than an efficiency loss.

Weak signal: one-off fixes — "retrain on the chargeback data this time." That patches this blind spot and leaves intact the loop that manufactures the next one.

The reveal

The metrics were honest — about the slice of the world the loop still examined. The system had optimized itself into a corridor.

Why this happens — the mechanics

Selection bias amplification is the engine here. A fraud model isn't just a predictor — it is the gatekeeper for which cases ever get investigated, and investigation is what creates labels. Machine learning (ML) systems trained on their own downstream labels inherit that gate: regions the model never flags never produce labels, so to the next generation those regions simply do not exist as fraud. What starts as v1's mild blind spot becomes v2's structural certainty, because v2 has literally never seen a counterexample.

The compounding is what makes it vicious. Each generation reallocates its capacity toward the corridor where labels are dense, getting genuinely sharper there — which is why offline precision and recall improve. But the evaluation sets are drawn from the same filtered stream, so the metrics measure performance inside the corridor and say nothing about the widening territory outside it. The metrics stay green because reality is decaying: the narrower the corridor, the better the model looks on it.

Meanwhile the adversary's incentive gradient points exactly at the blind spot. Fraud that resembles known patterns gets caught; fraud that doesn't gets through, profits, and scales. The loop doesn't just fail to see the new pattern — it actively subsidizes it for as long as the corridor holds. That is why discovery came from network-level chargebacks: only a data stream whose existence didn't depend on the model's choices could carry the bad news.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant M as Model vN
    participant IV as Investigators
    participant LB as Training labels
    participant EX as External chargebacks
    loop Every retrain cycle
        M->>IV: Flags cases in the<br/>known-fraud corridor
        IV->>LB: Labels exist only for the<br/>flagged region (silent)
        LB->>M: vN+1 trains on corridor<br/>labels - sharper there,<br/>blinder elsewhere (silent)
    end
    Note over M: Offline metrics improve<br/>every generation
    Note over M: New fraud pattern grows in<br/>the blind spot for 8 months
    EX->>IV: Network-level chargeback<br/>reports reveal the losses
    Note over M,EX: Found by external data -<br/>not by any metric

Solution — where the guardrails go:

flowchart TD
    A["Model vN flags cases"] --> B["Flagged cases get investigated"]
    S["Score-blind random audit slice"] --> C["Labels from outside the<br/>corridor"]
    X["External outcome feed:<br/>network-level chargebacks"] --> C
    B --> D["Labels from the flagged region"]
    D --> E["Model vN+1 training data"]
    C --> E
    E --> A
    C -.-> M["Metrics computed on the<br/>audited sample"]
    M -.-> AL["Blind-spot growth alarms<br/>in weeks, not months"]

What good looks like

• Size the audit slice statistically, and write the numbers down: a score-blind random sample large enough that a fraud pattern reaching X basis points of volume is detectable within N weeks at agreed confidence.
• Make the audit slice score-blind by construction — sampled before scoring, investigated with the same rigor as flagged cases, and protected as a permanent budget line item, not a discretionary one.
• Compute headline precision/recall on the audited sample; keep corridor metrics if useful, but label them as corridor metrics.
• Contract at least one outside-the-loop outcome feed (network chargebacks, bureau outcomes) and reconcile it monthly against model catches — the gap between the two is the measured blind spot.
• Alarm on blind-spot growth: fraud found in the audit slice or external feed but never flagged by the model, trended per month.
• Track training-data lineage per model generation — what fraction of labels came from model-flagged investigations vs. audits vs. external feeds. A ratio drifting toward 100% model-flagged is a launch-review red flag.
• Add "where does outside-the-loop ground truth come from?" as a required field in the model card for any model that gates its own labels.

The lesson

A model that filters its own future training data narrows every generation. Buy ground truth from outside the loop: a permanent random-audit slice (score-blind sampling), external outcome feeds, and metrics computed on the audited sample — not on the filtered stream. Exploration isn't waste; it's the sensor budget.

Named concepts: feedback loops in ML, selection bias amplification, exploration budgets, independent ground truth.

Real-world echoes

• Zillow Offers shutdown (2021) — Zillow's home-buying algorithm operated inside its own feedback loop: the offers it made determined which homes it bought, and adverse selection (sellers accept most eagerly exactly where the algorithm overpays) plus a shifting market left it holding inventory worth less than it paid. The unit was shut down with write-downs north of $500 million — the cost of metrics that described the algorithm's interaction with the market rather than the market itself.

Transfer

For each model whose outputs decide what gets labeled next — where does its outside-the-loop ground truth come from? "Nowhere" is a finding, not an answer.

Make it a ticket: Inventory every model that gates its own labels; for each, record its outside-the-loop ground-truth source and the size of its score-blind audit slice. File one gap ticket per "nowhere."

Related

• Drill 40 — The Prophecy That Fulfilled Itself — the same loop with an intervention in the middle: the model's action contaminates the outcome it's judged on.
• Drill 41 — The Coached Application — an adversary curates the inputs instead of the loop curating the labels; same fiction, different author.
• Drill 1 — The Model That Scored Everyone the Same (Theme 1) and Drill 46 — The Poisoned Feedback (Theme 7) — green metrics over decaying decisions, and an attacker feeding the retraining loop on purpose.