BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 40 — The Prophecy That Fulfilled Itself

Theme: Feedback Loops & Adversaries · Format: 20-min drill · Difficulty: advanced

The story

A churn-risk model flags accounts likely to leave; flagged accounts are routed to a cost-saving "light touch" service tier (why invest in the doomed?). A year later, the model looks brilliant: flagged accounts churned at triple the rate. The team celebrates its predictive power.

⏸ Stop here and ask the room: did the model predict churn, or manufacture it?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Let the room argue both sides of the stop question for a minute before revealing that the honest answer — "you can't tell from this data" — is the point of the drill.

"What did the intervention do to the outcome being measured?"

Listen for: the treatment changed the outcome's causal history — flagged accounts received worse service, so their churn rate reflects prediction and punishment, inseparably. The label "churned" no longer means what the evaluation assumes it means.

Weak signal: "the model was probably still directionally right." Maybe — but nothing in this data can establish it, and confidence without an identification strategy is exactly the failure being drilled.

"How would you separate prediction quality from treatment effect?"

Listen for: randomization at the point of action — among flagged accounts, a random holdout treated normally. Prediction quality is flagged-holdout vs. unflagged churn; treatment effect is treated-flagged vs. flagged-holdout. Two comparisons, two questions, two owners.

Weak signal: comparing flagged-treated to unflagged accounts, or reaching for covariate adjustment on the contaminated observational data when a small, cheap experiment answers the question cleanly.

"What would an honest evaluation require?"

Listen for: the holdout as a permanent fixture with a pre-agreed size, model metrics computed only on it, and the treatment's own effect — including harm, does light touch cause churn? — reported alongside. Someone should say out loud that prediction and action are different questions.

Weak signal: a one-time backtest on pre-intervention history. Old data ages out fast and can't evaluate the current loop, where flagging changes treatment every cycle.

The reveal

Both, and you can't tell the proportions: the light-touch tier caused churn among customers who'd have stayed with normal service. The model's "accuracy" is partly its own echo. (The same structure hides in collections: aggressive treatment of predicted-defaulters can cause the default.)

Why this happens — the mechanics

A prediction that triggers an intervention is no longer just a prediction — it's a cause. Once flagged accounts are routed to a worse service tier, each account's outcome is a function of two things: who they were, and what you did to them. The evaluation, though, assumed outcomes were independent of the model's opinion — that churn would have unfolded the same way if nobody had looked. Deployment broke that assumption on day one, and every subsequent "accuracy" number quietly measures prediction quality plus treatment damage, in unknown proportions.

The vicious part is the direction of the bias: treatment contamination flatters the model. Degrading service to predicted churners pushes borderline customers out, so the flag appears more accurate than it is — the model literally earns its validation by causing the outcome. That validation then buys scope: more accounts flagged, more downgraded, more manufactured churn confirming next year's even-better metrics. Nothing in the loop pushes back, because the only customers who could disprove the flag — flagged accounts given normal service — are precisely the ones the cost-saving policy eliminated.

The same structure recurs wherever predictions route people into different treatment: collections (aggressive handling of predicted defaulters strains the very customers who might have paid), credit-limit cuts, priority staffing. Formally it's a self-fulfilling prophecy — the act of forecasting changes the system being forecast. The only clean instrument is randomization at the point of action, an intervention holdout, because it recreates for a small slice the world the evaluation pretends still exists.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant CM as Churn model
    participant ST as Service tier
    participant CU as Customers
    loop Every scoring cycle
        CM->>ST: Flags at-risk accounts
        ST->>CU: Flagged accounts routed<br/>to light-touch service
        Note over CU: Service degrades for people<br/>who would have stayed (silent)
        CU->>CM: Flagged accounts churn<br/>at triple the rate
    end
    Note over CM: Model looks validated -<br/>more accounts flagged and<br/>downgraded next cycle
    Note over CM,CU: A year later nobody can say<br/>how much churn was predicted<br/>and how much was caused

Solution — where the guardrails go:

flowchart TD
    A["Churn model flags at-risk<br/>accounts"] --> B["Random intervention<br/>holdout: flagged but<br/>treated normally"]
    A --> C["Remaining flagged accounts<br/>receive the treatment"]
    B --> D["Prediction quality<br/>measured on the holdout"]
    C --> E["Treatment effect measured<br/>against the holdout"]
    D -.-> M["Honest read: does the<br/>model predict churn?"]
    E -.-> N["Separate read: does light<br/>touch cause churn?"]

What good looks like

The lesson

When predictions trigger interventions, the outcome data is contaminated by the treatment — you can no longer evaluate the model on it naively. Hold out a random slice from the intervention (treat them normally), measure the model there, and measure the treatment's effect separately. Prediction and action must be evaluated as different questions.

Named concepts: self-fulfilling predictions, treatment contamination, intervention holdouts, causal evaluation.

Real-world echoes

Transfer

Which of our models trigger actions that could alter the very outcome they predict? Does any of them have an intervention holdout today?

Make it a ticket: List every model whose output triggers a treatment; for each, record whether an intervention holdout exists and its size. File a ticket to add one wherever the answer is no.

Related