Drill 91 — The Shadow Nobody Watched

Theme: Model Lifecycle & MLOps · Format: 20-min drill · Difficulty: core

The story

A card issuer runs a real-time authorization model. Every time a cardholder taps at a terminal, the model scores the transaction and the decision engine approves or declines in under 100 ms — tens of millions of times a day. The model is retrained monthly on the last thirteen months of authorizations and swapped in through the same one-click deploy the team has used for two years: the new artifact replaces the old one, and from its first second it owns 100% of authorizations.

This month's retrain looks great offline. AUC is up two points on the holdout, the confusion matrix is clean, the calibration plot is tight. The MLOps checklist is all green — unit tests pass, the model card is filled in, the artifact is signed and promoted. At 09:00 on a Tuesday the new version goes live to every authorization on the network.

By 09:40 the decline rate on one slice — contactless purchases under $50 — has quietly doubled. The cause is small and ordinary: an upstream feature-pipeline change shifted the units on avg_ticket_30d, and the freshly trained model, having learned on the shifted feature, now reads routine small purchases as anomalous and declines them. But the decline rate the team watches is one blended number on a dashboard that refreshes every fifteen minutes, and small-ticket declines are noisy at the best of times. Nobody's phone rings. Call-center hold times climb first. By the time a fraud-ops analyst ties the complaint spike back to the 09:00 deploy, the model has spent ninety minutes declining good small-ticket transactions across the entire card base.

⏸ Stop here and ask the room: the model passed every offline gate and shipped clean. Why did a regression a 1% canary would have caught in an hour instead hit every cardholder at once?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

The model passed every offline check. So what actually failed — the model, or the way it shipped?

• Listen for: the deploy strategy. A retrain regression is ordinary and expected; what turned it into a book-wide incident is that the new version went to 100% of live decisions in one irreversible step, with no shadow or canary stage. Offline metrics answer "is this model good on the data I already have"; they cannot catch a regression that only appears on the live feature distribution. The safety belongs in the rollout, not the eval.
• Weak signal: "add more holdout data / better offline tests." You cannot offline-test your way out of a live-only regression — the feature shift happened in production, after the eval. The move is to limit exposure while you learn, not to eval harder.

If you could add ONE stage between "artifact signed" and "100% of traffic," what would it be and what would it watch?

• Listen for: a canary — route 1% of live authorizations to the new model, and gate promotion on live metrics cut by segment (decline rate, approval rate). Even better, a shadow stage first: score live traffic with the new model but keep acting on the champion, and compare the two score streams before any real decision rides on it. Name the metric and the gate, not just "a canary."
• Weak signal: "deploy to staging first" or "replay yesterday's traffic." Staging isn't live traffic and a replay won't surface a feature-pipeline shift that is happening right now on today's flow.

The decline rate did move on a dashboard. Why didn't that stop the rollout?

• Listen for: a dashboard a human watches is not a gate. Nothing was wired to halt the deploy automatically; the signal was blended (a 2% move network-wide was a 40% move in one slice), lagged fifteen minutes, and noisy. A gate needs a specific metric, a segment cut, a threshold, and an automatic action — halt and roll back — with no human in the loop.
• Weak signal: "watch the dashboard more closely" or "add an alert." An alert that pages a human still burns ten-plus minutes at 100% exposure; by then the whole book is affected. The exposure is the problem, not the alerting latency.

The regression hit every cardholder at once. What deploy input would have bounded that?

• Listen for: blast radius as an explicit deploy parameter — the maximum share of decisions a new version may own before its gates pass — defaulting to 1%, never 100%, and raised only with sign-off. Pair it with a business-metric guardrail (approval rate, authorization value) that auto-halts and rolls back on breach. Blast radius should be a number you choose, not the default of "all of it."
• Weak signal: "roll back faster." Faster rollback helps, but if you started at 100% the damage is already smeared across the entire book before anyone reacts.

The reveal

The retrain was fine in the way that mattered least and broken in the one that mattered most: a strong offline model fed one shifted feature on live traffic. What made it an incident was the deploy. The team moved authorization scoring onto a progressive-delivery pipeline. Now every new version first runs in shadow — it scores live authorizations for a full day while the champion keeps making the actual decisions, and the two score streams are compared. Then it takes real traffic in stages — 1% → 5% → 25% → 100% — and each promotion is gated on live metrics cut by segment. A business-metric guardrail on approval rate auto-halts and rolls back if the number leaves its band. And blast radius — the maximum share of decisions a version may own before its gates pass — is now an explicit input to every deploy, defaulting to 1%. Re-run against the same regression, the 1% canary trips the small-ticket decline gate in under an hour, with 1% of cardholders touched instead of all of them.

Why this happens — the mechanics

A retrained model is a change to production behavior, but an offline eval is only a sample of the past. The eval and the deploy answer two different questions: the eval asks "is this model good on the data I have," and the deploy asks "what happens when this model touches live traffic." A regression that only manifests on the live feature distribution or the live traffic mix — a shifted feature, a changed population, a new merchant category — is invisible to every offline gate and shows up only when real requests flow through the model. Shipping straight to 100% collapses those two questions into a single irreversible step, so the first time anyone learns the deploy's answer is with the entire book already exposed.

The organizational root is that model deploys inherited the "swap the artifact" pattern from a simpler era, and it kept working because most retrains were fine — the rare bad one just hadn't happened yet. Progressive delivery is standard discipline for application code (feature flags, canaries, staged rollouts), but models often skip it because "it's just a new weights file." Yet the blast radius of a bad weights file is every decision it makes, and for a model on the hot path that is the whole book in one step. The missing idea is simple: a model version deserves the same staged, gated, reversible rollout you would give any risky code change — because that is exactly what it is.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant RT as Retrain
    participant DP as Deploy
    participant AM as Auth model
    participant CH as Cardholders
    participant OP as Ops
    Note over RT: Offline gates green<br/>AUC up 2 points
    RT->>DP: Sign new artifact
    DP->>AM: Promote to 100%<br/>at 9am Tuesday
    Note over AM: Shifted feature units<br/>small buys look odd
    AM--xCH: Decline good<br/>small-ticket buys
    Note over AM,OP: Blended decline rate<br/>15-min lag, noisy
    CH->>OP: Call-center complaints
    Note over RT,OP: Found by humans<br/>90 min, whole book

Solution — where the guardrails go:

flowchart TD
    A["New model version"] --> B["Shadow: score live,<br/>champion still acts"]
    B --> C["Canary 1% of<br/>authorizations"]
    C --> G1["Metric gate: decline rate<br/>by segment vs champion"]
    G1 -.-> HALT["Auto-halt + rollback<br/>(blast radius = 1%)"]
    C --> D["Promote 1 to 5 to<br/>25 to 100%"]
    D --> G2["Business guardrail:<br/>approval-rate band"]
    G2 -.-> HALT
    HALT --> H["Rollback in minutes,<br/>1% of book touched"]

What good looks like

• A shadow stage before any real decision rides on it: every new version scores 100% of live traffic for at least 24 hours while the champion makes the actual decisions; promote only if the decline-rate delta stays inside a band — say < 0.5pp overall and < 1pp on any top-20 segment — versus the champion on the same requests.
• Canary with automatic gates: route 1% → 5% → 25% → 100%, holding each stage long enough to accumulate signal (≥ 30 minutes or ≥ 10,000 decisions, whichever is later), and gate each promotion on live metrics cut by segment. A failing gate stops promotion automatically — no human in the loop, no override without sign-off.
• A business-metric guardrail with auto-halt: wire the deploy directly to approval rate, authorization value, and decline rate; define a band (e.g. approval rate within ±1pp of the trailing 7-day baseline) and auto-halt and roll back on breach. The metric that pays the bills is the backstop of last resort, not a proxy.
• Blast radius as an explicit deploy input: make "max share of decisions this version may own before its gates pass" a required parameter, default 1%, require sign-off to raise. 100% is never the starting exposure for a hot-path model.
• Segment-level monitoring, never just blended: cut decline/approval by the dimensions that hide in a blended number — transaction size, channel, merchant category, issuer BIN. A blended 2% move can be a 40% move in one slice.
• An instant, rehearsed rollback: keep the champion warm so rollback is one click and takes seconds, and game-day a deliberately bad canary quarterly to prove the gate halts promotion and the rollback actually fires.

The lesson

A model version is a change to production behavior, not just a new file — ship it the way you would ship risky code: shadow it, canary it behind automatic gates, and make its blast radius a number you choose, never a default of 100%. The offline score tells you the model is good; only staged exposure tells you it is safe.

Named concepts: shadow deployment, canary release, automated rollout gates, blast-radius control.

Real-world echoes

• Knight Capital, August 1, 2012. New trading code went live to the full production order flow with no canary and no staged exposure; a latent flag re-activated dead logic, and the firm fired millions of erroneous orders, losing roughly $440 million in about 45 minutes. It is the same class of failure wearing a trading costume: a change reached 100% of a hot-path decision system in one step, with no progressive rollout and no automatic halt, so an ordinary defect became a firm-ending, book-wide event before a human could react.

Transfer

For your highest-stakes model on the hot path, what share of decisions does a freshly promoted version own in its first minute — 1%, or 100%? If the honest answer is 100%, you are one bad retrain away from a book-wide incident, and "the retrains are usually fine" is not a control.

Make it a ticket: put your top hot-path model behind a shadow-then-canary pipeline with a 1% default blast radius and an approval-rate auto-halt gate; then game-day a deliberately bad canary and prove the gate stops promotion before it ever reaches 5% exposure.

Related

• Drill 87 — Train-Serve Skew, theme 15 — the sibling: the exact class of live-only feature regression a canary is built to catch before it spreads.
• Drill 88 — The Drift Nobody Watched, theme 15 — degradation that arrives slowly; here the same kind of harm arrives all at once at 100%.
• Drill 16 — Upgrade That Broke Nothing Visibly, theme 3 — a model swap whose regression was invisible offline and only surfaced on live traffic.
• Pattern #132 — Progressive Delivery / Canary Releases — the reusable rollout discipline this drill is an instance of.