Drill 90 — The Rollback With No Model

Theme: Model Lifecycle & MLOps · Format: 20-min drill · Difficulty: core

The story

A bank's card-fraud service scores every transaction in real time — a few hundred milliseconds to decide approve, challenge, or decline. The model is retrained nightly on the last ninety days of labelled transactions; the training job writes its output to s3://models/fraud/latest.pkl, and the scoring service loads whatever sits at that path on boot and on a rolling hourly refresh. This has run cleanly for two years. Nobody thinks about the artifact path because nobody has ever needed to.

Tuesday's nightly run picks up a feature-pipeline change merged the afternoon before, and the freshly trained model — call it the Tuesday model — goes live at the 06:00 refresh. By 09:30 the fraud-ops queue is on fire: the decline rate on legitimate card-present transactions has roughly doubled, call-centre hold times are climbing, and a payments partner has opened a Sev-2 asking why good customers are being blocked at the terminal. The model hasn't crashed. Latency is flat, error rate is zero, every health check is green. It is simply wrong — too aggressive — and it is declining real customers at scale.

The incident commander makes the obvious, correct call: roll back to yesterday's model. Monday's model was fine; put it back and stop the bleeding while we investigate. An engineer goes to fetch it — and there is nothing to fetch. latest.pkl is Monday's model until 06:00 and Tuesday's after, because the training job overwrites the same key every night. The S3 bucket has no object versioning. The registry stores a single pointer called "current," and it points at latest.pkl. Monday's artifact was overwritten in place six hours ago. There is no previous model anywhere on disk, in the bucket, or in the registry.

The room realizes there is no rollback. The only path back to Monday's behaviour is to retrain Monday's model — but the feature code has changed, the ninety-day training window has rolled forward a day, and the person who knows the old pipeline is asleep. The bleed continues while the team tries to reconstruct a model that existed, and worked, this morning.

⏸ Stop here and ask the room: the fix everyone agrees on is "roll back to yesterday's model." What has to have been true, before the incident, for that sentence to be executable in under five minutes — and which of those things does this team have?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

The decision is "roll back." Why can't they?

• Listen for: rollback is only a real option if the previous artifact still exists and is reachable. This team stores exactly one model — the training job overwrites latest in place, the bucket has no versioning, and the registry points at a mutable path. "Roll back" presumes a thing to roll back to; here the prior model was destroyed by the deploy that replaced it. A model version is data, not code — you cannot git revert a binary you overwrote.
• Weak signal: "just redeploy the old version" or "roll back the code." The code change is not the artifact; even reverting the feature commit doesn't hand you Monday's trained weights, and retraining reproduces neither the same window nor the same model.

The service is 200-OK and green. So what is actually broken, and why did no monitor catch it?

• Listen for: the model regressed on a business metric — legitimate-decline rate — that no infrastructure signal watches. The system is behaving exactly as coded; the weights are just worse. This is the defining shape of an ML failure: correctness lives in the outputs and the outcomes, not in the control flow, so uptime, latency, and error rate all stay green while the model quietly declines good customers.
• Weak signal: "add more uptime monitoring" or "it'll show up in the error logs." A too-aggressive model throws no errors; the only catchable signal is the decision distribution against baseline, which nobody was comparing.

You've retrained under fire and it's back. What do you change so the next bad model is a two-minute rollback, not an outage?

• Listen for: make artifacts immutable and versioned (never overwrite latest), keep the last few known-good models hot so rollback is repointing an alias not rebuilding a binary, write a one-command rollback runbook, and rehearse it in a game-day so it works when it's needed. Rollback has to be a boring, tested operation, not an improvisation invented during a Sev-2.
• Weak signal: "we'll be more careful about what we deploy" or "add a bigger review." Better deploys reduce how often you need to roll back; they do nothing for the day a good-looking model turns out bad, which is when you need the rollback to exist.

The reveal

There was no previous model because every night's training job wrote to the same key and the bucket kept no versions — the deploy that shipped the bad model had already deleted the good one. Recovery took just over three hours: the team pinned the feature code back to Monday's commit, reran the training job against a reconstructed ninety-day window, eyeballed that the decline rate looked sane, and promoted the result — effectively retraining a model that had existed and worked six hours earlier. The permanent fix had nothing to do with the bad model itself: they switched to immutable, versioned artifacts (fraud/2024-06-11T06:00/model.pkl, never overwritten), kept the last three known-good versions loaded and warm in the service, and made rollback a single command that repoints the current alias to a prior version and completes in under two minutes — then rehearsed it. The bad Tuesday model was never the hard part. The hard part was that "put yesterday's model back" turned out to be a sentence the system could not execute.

Why this happens — the mechanics

A trained model is a data artifact, not source code, and it does not live in your version-control system. Code rollback is a solved, reflexive problem — every team can revert a commit and redeploy — so teams unconsciously assume model rollback is the same reflex. It is not. Reverting the training code to yesterday's commit does not give you yesterday's model: retraining reruns against a different data window, non-deterministic ordering, new library versions, and a moved-forward label set, and produces a different model that merely resembles the old one. The only thing that is truly "yesterday's model" is the exact bytes that were serving yesterday — and if the training job overwrote them, or you only ever stored "latest," those bytes no longer exist. Rollback is impossible not because the tooling failed but because the artifact was never retained.

The organizational root is that the training pipeline and the deployment story were built by different reflexes at different times. The training job was written to produce a good model, and "write to latest" is the simplest possible sink — it works perfectly until the first time someone needs the previous one. Deployment safety — versioned artifacts, a registry, staged promotion, one-command rollback — is a separate discipline that nobody added because, for two years, no model was ever bad enough to need it. The failure is a best practice (retrain nightly, always serve the freshest model) applied without the matching safety net (never destroy the model you're replacing). And it is invisible until the exact moment it matters, because a rollback capability you never exercised looks identical to one you have — right up until you reach for it during a Sev-2 and find the shelf empty.

Diagrams

Problem — the deploy that destroyed its own fallback:

sequenceDiagram
    autonumber
    participant TR as Nightly train job
    participant ART as Artifact store<br/>(only "latest")
    participant SVC as Fraud model<br/>service
    participant OPS as On-call
    Note over TR,ART: nightly job OVERWRITES<br/>latest.pkl in place
    TR->>ART: write latest (Tuesday)
    ART->>SVC: serve Tuesday model
    Note over SVC: legit-decline rate 2x,<br/>200 OK, zero errors
    SVC--xOPS: good customers declined
    Note over OPS: decision — roll back<br/>to Monday
    OPS->>ART: fetch previous model
    ART--xOPS: no prior artifact
    Note over OPS,ART: nothing to roll back to —<br/>retrain under fire

Solution — where the guardrails go:

flowchart TD
    A["Train job writes a new<br/>immutable version"] --> B["Registry: promote by<br/>version, never overwrite"]
    B --> C["Keep last N=3 good<br/>models loaded and hot"]
    C -.-> AL["One-command rollback:<br/>repoint current alias"]
    D["Regression detected<br/>(legit-decline 2x baseline)"] -.-> AL
    AL --> E["Quarterly game-day<br/>rehearses the rollback"]
    E -.-> AL

What good looks like

• Immutable, versioned artifacts — every training run writes to a new, content-addressed or timestamped path (fraud/2024-06-11T06:00/model.pkl) and never overwrites an existing one; enable bucket object-versioning so an accidental overwrite is still recoverable. The rule is absolute: a deploy must never destroy the artifact it replaces.
• A model registry as the source of truth — models are promoted by version through stages (staging → production), and "production" is a mutable alias pointing at an immutable version. Rolling back is moving the alias to a prior version, never touching the artifacts themselves.
• Keep the last N known-good models hot — the serving layer preloads the current version plus at least the previous 2–3, so a rollback is an in-memory alias flip in seconds, not an S3 fetch, a cold load, and a warm-up under load. Retain a minimum of the last 3 production versions (and their metadata) for a defined window, e.g. 90 days.
• A one-command rollback runbook — rollback fraud-model --to <version> (or the equivalent registry call) that a single on-call engineer can run without tribal knowledge; it repoints the alias, drains to the prior model, and confirms the switch, targeting a rollback completed in under 5 minutes.
• A business-metric regression alarm — monitor the legitimate-decline rate (and approve/challenge/decline mix) against a rolling baseline and page when it moves beyond a written bound, e.g. decline rate > 1.5× the trailing 7-day median for 15 minutes. This is the signal that triggers the rollback, since uptime and error rate never will.
• A rollback game-day, on a schedule — at least quarterly, deliberately promote a known-bad model in a controlled window and have on-call execute the runbook end to end, measuring the actual time-to-rollback. An untested rollback is a hypothesis, not a capability.

The lesson

A model is a data artifact, not code — you can only roll back to a version you deliberately kept, hot and reachable, and only if you've rehearsed the switch. Never let a deploy overwrite the model it replaces, and prove the rollback works before the incident that needs it.

Named concepts: immutable versioned artifacts, model registry, instant rollback (hot standby), rollback game-day.

Real-world echoes

• No single well-documented public ML incident maps cleanly to this exact failure, so none is cited here. The nearest general rhyme is the industry-wide move to immutable, versioned deployment artifacts (container image digests, model registries such as MLflow) precisely so that "roll back to the last known-good version" is always an executable operation rather than a reconstruction job — the discipline this drill is about.

Transfer

For your highest-stakes model: if it regressed right now and the call were "put the previous version back," could an on-call engineer do it in under five minutes — and does the previous version's exact bytes still exist, hot and reachable? Trace where your training job writes its output and ask what happens to the artifact it replaced.

Make it a ticket: switch the model's training output to immutable versioned paths with bucket versioning on, retain the last 3 production versions hot, write and time a one-command rollback runbook, and schedule a quarterly game-day that promotes a known-bad model and rehearses the rollback to a measured under-5-minute target.

Related

• Drill 87 — Train-Serve Skew, theme 15 — the sibling lifecycle failure where a model looks healthy on every infra metric while quietly producing wrong outputs; here the wrongness triggers a rollback that doesn't exist.
• Drill 88 — The Drift Nobody Watched, theme 15 — the slow cousin: a model degrading over time you'd also want to roll back from, which likewise requires a retained prior version.
• Drill 1 — The Model That Scored Everyone the Same, theme 1 — the same silent-scoring blindness: a green service producing business-broken outputs no infrastructure signal catches.
• Pattern #132 — Immutable Artifacts & Versioned Deploys — the reusable enforcement behind "never overwrite what you're replacing."