Drill 57 — The Rollback That Couldn't

Theme: Migrations & Releases · Format: 20-min drill · Difficulty: advanced

The story

A release bundles an application deploy with a schema change: a legacy column is rewritten into a new format and then dropped, all in one PR (pull request), one migration, one release. The reviewer praises the tidiness — no dangling halfway states, no follow-up tickets. Within hours a serious bug surfaces in the new application version. The rollback plan says "redeploy the previous version," but the previous version reads the column that no longer exists — rollback is structurally impossible. The team fixes forward under fire; it takes 14 hours.

⏸ Stop here and ask the room: the rollback plan was written down and reviewed. Why was it a lie?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

"What exactly made this release irreversible — the code, the schema change, or the coupling?"

Listen for: the coupling — the code alone rolls back with a redeploy, and even the schema change would have been survivable if the old column had still existed. Bundling a destructive schema step with the app deploy is what welded the two together. Sharpest answers add the time dimension: the release destroyed the state its own rollback depended on.

Weak signal: "The schema change — never drop columns." Too blunt. Columns do eventually get dropped; the discipline is when and with what — a drop weeks later, in its own release, is fine. Blaming the drop itself misses the sequencing lesson.

"What order of operations keeps every step independently reversible?"

Listen for: expand-contract walked in order — add the new column alongside the old; ship code reading both formats; migrate the data; ship code writing only the new format; drop the old column in a later release once nothing references it. The test someone should articulate: at every step, the previous app version still runs against the current schema.

Weak signal: a vague "do it in stages." How many stages, and what makes a stage boundary safe? If they can't state the invariant — schema always compatible with both the current and previous code version — the stages are just a slower version of the same coupling.

"What's the standing rule about destructive schema changes and app deploys?"

Listen for: a rule crisp enough to enforce mechanically — destructive schema changes (drops, renames, type-narrowing) never ship in the same release as application code; the destructive step always ships last, and alone, after a soak period. Bonus: someone proposes the CI (continuous integration) check that blocks the combination automatically.

Weak signal: "We'd flag it in review." This release was reviewed — and praised. A rule that lives in reviewer judgment loses to tidiness aesthetics every time; it has to live in tooling or in a checklist with teeth.

The reveal

The coupling was chosen deliberately, for tidiness — "one PR, one release" — and everyone involved considered it good engineering hygiene. Tidiness cost the team its rollback option at exactly the moment the option was worth 14 hours of outage.

Why this happens — the mechanics

The trap here is that the dangerous choice looks like the disciplined one. Review culture rewards atomic changes: one PR that tells a complete story, no halfway states, no follow-up tickets that might rot in the backlog. Those instincts are right for code, where "atomic" means revertable in one step — and exactly wrong at the code-schema boundary, where the database carries state that a git revert cannot restore. The reviewer praised tidiness because on every axis the review process measures, tidiness is quality. Nobody's checklist measured reversibility.

Rollback options don't announce their own death — they decay silently, one coupling decision at a time. The rollback plan said "redeploy the previous version" and was written against the code diff, implicitly assuming the world the previous version needs would still exist. Destructive operations are the special case that breaks the assumption: a drop is the one step whose undo requires information that is simply gone. Every other step in this release was reversible on its own; the drop, welded to the deploy, converted an ordinary app bug into a structural trap. And no tool flagged the moment the option died — rollback plans are prose, and prose doesn't fail CI.

Expand-contract feels bureaucratic precisely because its cost is visible and its benefit is invisible: four small boring releases instead of one satisfying one, plus a follow-up ticket, plus weeks of an "ugly" duplicated column. The team pays that overhead on every migration; the payoff arrives only on the release where something breaks — at which point rolling back one small step costs minutes, not 14 hours of fixing forward under fire. Like insurance, it's only ever a waste in the timelines where nothing goes wrong.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant RL as Release
    participant SC as Schema
    participant BG as App bug
    participant RB as Rollback attempt
    RL->>SC: One PR - app deploy plus<br/>rewrite and drop column
    Note over SC: Old column gone -<br/>rollback path destroyed<br/>(silent)
    Note over RL: Reviewer praises the<br/>tidiness
    BG->>RL: Serious bug surfaces<br/>within hours
    RL->>RB: Plan says redeploy the<br/>previous version
    RB--xSC: Old code reads the<br/>dropped column
    Note over RB: Rollback structurally<br/>impossible
    Note over RL,RB: Fix-forward under fire -<br/>14 hours

Solution — where the guardrails go:

flowchart TD
    A["Release 1: add new column,<br/>keep the old one"] --> B["Release 2: code reads<br/>both, writes both"]
    B --> C["Backfill migrates old data<br/>to the new format"]
    C -.-> V["Reconciliation verifies<br/>both columns agree"]
    V -.-> AL["Divergence blocks the next<br/>stage"]
    C --> D["Release 3: code writes new<br/>format only"]
    D --> E["Release 4, weeks later:<br/>drop the old column"]
    D -.-> R["Bug in any release: roll<br/>back that release alone"]
    E --> F["Destructive step ships<br/>last, and alone"]

What good looks like

• Expand-contract written as a numbered release plan — Release 1: add new column, keep old. Release 2: code reads both, writes both. Release 3 (after backfill and reconciliation): code writes new only. Release 4, weeks later: drop the old column. Each release is a separate deploy with its own rollback.
• The N−1 compatibility invariant enforced, not assumed — every schema version must run correctly under both the current and the previous application version; CI runs the previous version's test suite against the migrated schema before any release ships.
• A migration linter in CI that fails any changeset combining a destructive operation (DROP, RENAME, type-narrowing) with application code — and flags destructive statements for a mandatory soak delay after the last code reference disappears.
• "We undo this by ___" as a required field in the release template — one line per step, filled in before review. A blank line, or "fix forward," blocks the release the way a failing test would.
• Rollback rehearsed, not just written — the staging pipeline actually rolls each release back after deploying it; a rollback that has never been executed is a rumor with a checkbox.
• Reconciliation as a stage gate — between "writes switched to new format" and "drop the old column," a diff job proves old and new columns agree; nonzero divergence freezes the contract step.
• A standing calendar rule for contracts — destructive steps ship in their own release, in a quiet window, no sooner than N weeks after Release 3, with a code search proving nothing references the doomed column.

The lesson

Use the expand-contract pattern (also called parallel change): expand — add the new column and ship code that reads both formats; migrate the data; ship code that writes only the new format; then contract — drop the old column in a later release, once nothing references it. Every step is small, boring, and independently reversible. Never couple a destructive schema change to an app deploy; the destructive step always ships last, and alone.

Named concepts: expand-contract (parallel change), rollback-safety, decoupled deploys, fix-forward readiness.

Real-world echoes

• Knight Capital, 2012 — a deploy reached only seven of eight servers; the eighth kept running old code, and a reused feature flag meant the old code interpreted new orders through defunct logic. Roughly $440 million evaporated in about 45 minutes and the firm barely survived. The rhyme with this drill: old and new versions will coexist during every release — safety depends on designing for that coexistence instead of assuming it away.

Transfer

Look at our next release that touches both code and schema. For each step, complete the sentence "we undo this by ___" — any blank is the drill.

Make it a ticket: Add the "we undo this by ___" field to our release template and write the CI rule that blocks destructive schema statements from shipping in the same changeset as application code — with the destructive-step soak delay documented as the override path.

Related

• Drill 56 — The Lock at 10 A.M. — the mechanics half of safe schema change; this drill is the sequencing half.
• Drill 54 — The Big Bang — the same option-destruction at system scale: nothing to roll back to, by choice.
• Drill 58 — The Flag That Forked Reality — what the coexistence period turns into when the contract step never comes.