Theme 9 — Migrations & Releases
The reflex this theme builds: every migration is an experiment — design the rollback before the rollout.
Why this theme matters
Migrations concentrate risk the way no other engineering activity does. A normal change touches one component; a migration touches everything at once — data, schema, code paths, routing, and the assumptions of every downstream consumer — and it does so under deadline pressure, because migrations are always "almost done" and the old system's decommission date is always already on a slide. Worst of all, the failure modes are asymmetric: the forward path gets planned for months, with runbooks and dry runs and sign-offs, while the backward path gets improvised at 2 a.m. by whoever is awake, using whatever wasn't deleted yet.
The recurring root cause in every drill here is the same: treating a migration as a one-way project instead of a reversible experiment with checkpoints. An experiment keeps its control group alive, measures divergence continuously, and defines in advance what result means "stop." A project has a Gantt chart and a party at the end. The drills that follow are five ways the project mindset burns you — big-bang cutovers, unverified dual-writes, unrehearsed schema changes, releases that destroy their own rollback, and "temporary" flags that fork reality. Abbreviations are expanded on first use; the full reference is glossary.md.
The drills
| # | Drill | One-line hook | Problem diagram | ~Difficulty |
|---|---|---|---|---|
| 54 | The Big Bang | One heroic weekend, old servers wiped by Tuesday — on Wednesday borderline decisions have flipped and there is no control group left to ask. | timeline | intro |
| 55 | The Dual-Write Drift | Dual-write runs for five months with a catch block swallowing 2% of new-store writes — the cutover ships months of missing payment history. | sequence | core |
| 56 | The Lock at 10 A.M. | A migration that took seconds on 10,000 staging rows holds a 40-minute table lock on 400 million production rows — at peak, on a Tuesday. | sequence | core |
| 57 | The Rollback That Couldn't | One tidy release rewrites and drops a column alongside the app deploy — the rollback plan points at code that needs the dropped column. | sequence | advanced |
| 58 | The Flag That Forked Reality | A "temporary" 50% flag outlives its sprint by months — two diverging pipelines, irreproducible bugs, and an incident that takes 3x longer. | timeline | core |
Best practices — the migration & release checklist
What to actually do, distilled from drills 54–58:
- Design the rollback before the rollout. For every migration step, write down "we undo this by ___" before it ships. A step with no answer gets split, redesigned, or moved to the end. An untested rollback plan is a rumor.
- Run old and new in parallel before any cutover — both systems process every input, and every diff must be zero or individually explained. Run for weeks, across a full business cycle, so the rare input mix flows through both.
- Keep the old path executable until the new one has survived a full cycle. Decommissioning is the last step of a migration, never a milestone in the middle. Deleting the control group early is destroying the evidence.
- Never dual-write without reconciliation. Stand up the diff-until-zero job on day one of the migration, publish the divergence count daily, and alert on anything nonzero. Nonzero divergence is a stop-ship, not a backlog item.
- Write cutover gates in advance, as pass/fail criteria — for example "reconciliation reads zero for 30 consecutive days" and "decision diffs explained for a full cycle." The gate is proof, not the absence of complaints.
- Rehearse every migration against production-sized data. A restored backup copy is the perfect rehearsal target, and the exercise doubles as a restore drill (see Theme 1's rule: an alert that has never fired is unverified code — the same is true of a backup that has never been restored). Staging with 10,000 rows validates syntax, not mechanics.
- Use online schema-migration strategies for large tables: add columns as nullable, backfill in small batches, add constraints and defaults last. Measure lock duration in rehearsal; schedule for traffic troughs; write the kill-and-resume plan before starting.
- Follow expand-contract (parallel change) for every schema evolution: expand, migrate, switch writes, and only then contract — with the destructive step in its own release, weeks later. Every step independently reversible.
- Never couple a destructive schema change to an app deploy. "One PR (pull request), one release" tidiness is exactly what converts an app bug into an unrollbackable outage. The drop ships last, and alone.
- Treat migration flags as loans: every flag is born with a deadline, a named owner, a ratchet plan (25%, then 50%, then 100%), and a deletion ticket created the same day. Review all flags weekly; a flag past its deadline pages its owner.
- Log the path taken. While two code paths exist, every request records which one served it — so "which pipeline did the affected requests take?" is a query, not an archaeology project, and incident debugging doesn't triple.
- Delete the old path deliberately. A migration isn't done at 100% traffic; it's done when the old code, the old store, the flag, and the dual-write are gone, and a search proves nothing references them. Until then it's two products.
Also see
- Facilitator guide — how to run drills, and the other ten themes.
- Glossary — every abbreviation, explained in plain language.