Drill 54 — The Big Bang

Theme: Migrations & Releases · Format: 20-min drill — also works as a 60-min tabletop · Difficulty: intro

The story

The legacy decisioning system is replaced over one heroic weekend: old system off Friday at 6 p.m., data migrated Saturday and Sunday, new system live Monday morning. By Tuesday the old servers are wiped and decommissioned — "no going back keeps us honest," says the program lead, and everyone nods. Wednesday afternoon, an underwriter flags something odd: applications she'd expect to decline are getting approved. The root cause is subtle — the old system truncated the debt-to-income (DTI — monthly debt payments divided by monthly income) ratio to two decimal places, the new one rounds half-up — and hundreds of borderline applications have now been decided differently. There is no old system to compare against, and nothing to roll back to.

⏸ Stop here and ask the room: the control group has been deleted. How do you even find out which decisions changed?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Running it as the 60-min tabletop: after the reveal, hand the room the remediation problem — reconstruct the old system's decisions with no old system — and let them discover for themselves how little evidence survives a wiped control group. Assign roles (program lead, underwriter, compliance, engineering) and make them draft the customer letter.

"What would a parallel run have shown, and how long would it have needed to run to catch a borderline-only difference?"

Listen for: both systems decide every application and a daily diff report targets zero — the DTI rounding difference only fires on borderline ratios, so the run needs weeks of real application mix, ideally a full business cycle, before the rare cases have flowed through both.

Weak signal: "We'd have replayed a test set through both systems." A curated test set contains the cases someone already thought of — the borderline mix is precisely what nobody thinks of, and only production traffic supplies it.

"What did 'no going back keeps us honest' actually purchase?"

Listen for: it purchased nothing — it deleted the control group and the evidence, and dressed the deletion up as discipline. Commitment is built with checkpoints and gates, not by burning the boats. Someone should name it: irreversibility is a cost, never a virtue.

Weak signal: agreement that it "focused the team." Push back — focused on what? A team cannot be focused onto correctness it has no way to measure.

"What conditions make it safe to decommission the old path?"

Listen for: written pass/fail criteria — diffs zero or individually explained across a full business cycle, the new path surviving month-end and quarter-end, a named owner signing off, and decommission scheduled as the last step of the migration, never a milestone in the middle.

Weak signal: calendar answers — "after 30 days" with no measurement attached. Time proves nothing without a diff count; thirty quiet days of unmeasured divergence is exactly this story.

The reveal

The team spends weeks reconstructing what the old system would have decided — from memory, an out-of-date functional spec, and screenshots in old support tickets. For a chunk of the borderline cases the reconstruction is an educated guess, and the remediation letters have to say so in careful language.

Why this happens — the mechanics

Big-bang cutovers are almost never chosen for engineering reasons — they're chosen by the calendar. The old system's decommission date is on a slide months before the new system works, the licence renewal or the data-centre exit is booked, and every week of overlap is a visible line item: two sets of servers, two support rotas, two licence bills. Deadline pressure concentrates all of the migration's risk into a single weekend precisely because spreading it out looks expensive — the cost of parallel running shows up on an invoice, while the cost of having no control group shows up only when it's too late to buy one.

The "no going back" rhetoric survives because it flatters everyone. It sounds like decisiveness, it saves money immediately, and it converts a hard engineering question — how will we know the new system decides the same way? — into a motivational one. But rollback options don't disappear at a single moment; they decay. The instant the new system starts making decisions the old one never saw, back-migration becomes its own migration; a week later the data models have drifted; a month later the people who knew the old system have been reassigned. Wiping the servers on Tuesday just made the decay rate infinite.

The deepest trap is epistemic: a migration changes the very system you would use to detect the change. The new system's dashboards were green all week — because they measure the new system against itself. Only a comparison against the old behaviour could reveal that "correct-looking" and "same as before" had diverged, and that comparison requires the control group the program lead deleted. This is why the theme insists a migration is an experiment: an experiment that destroys its control group before the results are in hasn't finished early — it has stopped being an experiment.

Diagrams

Problem — how the failure unfolds:

timeline
    title The big-bang week
    Friday 6 pm : Old system switched off
    Weekend : Data migrated to new system
    Monday : New system live : DTI rounds differently (silent) : Borderline apps flip (silent)
    Tuesday : Old servers wiped : No going back keeps us honest
    Wednesday : Underwriter flags odd approvals : No control group left : Nothing to roll back to

Solution — where the guardrails go:

flowchart TD
    A["New system ready"] --> B["Parallel run: both systems<br/>decide every application"]
    B --> C["Daily decision-diff<br/>report, target is zero"]
    C -.-> D["DTI rounding difference<br/>surfaces in week one"]
    D --> E["Fix shipped, parallel run<br/>continues"]
    E --> F["Diffs zero or explained<br/>across a full business<br/>cycle"]
    F --> G["Reversible cutover: new<br/>system primary, old path<br/>kept executable"]
    G -.-> H["Any regression: switch<br/>back the same day"]
    G --> I["Decommission only after<br/>the new path survives a<br/>full cycle"]

What good looks like

A parallel-run diff report as a first-class deliverable — every application gets decided by both systems; a nightly job emits one row per divergence (application ID, old decision, new decision, diff category) and the count is on the team's dashboard, target zero.
A disposition rule for every diff — each nonzero row is either a bug (fix and re-run) or an intended change (documented, with sign-off from whoever owns the decision policy). "Probably rounding" is not a disposition.
A cutover-gate document written before dates are announced — pass/fail criteria such as "diffs zero or explained for one full business cycle, including month-end," plus the named person with authority to say "not yet."
Reversible cutover mechanics — traffic moves behind a router flag, not a decommission; the old path stays deployed and executable, with a weekly heartbeat transaction proving it still runs.
Decision logs from both systems retained side by side — so "which decisions changed?" is a query over stored outputs, not an archaeology project through support-ticket screenshots.
A decommission checklist that is the last page of the runbook — old path survives a full business cycle as the fallback, then a search proves nothing references it, then and only then are servers reclaimed.
Overlap costs budgeted up front — the parallel-run months (double infrastructure, diff-triage time) are in the migration plan and its funding, so nobody is tempted to buy the schedule back by shooting the control group.

The lesson

Run the two systems in parallel before cutover: both decide every application, and every diff must be zero or individually explained — for weeks, not hours, so a full mix of application types flows through both. Keep the old path executable, not decommissioned, until the new one has survived a full business cycle. Deleting the control group before the experiment ends isn't honesty; it's destroying the evidence.

Named concepts: parallel run / shadow mode, reversible cutover, decision reconciliation, migration as experiment.

Real-world echoes

TSB Bank, 2018 — the UK bank moved off its former parent's banking platform in one big-bang weekend migration. Post-cutover chaos locked roughly 1.9 million customers out of online and mobile banking, disruption ran for weeks, the CEO resigned, and regulators later fined TSB £48.65m. This drill's exact shape, at national scale.
HealthCare.gov, 2013 — a big-bang go-live with no meaningful load rehearsal: the site effectively collapsed on launch day, a handful of enrolments succeeded, and a months-long rescue effort followed. What "the first real test is production" costs.

Transfer

Name our next planned cutover. What is its control group, how long does it stay alive, and who has the authority to say "not yet"?

Make it a ticket: For the next cutover on the roadmap, write the cutover-gate document — parallel-run duration, diff-report owner, pass/fail criteria, and the named "not yet" authority — and attach it to the migration epic before any go-live date is announced.

Drill 55 — The Dual-Write Drift — the same missing verification, on the write path instead of the decision path.
Drill 57 — The Rollback That Couldn't — destroying the way back inside one release instead of one weekend.
Drill 32 — The Backup That Wouldn't Restore (Theme 5 — People & Process): the same reflex — a fallback you've never exercised is a rumor, whether it's a backup or an old system "we could always turn back on."