Drill 55 — The Dual-Write Drift

Theme: Migrations & Releases · Format: 20-min drill · Difficulty: core

The story

A months-long store migration uses the textbook pattern: the application writes every record to both the old database and the new one. The new-store write is fire-and-forget — wrapped in a catch block that swallows errors, "so the migration can never hurt production." A serialization bug quietly fails about 2% of writes on the new store only; the old store, still serving all reads, is perfect, so nothing anywhere looks wrong. Cutover day goes smoothly, dashboards green, congratulations all round. Then the complaints start: customers are missing months of payment history.

⏸ Stop here and ask the room: dual-write ran for five months. What would have caught this in week one?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

"What does dual-write actually guarantee if nothing verifies the two stores agree?"

Listen for: nothing — dual-write is a mechanism for keeping two stores in sync, not evidence that they are. Someone should land on the theme's phrase or its equivalent: without verification, consistency is a rumor. Bonus points for spotting that the catch block converts every failure into silence by design.

Weak signal: "It guarantees the new store gets the data, minus rare errors." That's the trap restated — 2% for five months is not rare, and "minus errors nobody counts" is precisely the hole the customers fell through.

"Where should reconciliation live, how often should it run, and what happens when it finds a nonzero diff?"

Listen for: a standing job, independent of the write path, diffing the stores from day one of the migration — daily at minimum — publishing the divergence count where the team sees it, and alerting on anything nonzero. Nonzero is a stop-ship that triggers fix-plus-backfill, not a ticket.

Weak signal: "We'd spot-check some records before cutover." A one-off sample at the end catches nothing that happened in month two — and a 2% failure rate needs a full diff, not a spot check, to size the backfill.

"What should the cutover gate have required in writing?"

Listen for: a written pass/fail criterion — "reconciliation has read zero for N consecutive days" — signed before the cutover date exists. The gate is proof of agreement, not the absence of complaints; green dashboards on the old store prove nothing about the new one.

Weak signal: "Sign-off from the leads." Sign-off on what evidence? A gate without a measurement is a meeting, and this team's meeting would have passed — everything looked fine.

The reveal

The reconciliation job that would have caught it — diff the two stores, report divergence — was on the backlog the whole time. The ticket was written in week one and deprioritized every sprint since, with the same justification each time: "dual-write is working." It would have found the 2% on its first run.

Why this happens — the mechanics

The catch block is the give-away: it was written to protect production, and it worked — by making the new store unverifiable. That's the structural asymmetry of every dual-write. Errors on the old store hurt customers today, so they get retries, alerts, and on-call attention. Errors on the new store hurt customers after cutover, months away, so the safest-looking engineering choice is to swallow them. The failure isn't sloppy code; it's that the write path's incentives all point toward silence exactly where silence is most expensive.

Then the silence gets read as evidence. "Dual-write is working" was technically true every single sprint — the writes were being attempted, production was unharmed, dashboards were green. Nobody was lying; they were measuring the wrong store. This is how the reconciliation ticket loses every prioritization contest for five months: it produces no feature, its absence produces no incident (yet), and the system it would vindicate looks fine without it. Verification work is always in that position — its value is invisible right up until the day it's retroactively priceless.

And the debt compounds quietly. In week one, the diff is a day of missing writes — a trivial backfill. By month five it's millions of records, some of which can no longer be reconstructed because the events that produced them have aged out of upstream logs. Reconciliation-from-day-one isn't just earlier detection; it keeps the repair cheap while it's still cheap. The cutover gate is where all of this becomes enforceable: written in advance, it converts "nobody has complained" — which this migration passed — into "the stores provably agree," which it would have failed on the first run.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant APP as App
    participant OLD as Old store
    participant NEW as New store
    participant CU as Customers
    APP->>OLD: Every write lands -<br/>old store serves all reads
    APP--xNEW: 2% of writes fail -<br/>serialization bug (silent)
    Note over APP: Catch block swallows<br/>the errors (silent)
    Note over OLD: Reads look perfect -<br/>dashboards green
    Note over APP,NEW: Five months of quiet drift -<br/>no job diffs the stores
    APP->>NEW: Cutover - new store is<br/>now source of truth
    CU->>NEW: Where did months of my<br/>payment history go?
    Note over NEW,CU: Found by complaints,<br/>not by reconciliation

Solution — where the guardrails go:

flowchart TD
    A["App writes to old and new<br/>stores"] --> B["Reconciliation job diffs<br/>the stores from day one"]
    B -.-> C["Divergence count published<br/>daily"]
    C -.-> AL["Alert fires on any nonzero<br/>divergence"]
    AL --> D["Serialization bug found in<br/>week one"]
    D --> E["Bug fixed, failed writes<br/>backfilled"]
    E --> F["Diff-until-zero: 30<br/>consecutive clean days"]
    F --> G["Cutover gate: zero<br/>divergence, in writing"]
    G --> H["Cutover proceeds on proof,<br/>not silence"]

What good looks like

A reconciliation job specified like a product — keyed diff of both stores (primary key plus a content hash per record), running nightly from day one of the migration, emitting three numbers: missing-in-new, missing-in-old, content-mismatch.
Tiered checks for cost control — cheap row counts and per-partition checksums hourly; the full deep diff nightly; a complete historical sweep weekly. Divergence localizes to a partition before anyone greps.
The divergence count on the team dashboard, next to the sprint burndown — published daily, with an alert on any nonzero value. A number the team sees every morning cannot be deprioritized into oblivion.
New-store write failures logged loudly even though they don't block — the catch block may protect the request path, but it must increment a metric and emit the failed payload to a dead-letter queue (a holding area for failed messages) so every miss is replayable.
A backfill runbook written before it's needed — how to replay from the dead-letter queue or upstream source, how to verify the repair, and how long upstream events remain available (that retention window is the real deadline for detection).
The cutover gate in writing — "reconciliation reads zero for 30 consecutive days" — agreed and signed before a cutover date is announced, so the date bends to the evidence and not the reverse.
Reconciliation kept alive after cutover, direction reversed — for the weeks the old store remains the fallback, the same job proves the new source of truth isn't drifting from it.

The lesson

Dual-write without continuous reconciliation is a rumor of consistency. Run diff-until-zero jobs from day one of the migration, publish the divergence count where the team sees it daily, and alert on anything nonzero. Nonzero divergence is a stop-ship: the cutover gate is "reconciliation has read zero for N consecutive days," not "nobody has complained."

Named concepts: dual-write hazards, reconciliation-until-zero, silent write failures, cutover gates.

Transfer

Where are we currently writing the same data to two places? Show the job that proves they agree — and if the answer is a backlog ticket, that's the finding.

Make it a ticket: Inventory every dual-write in our systems this sprint; for each one without a running reconciliation job, create the job spec (key, hash, cadence, alert threshold) and schedule its first run — the diff report from that first run is the acceptance criterion.

Drill 54 — The Big Bang — the same missing verification at cutover scale: proof, not silence, is the gate.
Drill 58 — The Flag That Forked Reality — two paths diverging without a diff job, in code instead of data.
Drill 13 — The Day That Happened Twice (Theme 2 — Data Disasters): the reconciliation rhyme — two sources of truth that nobody diffs until the disagreement becomes an incident.