Drill 26 — The Failback That Split the Brain

Theme: Scale & Cloud · Format: 20-min drill (also works as a 60-min tabletop) · Difficulty: advanced

The story

A regional incident triggers a clean failover to the DR region — the drill everyone rehearsed. Four hours later the primary recovers, and the team fails back. But some clients' DNS caches ignore the low TTL; a slice of traffic keeps writing to DR for twenty minutes while the rest writes to the primary. Two divergent copies of the decision ledger now exist, both "authoritative."

⏸ Stop here and ask the room: failover was rehearsed; what wasn't?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer. (Tabletop variant: extend the questions to 35 minutes and walk the reconciliation hour by hour — who freezes writes, who diffs the ledgers, who calls the customers.)

Why is failback harder than failover?

Listen for: asymmetry. Failover starts from a dead primary — there's only one live side, so there's nothing to disagree with. Failback starts from two healthy regions, and the whole difficulty is making exactly one of them authoritative while the other stands down — a coordination problem, not a recovery problem. Strong answers add: failover is rehearsed because it's scary, failback is skipped because it feels like "going home" — yet it's the step with the split-brain risk. Sharpest formulation: failback is a second failover, executed against a system that seems fine.

Weak signal: "it's the same procedure in reverse." That's precisely the belief that caused the incident — reversing traffic direction is symmetric; reversing write authority is not, because state has accumulated on the DR side and clients don't all move at once.

What guarantees that only one side accepts writes?

Listen for: the word mechanically. DNS TTL (time to live — how long resolvers may cache the answer) is advisory; resolvers, operating systems, and client connection pools routinely outlive it. The real answers live at the write path: fencing tokens or leases (the active region holds a lease; writes carry its token; storage rejects stale tokens), the database refusing writes outside the active region, or the application rejecting writes when it doesn't hold the lock. Strong answers state the invariant — "a region that is not active cannot commit a write, even if traffic reaches it" — and note that a rejected write is a visible error while an accepted one is silent divergence.

Weak signal: "we set the TTL to 60 seconds" or "the load balancer handles it." Both route most traffic correctly and prove nothing about the slice that ignores them — hoping clients comply is exactly the hope this incident falsified.

How do you reconcile twenty minutes of divergent decisions about real applications?

Listen for: an actual procedure, not a shrug. Freeze both ledgers; diff by business key and idempotency key, not by row; partition the divergence into disjoint classes — decisions made only in one region (merge), the same application decided identically in both (deduplicate), the same application decided differently in both (the painful class: re-decide under a documented policy, with compliance and customer-communication owners in the room, because customers were already notified of outcomes). Strong answers insist the tooling and the decision policy must exist before the divergence — inventing a merge policy during the incident is how twenty minutes becomes days.

Weak signal: "last write wins" or "replay DR's writes onto the primary." Blind replay double-books the deduplicate class and silently overwrites the conflict class — in a decision ledger, that's rewriting decisions customers were already told about.

The reveal

Reconciliation takes days: decisions made in both regions must be diffed, deduplicated, and in a few painful cases re-decided — with customers already notified of outcomes.

Why this happens — the mechanics

Split-brain is what happens when routing is asked to do the job of consensus. DNS (Domain Name System) answers "where should new lookups go?" — it says nothing about clients that cached the old answer, hold open connections, or run resolvers that treat TTLs as suggestions. Convergence is eventual and ragged by design. For read traffic, ragged is fine: a stale reader gets slightly old data. For write traffic, ragged is fatal: two regions each accept a share of the writes, each commits locally, and each is telling the truth as it knows it. Nothing errors, because accepting writes is what a healthy region does — the failure is silent precisely because both halves are working.

The reason this bites at failback rather than failover is state asymmetry. At failover the primary is dead — zero live writers, then one. At failback there are two healthy regions and hours of accumulated DR-side state; the operation must transfer authority, not just traffic, and authority is a distributed-systems property (leases, fencing, quorum) that no amount of DNS configuration provides. Teams rehearse failover because the triggering disaster is vivid; failback has no disaster attached, so it gets improvised — which is how the rehearsed half of the procedure works and the unrehearsed half generates the incident.

A decision ledger makes divergence maximally expensive because decisions have already left the system — customers notified, downstream processes triggered, regulators potentially involved. Reconciliation isn't a data-cleanup task, it's unwinding real-world commitments, which is why RTO (Recovery Time Objective — how fast you're back) gets all the attention while RPO-for-state (Recovery Point Objective — what data you can lose or duplicate) is the number that actually hurt here. And none of this appears in staging: test clients respect TTLs, caches are empty, and the stale-resolver population that produced the twenty-minute tail only exists in the real world's mess of corporate proxies and long-lived connection pools.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant PR as Primary region
    participant DR as DR region
    participant CL as DNS and clients
    participant LG as Ledger
    Note over PR: Regional incident strikes
    CL->>DR: Clean failover - all<br/>writes go to DR
    Note over PR: Recovers four hours later
    CL->>PR: Team fails back - DNS<br/>points to primary
    Note over CL: Some caches ignore the<br/>low TTL (silent)
    CL->>DR: Stale-DNS slice keeps<br/>writing to DR
    PR->>LG: Primary accepts writes
    DR->>LG: DR also accepts<br/>writes (silent)
    Note over LG: Two divergent copies -<br/>both claim authority
    Note over PR,LG: Twenty minutes of<br/>split-brain - days of<br/>reconciliation

Solution — where the guardrails go:

flowchart TD
    A["Failback rehearsed as a<br/>second failover"] --> B["Fencing enforces a single<br/>writer at all times"]
    B --> C["Writes rejected outside<br/>the active region"]
    C --> D["Stale-DNS traffic gets<br/>errors, not silent<br/>acceptance"]
    D --> E["Exactly one authoritative<br/>ledger survives"]
    C -.-> F["Alarm fires on any write<br/>attempt to inactive region"]
    G["Reconciliation tooling<br/>built before divergence"] --> E

What good looks like

• Single-writer enforced at the storage layer: the active region holds a lease; every write carries its fencing token; the ledger rejects any write whose token isn't current — so a stale client gets a retryable error instead of a silent commit.
• Write-rejection outside the active region as default behavior: the inactive region's application tier answers writes with an explicit "not the active region" error and a redirect hint — never a 200.
• An alarm on any write attempt at the inactive region — during failback, that alarm firing is expected and watched: it counts down the stale-DNS tail, and failback isn't declared complete until it reads zero for a defined period.
• A failback runbook with gates, not vibes: freeze writes at DR → confirm replication caught up → transfer the lease → verify zero committed writes at DR since the freeze timestamp → then move traffic. Each gate has an owner and an abort path.
• Failback rehearsed in game days as its own scenario — including a simulated stale-client population that keeps hitting the old region, so the team has seen the alarm and the tail before it's real.
• Idempotency keys on every decision write, so the reconciliation diff can distinguish "same decision recorded twice" from "same application decided twice" mechanically.
• Reconciliation tooling and a signed-off conflict policy built in peacetime: a ledger-diff job by business key, a classification report (merge / dedupe / conflict), and a pre-agreed policy for the conflict class with compliance's name on it.
• RPO for state written down and tested: the maximum tolerable divergence window is a stated number, and the game day measures the actual one against it.

The lesson

Failback is a second failover and needs its own rehearsal. Split-brain prevention is a design property — single-writer enforcement (fencing, leases), write-rejection outside the active region — not a hope about DNS TTLs. And decision systems need reconciliation tooling built before the divergence, because "it can't diverge" is a rumor (Drill 23's lesson, applied to state).

Named concepts: split-brain, fencing/single-writer, RTO/RPO for state (not just service), failback rehearsal.

Real-world echoes

• GitHub, October 21, 2018 — a 43-second network partition between a US East coast datacenter and the rest of GitHub's infrastructure caused automated failover to promote a new MySQL primary; writes existed on both sides of the partition that the other side lacked. Exactly this drill's shape: seconds of split-brain, then roughly 24 hours of degraded service while data was carefully reconciled and restored rather than risk serving inconsistent state.

Transfer

In our failover story, what enforces — mechanically — that two regions can't both accept writes? Who has tested the return trip?

Make it a ticket: Trace one write from client to ledger and name the mechanism that would reject it if this region lost authority mid-flight — if the answer is "DNS," file the fencing work, add the inactive-region write alarm, and schedule a failback (not failover) game day with the gate checklist.

Related

• Drill 23 — "Multi-AZ" — the parent lesson: an unexercised resilience claim is a rumor, whether about topology or about state.
• Drill 24 — The Autoscaler That Fed the Fire — the other drill where the recovery mechanism itself creates the incident and the human move must be rehearsed in advance.
• Drill 4 — The Stale Score (Theme 1 — Silent Failures): TTLs trusted beyond their guarantees, with silent divergence as the price.