Pattern — Quorum / Consensus

The problem it solves

Two nodes both believe they are in charge, and both accept writes. In Drill 26 — The Failback That Split the Brain, a failover moves all writes to the DR region; when the primary recovers and the team fails back, some clients' DNS caches ignore the low TTL and keep writing to DR for twenty minutes while the rest write to the primary. Now two copies of the decision ledger exist, each committing locally, each "authoritative" — nothing errors, because accepting a write is what a healthy region does. Reconciling twenty minutes of divergence takes days. Routing (DNS, load balancers) answers "where should new traffic go?" — it says nothing about who is allowed to commit. That question needs a mechanism that routing cannot provide.

The pattern

Let a majority decide. A quorum is any subset of more than half the nodes; a consensus protocol (Raft, Paxos, Viewstamped Replication) uses that majority to agree on exactly one leader and one totally-ordered log. The invariant that kills split-brain: a write is not committed until a majority has durably accepted it, and two disjoint majorities cannot exist in one cluster — so at most one leader can make progress at a time. When the network partitions, the side holding the majority keeps a leader; the minority side cannot reach quorum, so it steps down and refuses writes instead of silently diverging. Consensus turns "hope the stale side stops" into "the stale side mathematically cannot commit."

How to apply it

1. Run an odd cluster size; quorum = ⌊N/2⌋ + 1. Use 3 nodes (quorum 2, tolerates 1 failure) for most services, 5 (quorum 3, tolerates 2) where the coordinator is critical. Never run an even count: 4 nodes still need 3 to agree — you pay for a node that buys zero extra fault tolerance and doubles the tie surface. Spread the members across 3 availability zones so no single AZ holds a majority.
2. Elect a leader with leases and heartbeats, sized ~10:1. The leader holds a lease (e.g. 15 s) and renews it at 1/3 of TTL (every 5 s). Followers expect a heartbeat every ~100 ms and start an election after a randomized election timeout of ~1 s (etcd/Raft defaults) — roughly ten missed heartbeats before a challenge, so a brief blip doesn't trigger a needless election. Randomizing the timeout (e.g. 1000–1500 ms per node) prevents split votes.
3. Fence the write path with a monotonic token. Every write carries the current leader's fencing token — a monotonically increasing int64 bumped on each leadership change. Storage rejects any token lower than the highest it has seen, so a partitioned old leader whose lease expired gets an explicit error, not a silent commit. This is the mechanical guarantee Drill 26 was missing: the stale-DNS slice hitting DR would be rejected at the ledger, not accepted.
4. For quorum reads/writes of data, size W + R > N. In a Dynamo-style replicated store with N = 3, pick W = 2, R = 2 so the read and write sets always overlap by at least one node — every read sees the latest committed write. Reserve the consensus group for leadership and metadata; let bulk data replicate under it.

flowchart TD
    A["Client write + fencing token"] --> B["Leader proposes to the cluster"]
    B --> C{"Majority accept?<br/>(2 of 3)"}
    C -->|"yes"| D["Commit + ack — token recorded"]
    C -->|"no (minority / partitioned)"| E["Step down, reject write<br/>(retryable error, not silent)"]
    D --> F["Exactly one authoritative log"]
    E --> F

When to use it — and when not

Reach for it wherever exactly one writer must exist and being wrong is catastrophic: leader election, distributed locks, config/metadata stores, cluster membership, and the failover/failback control plane of Drill 26. Do not put raw consensus on a high-throughput or geo-distributed hot path — every commit costs a round trip to a majority (a cross-region quorum can add 100+ ms of write latency), and a 3-node group caps throughput at what one leader can drive. Keep the consensus group small (3–5) and coordinating, not carrying the bulk data. And note the honest limit: quorum guarantees a single writer but not zero downtime — during an election (that ~1 s window) writes pause. If you can tolerate multiple writers (naturally commutative CRDT state, or read-only replicas), you don't need consensus at all.

Signals it's working

• Exactly one leader at any instant — a leader_count gauge that is always 1 (an alarm if it is ever 0 for more than a few seconds, or ever 2).
• Elections are rare and fast — leadership changes converge in seconds, election rate near zero in steady state (a spike means flapping — usually heartbeat/timeout mis-tuned or an overloaded leader).
• Zero committed writes at the inactive side after the freeze timestamp — the Drill 26 gate: failback isn't "done" until the inactive region's write count reads 0 for a defined window.
• The rejected-fencing-token counter is your split-brain smoke alarm — healthy near 0; any sustained rise means a stale writer is being (correctly) fenced. Prove it in a game day: partition the leader and assert the minority refuses writes.

Seen in the wild

• etcd / Kubernetes — the control plane stores all cluster state in a Raft-backed etcd quorum (recommended 3 or 5 members); the API server's leader election guarantees one active controller-manager and scheduler.
• Apache ZooKeeper (ZAB) and HashiCorp Consul / Nomad (Raft) — quorum-based coordination, locks, and leader election for the systems built on top of them.
• Raft-backed databases — CockroachDB, TiKV, YugabyteDB, and MongoDB's replica-set elections all use majority quorum so a partitioned minority cannot accept writes.
• GitHub's 2018 outage (Drill 26's echo) — a 43-second partition let automated failover promote a second MySQL primary; writes landed on both sides precisely because routing, not quorum, was gating write authority.

See it

CardsScene carries the load-bearing config (odd N, quorum ⌊N/2⌋+1, 15 s lease renewed every 5 s, ~100 ms heartbeat / ~1 s election, fencing token), and CompareScene/ChainScene show the tradeoff: majority commits, minority steps down and is fenced — one authoritative log.

Named concepts: quorum, consensus (Raft/Paxos), leader election, fencing tokens, single-writer.