Pattern — Leader Election

The problem it solves

Some work must have exactly one owner at a time — one region that accepts writes, one node that runs the cron, one replica promoted to primary. The failure shape is what happens when two owners exist at once: in Drill 26, a failback points DNS back at the recovered primary, but a slice of stale-DNS clients keeps writing to the DR region for twenty minutes. Two "authoritative" copies of a decision ledger diverge, nothing errors — because accepting a write is what a healthy region does — and reconciliation takes days. Routing (DNS, a load balancer) was asked to do the job of consensus, and routing is advisory. Leader election is the technique that makes single-ownership a mechanical invariant instead of a hope.

The pattern

Elect one leader through a quorum (a majority of an odd-sized cluster agreeing), give that leader a lease — a lock with a TTL it must heartbeat to keep — and stamp every privileged action with a fencing token: a monotonically increasing number that goes up by one each time leadership changes hands. The resource being protected (the storage, the ledger) records the highest token it has ever accepted and rejects any write carrying a lower one. So even a "zombie" old leader — paused by a GC stall or partitioned past its lease expiry, still believing it leads — is fenced out at the write path, because the world moved on to a higher token. The lease bounds how long a stale leader can linger; the fencing token guarantees its writes cannot land regardless.

How to apply it

1. Run an odd-sized cluster and elect through a majority. 3 nodes tolerate 1 failure, 5 tolerate 2; even sizes buy nothing and risk split votes. Back the election with a real consensus store — etcd, ZooKeeper, or Consul — never a coin flip between two peers. A 2-node "cluster" cannot form a majority and needs a witness/tiebreaker.
2. Hold leadership as a lease with a TTL, and heartbeat at roughly ⅓ of it. Kubernetes' own leader election ships concrete, battle-tested numbers: LeaseDuration = 15s, RenewDeadline = 10s, RetryPeriod = 2s. The holder renews every 2s and must succeed within 10s or it voluntarily stops acting; a challenger may claim the lease only after the full 15s with no renewal. Heartbeat interval well below TTL keeps a healthy leader stable; TTL comfortably above your worst GC/scheduling pause keeps a paused leader from being falsely demoted.
3. Attach a fencing token to every privileged write and enforce it at the resource. Each new leadership term increments a monotonic int64 (etcd's mod-revision, ZooKeeper's zxid, or a SELECT … FOR UPDATE sequence). The token travels with the write; storage keeps max_token_seen and does WHERE incoming_token >= max_token_seen — a stale token is rejected, not applied. This is the load-bearing step: a lease alone is a timeout, and timeouts are advisory exactly like the DNS TTL that failed in Drill 26.
4. Make the inactive side reject, not route away. The non-leader answers privileged writes with an explicit error and a redirect hint — never a silent 200. In Drill 26's terms: the DR region must refuse the stale-DNS write at the write path, so the client gets a retryable error and the divergence never forms.
5. Rehearse the handoff. Practice a leadership change (and a failback) as its own game day with a simulated stale-client population, and watch the fencing-rejection counter tick down the zombie tail to zero before you call it done.

flowchart TD
    A["Quorum elects one leader<br/>(majority of 3 or 5 nodes)"] --> B["Leader holds a lease<br/>(TTL 15s, renew every 2s)"]
    B --> C["Every write carries a<br/>fencing token (monotonic int64)"]
    C --> D{"incoming_token >=<br/>max_token_seen?"}
    D -->|"yes (current leader)"| E["Write committed once"]
    D -->|"no (zombie / stale leader)"| F["Write REJECTED at storage,<br/>client gets retryable error"]

When to use it — and when not

Reach for it when a task demands a single writer or single actor and correctness breaks if two run at once: an active/DR write region (Drill 26), a promoted database primary (Patroni), a singleton scheduler, or a partition/shard owner. The cost is a real availability tradeoff: during an election — a lease expiry plus a renewal round — there is a window (up to the ~15s TTL) with no leader and privileged writes paused, and you inherit the operational weight of a consensus quorum. So don't elect a leader when the work is commutative or idempotent and safe to run everywhere — use leaderless replication or CRDTs and skip the coordination entirely. Don't use it when you can shard so each key has a natural sole owner without global election. And never ship the lease without the fencing token: a TTL you merely trust is Drill 26's DNS mistake in a new costume — a stalled leader can outlive its lease and still write.

Signals it's working

• leader_count per role holds at exactly 1. Page on 0 (no leader — writes are paused, an election is stuck) and on 2 (split-brain — the invariant is already broken). This is the single most important SLI.
• A fencing-rejection counter you can watch. In steady state it's 0; during a failover/failback it rises and then returns to 0 — that's the guard catching zombie writes exactly as designed. A rejection that never clears means a node still thinks it leads.
• Leadership-transition rate stays low. Frequent flapping means RenewDeadline is too tight against real GC/scheduling pauses — widen the lease before it demotes healthy leaders.
• Renewal latency < RenewDeadline (< 10s) with margin, and a game day that injects a GC pause longer than the heartbeat but shorter than the TTL shows the leader survives — while one longer than the TTL shows a clean, fenced handoff.

Seen in the wild

• Kubernetes — kube-controller-manager and kube-scheduler elect a leader via a Lease object with the 15s / 10s / 2s defaults; the client-go/tools/leaderelection package is the reference implementation.
• etcd (Raft), ZooKeeper (Zab), Consul, and Google's Chubby — the consensus/lock services everyone else builds on; ZooKeeper's ephemeral znodes and etcd leases are the canonical election primitive.
• Patroni (Postgres HA), Redis Sentinel, and Kafka's controller (KRaft) all elect a single primary/controller and fence the old one so a demoted node cannot keep committing.
• Martin Kleppmann's fencing-token write-up is the canonical explanation of why a lease without a token at the resource is unsafe.

See it

The ChainScene carries the mechanism — quorum elects one leader → lease (TTL 15s, renew 2s) → fencing token on every write → token >= max_seen? → stale token rejected — and the MetricLineScene shows the one SLI that matters: leader_count pinned at 1, paging on 0 or 2.

Named concepts: leader election, leases (TTL + heartbeat), fencing tokens, quorum/consensus