Drill 75 — The TTL That Was Too Long

Theme: Networking & DNS · Format: 20-min drill · Difficulty: core

The story

A payments platform runs active-passive across two regions, and the failover plan is one line in the runbook: if region A's health checks fail, point api.pay.example.com at region B. It has been rehearsed on a whiteboard and never in anger.

One Sunday, region A's database tier degrades and error rates climb. Ops declares failover and updates the DNS A record to region B's load balancer within four minutes — clean, by the book. Region B is healthy and nearly idle, waiting for the traffic.

The traffic never really comes. Hours pass and a large share of requests keep landing on the degraded region A. Customers keep seeing errors long after the incident channel says "failed over." Someone finally checks the record: its TTL is 86400 — twenty-four hours — set once, years ago, and never revisited. Every caching resolver at every ISP and corporate network, and every client's own cache, is still handing out region A's address.

⏸ Stop here and ask the room: region B was healthy and the DNS was changed in four minutes. Why did customers keep hitting the broken region for the rest of the day?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

What does a TTL actually control — and who obeys it?

Listen for: the TTL tells resolvers and clients how long to cache the answer; you don't control third-party resolvers, so a 24h TTL means up to 24h of stale routing. Propagation is not instant and you can't invalidate caches you don't own.
Weak signal: "just flush DNS" — you cannot flush the world's resolvers; flushing your own machine proves nothing about clients.

Is DNS even the right failover mechanism?

Listen for: DNS failover is slow and cache-dependent; a health-checked global load balancer / anycast keeps the client-facing address constant and shifts traffic at the routing layer in seconds. DNS is a coarse, slow tool for fast failover.
Weak signal: "just lower the TTL and we're done" — better, but resolvers clamp/ignore very low TTLs and long-lived connections still pin to the old region.

What should the TTL have been, and what's the tradeoff?

Listen for: failover-critical records get a low TTL (30–60s), accepting more DNS query volume for faster propagation; pre-lower TTLs before planned maintenance. There's a real speed-vs-load tradeoff.
Weak signal: "TTL 0" — many resolvers clamp it upward; it's neither free nor obeyed.

The reveal

The failover "worked" in the console in four minutes and did not work for users for most of the day — because the observable that matters is where requests land, not what the record says. The team dropped failover-record TTLs to 60 seconds and, more importantly, moved critical endpoints behind a health-checked global load balancer so the client-facing address never changes and cutover happens at the routing layer in seconds. They added external synthetic probes from several networks to measure the actual traffic shift, so "failover complete" now means observed, not edited.

Why this happens — the mechanics

DNS is a distributed cache you don't own. A resolver caches your answer for up to its TTL, and you cannot invalidate caches on machines you don't control — so DNS-based failover is bounded below by a TTL you had to choose in advance, and further by resolvers that clamp very low values upward. The console showing the new record is not the same as traffic having moved; the only trustworthy signal is where requests actually land, measured from outside your network.

Relying on DNS TTL for fast failover confuses "I changed the config" with "the world sees the change." It's the green-console-versus-reality gap of the silent-failure drills, wearing a networking costume — and the fix is the same: change the mechanism so the guarantee is structural (routing-layer failover), and measure the effect externally instead of trusting the edit.

Diagrams

Problem — how the failure unfolds:

timeline
    title Failover that didn't fail over
    section Sunday
        12.00 : Region A DB degrades, errors climb
        12.04 : Ops updates DNS A record to region B
        12.04-18.00 : Resolvers still serve cached region A (TTL 24h) : Users keep hitting the broken region
    section Monday
        Overnight : Caches slowly expire, traffic finally shifts
        AM : Record TTL discovered - set to 86400 years ago

Solution — where the guardrail goes:

flowchart TD
    A["Failover needed"] --> B["Health-checked global LB /<br/>anycast: address stays constant"]
    B --> C["Traffic shifts at routing<br/>layer in seconds"]
    A --> D["Failover DNS records: TTL 60s,<br/>pre-lowered before maintenance"]
    C --> E["External synthetics measure<br/>ACTUAL traffic shift"]
    D --> E
    E --> F["'Failed over' = observed, not edited"]

What good looks like

Failover-critical DNS records carry a low TTL (30–60s), set and verified in advance — not discovered at incident time.
Failover happens at the routing layer (health-checked global load balancer, anycast, or mesh) so the client-facing address is stable and cutover is seconds.
Pre-lower TTLs ahead of planned maintenance windows.
A DNS record inventory with owner + TTL, reviewed; no 24h TTLs on anything failover-critical.
External synthetic probes from multiple networks measure where traffic actually lands, so "failover complete" means observed shift.
Long-lived connections are drained on failover; clients re-resolve.

The lesson

DNS is a cache you don't own; its TTL is the floor on how fast you can fail over, and you set that floor long before the incident. Never confuse "I updated the record" with "traffic moved" — measure the shift from outside.

Named concepts: DNS TTL, propagation, health-checked failover, external synthetic monitoring.

Real-world echoes

DNS-TTL failover incidents are a recurring post-incident finding: teams that rely on DNS changes for regional failover routinely discover that stale caches extend the outage far past the config change. It's a large part of why CDNs and global load balancers move failover off DNS and onto anycast/health-checked routing.

Transfer

For each of your failover plans: what is the TTL on the record you'd change, and have you ever measured how long traffic actually takes to move? "We'd update DNS" is not a failover time.

Make it a ticket: inventory failover-critical DNS records, set their TTLs to 60s, move the critical path behind health-checked routing, and add an external synthetic that measures real traffic shift during a game-day failover.

Drill 26 — The Failback That Split the Brain (failover done wrong the other way).
Drill 6 — The Health Check That Checked Nothing (what "healthy" must actually prove).
Pattern #131 — Health-Check Design (liveness vs readiness).