Drill 78 — The DNS That Cached a Failure

Theme: Networking & DNS · Format: 20-min drill · Difficulty: core

The story

A rates service at a trading firm prices FX orders by calling a market-data vendor at rates.vendor.example.com. It is a boring dependency — a few thousand calls a minute, a healthy connection pool, a p99 under forty milliseconds. It has never been the thing that pages anyone.

At 09:14 the vendor has a ninety-second control-plane blip and their authoritative name servers briefly return SERVFAIL for the rates hostname. The firm's caching resolver does exactly what the RFC says: it caches the failure. The vendor's SOA minimum is 1800, so the resolver negatively caches SERVFAIL for thirty minutes. At 09:16 the vendor is fully recovered — their status page is green, their API is answering, other customers are fine.

The rates service is not fine. Every pricing call now dies before it leaves the box, with EAI_AGAIN / "temporary failure in name resolution." The connection pool has no host to connect to, so it never even opens a socket. On-call sees a wall of DNS errors, checks the vendor status page — green — and concludes it must be a transient network thing that will clear itself. It does not clear. For a full half hour after the vendor recovered, the firm cannot price a single order against that feed, and the desk is quoting stale.

⏸ Stop here and ask the room: the vendor recovered in ninety seconds and their DNS is answering normally again. Why does our service keep failing to resolve the name for the next thirty minutes?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

What exactly got cached — and for how long?

• Listen for: the resolver cached the negative answer (SERVFAIL), not a good one. Per RFC 2308 the negative-cache lifetime is bounded by the zone's SOA minimum (here 1800s) and the resolver's own max-ncache-ttl. Until that timer expires the resolver answers from cache and never re-queries the now-healthy vendor.
• Weak signal: "DNS was down" — it wasn't, it recovered in ninety seconds; the failure that outlived the outage is our cache, not their server.

Why does the recovery not heal us automatically?

• Listen for: nothing triggers a re-query. A cache hit is a cache hit whether the cached value is an address or a failure; the resolver won't look upstream again until the TTL expires. The caller retries, but retry re-runs the same cached lookup, so more attempts just re-read the same SERVFAIL faster.
• Weak signal: "just retry" — retrying against a cached negative answer changes nothing; you have to invalidate or bypass the cache, not hammer it.

Where should the guardrail live — and what number would you set?

• Listen for: cap the negative-cache TTL low (30–60s) so a stale failure self-heals fast; on a resolution failure, retry once with a fresh lookup that bypasses the negative cache; run a health check that forces re-resolution; and alarm on resolve-failure rate persisting past the vendor's recovery. The fix is to make failures cheap to cache, not to trust the vendor to never blip.
• Weak signal: "raise a ticket with the vendor" — their side is already healthy; the thirty-minute tail is entirely inside our resolver's config.

The reveal

The vendor outage was ninety seconds. Our outage was thirty-one minutes — ninety seconds of theirs plus the full 1800-second negative-cache TTL of ours, minus the overlap. The extra half hour was self-inflicted: a resolver configured to trust a downstream's SOA minimum, caching a transient failure for as long as the vendor happened to say. The team capped max-ncache-ttl to 30 seconds, added a fallback path that re-resolves with the cache bypassed after any resolution error, and gave the readiness probe a fresh lookup so a stuck negative entry trips the health check instead of the trading desk. The next vendor blip cleared in well under a minute and never paged.

Why this happens — the mechanics

Caches are symmetric about success and failure: a resolver that caches "here is the address" also caches "there is no address / SERVFAIL," and both are served from memory until their TTL runs out. Negative caching exists for a good reason — it stops a stampede of clients from re-querying for a name that genuinely doesn't resolve — but its lifetime is governed by the downstream zone's SOA minimum (RFC 2308) unless you cap it locally. So a ninety-second provider blip can hand your resolver a thirty-minute instruction to keep failing, and your resolver obeys it.

The trap is that the failure is invisible at the layer everyone checks. The vendor's status is green because the vendor is healthy; the outage now lives entirely in your resolver's cache, which no dashboard you're looking at exposes. Retries make it worse, not better: each retry re-reads the cached SERVFAIL, so the service looks like it's "trying hard" while doing nothing. This is the same green-console-versus-reality gap as the TTL failover drill — a cache you don't think about, holding a stale answer past the event that should have cleared it.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant App as Rates Service
    participant R as Caching Resolver
    participant D as Vendor DNS
    participant P as Vendor API
    App->>R: resolve rates host
    R--xD: query during<br/>vendor blip
    D--xR: SERVFAIL returned
    Note over R: negative-cache SERVFAIL<br/>for 30 min (SOA min)
    Note over P: vendor recovers<br/>in 90 seconds
    App->>R: resolve rates host
    R--xApp: cached SERVFAIL<br/>no re-query
    Note over App,P: pricing keeps failing<br/>long after recovery

Solution — where the guardrails go:

flowchart TD
    A["Resolve fails<br/>(SERVFAIL)"] --> B["Cap negative-cache TTL<br/>to 30 seconds"]
    A --> C["On failure, re-resolve<br/>bypassing the cache"]
    A --> D["Readiness probe forces<br/>a fresh lookup"]
    B --> G["Stale failure clears in<br/>30s, not 30 min"]
    C --> G
    D --> G
    G -.-> AL["Alert: resolve-fail rate<br/>persists past recovery"]

What good looks like

• Cap the negative-cache TTL locally. Set the resolver's max-ncache-ttl (and equivalents) to 30–60s so a downstream's SOA minimum can never pin a transient failure in your cache for minutes. Never inherit a stranger's 1800s.
• Bypass the cache on failure. On any resolution error, retry once with a fresh lookup that skips the negative cache (or flush that entry), so recovery is picked up in seconds rather than on TTL expiry.
• Health-check the resolution, not just the socket. The readiness probe does a fresh lookup of the real dependency hostname every 10s; a stuck negative entry fails the probe and pulls the instance from rotation instead of silently failing user calls.
• Alarm on the tail, not the blip. A DNS resolve-failure rate SLI: warn at >1% for 60s, page when resolve failures persist past the point the dependency is observably healthy again — that gap is the signature of a cached failure.
• Bound client-side DNS caching. HTTP clients and connection pools re-resolve on a short TTL (JVM networkaddress.cache.ttl and equivalents ≤ 30s) so a pool doesn't pin a dead answer for the life of the process.
• Keep your own SOA minimum sane. On zones you own, set the SOA minimum to ≤ 300s so you never hand your callers a long negative-cache instruction.

The lesson

A cache remembers failures as faithfully as successes, and a negative-cache TTL is an instruction to keep failing that you often inherit from someone else. Cap it, bypass it on error, and alarm on failures that outlive the outage — because the tail is yours, not the vendor's.

Named concepts: negative caching, SERVFAIL, negative-cache TTL, cache-bypass on failure.

Real-world echoes

• Resolver negative-cache (RFC 2308) outages are a recurring post-incident finding: a downstream returns SERVFAIL for seconds, and callers keep failing for minutes because their resolver negatively cached the answer for the zone's SOA minimum. The lesson repeats across providers — the length of your outage is set by your negative-cache TTL, not by how long the dependency was actually down.

Transfer

Pick your most critical downstream dependency: what is the negative-cache TTL its DNS could hand your resolver right now, and would a ninety-second blip on their side become a thirty-minute outage on yours? "DNS just works" is not an answer — name the cap.

Make it a ticket: cap max-ncache-ttl to 30s on the service resolvers, add a cache-bypassing re-resolve on resolution failure, give the readiness probe a fresh lookup, and add a resolve-failure SLI that pages when failures persist past dependency recovery.

Related

• Drill 75 — The TTL That Was Too Long (the positive-cache sibling: stale good answers outliving a failover).
• Drill 6 — The Health Check That Checked Nothing (why a probe must exercise real resolution, not a cached socket).
• Drill 21 — The Thundering Herd (why negative caching exists — and how retries against a cached failure still stampede).
• Pattern #131 — Health-Check Design (liveness vs readiness).