Drill 80 — The Retry Storm

Theme: Networking & DNS · Format: 20-min drill · Difficulty: core

The story

A card-processing platform runs a fleet of about four hundred application pods that all talk to one internal ledger service on the hot path: every authorization writes a ledger entry before it returns an approval to the merchant. Normal ledger latency is nine milliseconds, and the client library retries failed calls up to three times, because transient blips happen and a retry almost always succeeds. This has been true for years.

At 14:02 the ledger service does a routine garbage-collection pause — about two seconds where it answers nothing. Two seconds is nothing; the retry logic exists precisely for this. But every one of the four hundred pods has requests in flight, every one of those requests times out at almost the same instant, and every one of them retries immediately, together. The ledger, just finishing its pause, is now hit with the normal load plus the entire backlog of first retries plus the new arriving traffic — several times its usual rate, all at once. It falls behind, so more calls time out, so more clients retry.

The GC pause ends at 14:02. The ledger service is healthy hardware with no bug. And yet at 14:20 it is still pinned at the redline, still timing out, still down. Engineers restart it; it's flattened again within seconds of coming back. The thing that started the incident is long over, and the incident will not end.

⏸ Stop here and ask the room: the original cause — a two-second GC pause — cleared in seconds. So why is the service still down eighteen minutes later, and why does restarting it not help?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

What is actually generating the load at 14:20 — name it.

Listen for: the retries themselves are now the load. The system has entered a metastable failure: a sustaining feedback loop (timeout → immediate synchronized retry → more load → more timeouts) that keeps itself alive after the trigger is gone. Load is a multiple of real demand because each real request is on the wire three-plus times.
Weak signal: "the ledger must have a bug" or "just give it more memory" — treating the trigger as the cause and missing that the load is self-generated.

Why does restarting the ledger not fix it?

Listen for: a restart doesn't remove the load — the four hundred clients are still hammering, so the fresh instance is saturated on arrival. You have to break the loop from the client side (stop retrying so hard) or shed load at the server side, not just add capacity into a positive-feedback loop.
Weak signal: "scale the ledger horizontally" — often makes it worse: more capacity invites more retries to land, and every replica gets the synchronized wave.

The retries were a best practice. What made them dangerous?

Listen for: they retry immediately and in lockstep with no backoff and no jitter, and there is no cap on what fraction of traffic may be retries. Retries are safe only when they back off, spread out in time, and are budgeted so they can never exceed a small share of real traffic.
Weak signal: "remove the retries" — throws away real resilience; the fix is disciplined retries, not none.

The reveal

The trigger was a two-second GC pause; the outage was a retry storm the platform inflicted on itself. Because every client retried immediately and in sync, real demand got multiplied roughly five-fold and stayed there — a metastable failure that outlived its cause by twenty minutes and survived restarts. The fix was not more ledger capacity. It was exponential backoff with jitter so retries spread out instead of stacking, a retry budget (token bucket) capping retries at ten percent of traffic, a circuit breaker that trips to fail-fast when the ledger is clearly down, and load shedding on the ledger so it protects a healthy core instead of dying under everything. Load fell back to real demand and the system recovered on its own.

Why this happens — the mechanics

A retry is a good idea in isolation and a bomb in aggregate. When many independent clients experience the same upstream blip at the same instant and all retry on the same fixed schedule, their retries are phase-locked: they arrive together as a wave. That wave is larger than normal load (it carries the backlog plus new traffic), so it induces more failures, which schedule the next synchronized wave. This is a positive feedback loop, and once its gain exceeds one, the system has two stable states — healthy, and pinned — and a small shove moves it from the first to the second, where it stays. That is the definition of a metastable failure: the sustaining effect (self-generated retry load) is decoupled from the trigger (a momentary pause), so removing the trigger, or restarting the victim, does nothing.

The organizational trap is that every ingredient was a best practice. Retries improve reliability. A short timeout is prudent. The client library's defaults — retry three times, right away — are what most teams ship. Nobody added the counter-pressure: no jitter to de-synchronize the wave, no budget to cap retries as a fraction of traffic, no breaker to stop calling a service that is plainly down, no shedding so the server sheds instead of collapses. The danger lives in the interaction, so no single code review catches it; it only appears at fleet scale during a real blip.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant C as Client fleet
    participant G as API gateway
    participant L as Ledger service
    C->>L: authorize, write entry
    Note over L: 2s GC pause<br/>answers nothing
    L--xC: timeouts across fleet
    Note over C: all clients retry<br/>at the same instant
    C->>L: synchronized retries
    C->>L: plus new traffic
    Note over L: load about 5x normal<br/>service stays pinned
    L--xC: still timing out
    Note over C,L: cause cleared, outage<br/>continues (metastable)

Solution — where the guardrails go:

flowchart TD
    A["Downstream blip"] --> B["Client wants to retry"]
    B --> G1["Exponential backoff<br/>+ jitter (spread the wave)"]
    B --> G2["Retry budget: token<br/>bucket, cap 10% of calls"]
    B --> G3["Circuit breaker: open at<br/>50% errors over 10s"]
    A --> S["Recovering ledger"]
    S --> G4["Load shedding: drop low<br/>priority over 80% CPU"]
    G1 -.-> AL["Retry load stays bounded<br/>service recovers on its own"]
    G2 -.-> AL
    G3 -.-> AL
    G4 -.-> AL

What good looks like

Exponential backoff with full jitter on every retry: base 100ms, doubling per attempt, capped at ~10s, with the delay chosen uniformly at random in [0, backoff] so two clients never retry in lockstep.
A retry budget (token bucket): retries may not exceed ~10% of successful requests; when the bucket empties the client stops retrying and fails fast. This caps the multiplier — a blip can add at most 10% load, never 400%.
A circuit breaker per downstream: trip to open when the error rate exceeds ~50% over a 10-second window (min ~20 requests), fail fast for a cooldown (e.g. 5s), then half-open to probe with a single request before closing. A dead dependency gets zero retries, not three.
Server-side load shedding: the ledger sheds low-priority and already-timed-out work above ~80% CPU (or a bounded concurrency limit / LIFO queue), so it serves a healthy core instead of collapsing under the full wave.
A retry-amplification SLI: alarm when retries / total_requests exceeds ~20% for 1 minute — that ratio climbing is the earliest signal a storm is forming.
A game-day that injects a 2–5s stall into a hot dependency and asserts the fleet recovers on its own after the stall clears, with retry load returning to baseline within a defined budget.

The lesson

A retry is a good idea in isolation and a bomb in aggregate; without backoff, jitter, and a budget, synchronized retries become a self-sustaining load that outlives the blip that started it. Design retries to spread out and to be capped, and give the server a way to shed load — so a momentary failure decays instead of latching.

Named concepts: metastable failure, exponential backoff with jitter, retry budgets, circuit breakers, load shedding.

Real-world echoes

Retry-storm / metastable-failure post-mortems are a recurring class in large-scale operations write-ups (notably from AWS and Google SRE practice): a brief dependency stall triggers synchronized client retries that multiply load and keep the recovering service pinned long after the trigger clears, with restarts and added capacity failing to help. The published remedies are exactly this drill's — jittered backoff, retry budgets, circuit breakers, and load shedding — because they attack the sustaining loop rather than the trigger.

Transfer

Pick your busiest internal dependency: if it stalls for two seconds during peak, do your clients retry immediately and in lockstep, and is there any cap on what fraction of traffic can be retries? If you can't name the backoff, the jitter, and the budget, assume you have a retry storm waiting.

Make it a ticket: audit the retry config on your top three hot-path clients; add full-jitter exponential backoff and a 10% retry budget, wrap each downstream in a circuit breaker, and run a game-day that stalls one dependency for 3 seconds to prove the fleet self-recovers.

Drill 75 — The TTL That Was Too Long (sibling in Networking & DNS: another case where the config looked fine but the fleet-level behavior was the real failure).
Drill 21 — The Thundering Herd (synchronized clients hitting one resource at the same instant — the same phase-locking, a different trigger).
Drill 24 — The Autoscaler That Fed the Fire (a control loop that adds load into a struggling system, another positive-feedback / metastable trap).
Pattern #110 — Backoff, Jitter & Retry Budgets (the reusable fix).