Drill 21 — The Thundering Herd

Theme: Scale & Cloud · Format: 20-min drill · Difficulty: intro

The story

To "align with business hours," the nightly feature-cache flush is moved from 3 a.m. to 9 a.m. — a one-line cron change, no review needed. At 9:00:01 the cache is cold exactly when traffic peaks. Every request misses and hits the feature store. Clients time out and retry — three attempts each, no jitter. The feature store, now serving 4× load from retries alone, saturates. Total scoring outage: 45 minutes.

⏸ Stop here and ask the room: walk the amplification chain. Where could it have been broken?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Which individual components misbehaved? (Trick: none.)

Listen for: the recognition that every component behaved to spec — the cron fired on schedule, the cache flushed as configured, the clients retried as designed, the feature store served until it couldn't. The failure lives in the composition: three sensible local policies formed one global amplifier. Strong answers name the chain explicitly — cold cache × peak traffic × synchronized retries — and point at seams (jitter, retry budget, flush timing) rather than boxes.

Weak signal: the room hunts for a culprit component — "the cache was misconfigured," "the feature store was under-provisioned." Blaming a box means the group will fix a box, then rebuild the same amplifier around the bigger box.

Why did retries make it worse?

Listen for: retries are extra load aimed at the component least able to absorb it — and without jitter they arrive in synchronized waves, because every client that timed out in the same second retries in the same second. Three attempts each turns one request into up to four; the store must now recover under 4× load, which it never will. Bonus points for the exact phrase: retries convert partial failure into total failure.

Weak signal: "the timeout was just too short" or "we need more retries with longer waits." Tuning timeouts changes when the wave lands, not whether it lands — the missing pieces are decorrelation (jitter) and a cap (budget, breaker).

What makes a schedule change "harmless"?

Listen for: nothing does — a schedule change relocates a stress event in time, and time is load-bearing. The 3 a.m. slot was a deliberate design decision that had become invisible; moving it required asking "what does 9 a.m. coincide with?" Strong answers connect this to the config-as-deploy rule: same review, same canary, same post-change verification as code.

Weak signal: "cron changes should just get a second approver." Approval without the what-does-this-coincide-with question is a rubber stamp — the checklist matters more than the signature.

The reveal

Every layer did its job: cache flushed on schedule, clients retried on failure, autoscaler scaled. The composition failed — three sensible policies formed one amplifier.

Why this happens — the mechanics

Local correctness does not compose into global correctness. Each policy in this story was written against assumptions the other policies quietly violated: the flush schedule assumed low traffic, the cache assumed misses were rare, the retry logic assumed failures were independent and transient. Stack them and you get an emergent machine nobody designed — a load amplifier with the cache as its trigger and the retry policy as its gain stage. No component owner could see the whole machine, because the machine only exists at composition time.

The retry storm has a precise mechanism: synchronization. Deterministic retry delays act like a metronome — every client that failed at 9:00:01 retries at 9:00:03, fails together, retries together. The feature store never sees average load; it sees walls of correlated demand, each wall taller than the last because new requests keep arriving on top of the retries. Jitter (randomizing each delay) decorrelates the clients and flattens the walls; a retry budget caps how tall any wall can get; a circuit breaker removes the wall entirely once the store is clearly struggling. Without those three, the retry policy is a positive-feedback loop: more failures → more retries → more load → more failures.

Staging never shows this class of failure because it requires a coincidence at scale: a cold cache, peak concurrency, and thousands of clients failing in the same second. Staging has none of these — traffic is thin, caches are small, and a full retry storm needs production-shaped volume to ignite. The only places this machine can be observed are production and a deliberately engineered load test, which is why the amplification factor has to be computed on paper (multiply the retry counts per layer) before it is ever measured in anger.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant CR as Cron
    participant CA as Cache
    participant CL as Clients
    participant FS as Feature store
    CR->>CA: Flush moved from 3am<br/>to 9am (silent)
    Note over CA: Cold exactly at<br/>traffic peak
    CL->>CA: Peak requests arrive<br/>at 9am sharp
    CA--xCL: Every request misses
    loop Retry amplification - 3 attempts, no jitter
        CL->>FS: Misses hammer the<br/>feature store
        FS--xCL: Requests time out
        CL->>FS: Clients retry in lockstep
    end
    Note over FS: Serving 4x load from<br/>retries alone
    Note over FS: Saturates
    Note over CL,FS: 45-minute scoring outage

Solution — where the guardrails go:

flowchart TD
    A["Cron change reviewed as a<br/>deploy"] --> B["Flush kept off peak and<br/>staggered"]
    B --> C["Cache warms before traffic<br/>ramps"]
    C --> D["Residual misses reach the<br/>feature store"]
    D --> E["Exponential backoff with<br/>jitter spreads retries"]
    E --> F["Retry budget caps total<br/>amplification"]
    F --> G["Circuit breaker stops<br/>hammering a struggling<br/>store"]
    G --> H["Load shedding protects<br/>core scoring"]
    D -.-> I["Alarm fires on cache miss-<br/>rate spike"]

What good looks like

Full jitter on every retry: sleep = random(0, min(cap, base × 2^attempt)) — the delay is drawn from the whole interval, not added as a ±10% wobble, so failed clients spread out instead of marching in step.
A retry budget per client and per service: retries may be at most ~10% of request volume (token-bucket enforced); when the budget is spent, failures return immediately instead of amplifying.
Circuit breakers with explicit thresholds: open after ≥50% failures in a rolling 10-second window (minimum 20 requests), stay open for 30 seconds, then allow a single half-open probe — numbers written down, not vibes.
Staggered cache expiry: per-entry TTL (time to live) randomized ±10–20% and flushes rolled in batches, so no single instant leaves the whole cache cold.
Cache warming as part of the flush job: pre-load the hottest N keys before the old entries are dropped, and alarm on cache miss-rate spikes so a cold cache pages someone before the store saturates.
Cron and schedule changes go through deploy review with one mandatory question on the template: what does the new time coincide with? (traffic peak, batch windows, other crons, market open).
A worst-case amplification number in the runbook: for each critical path, the product of retry counts at every layer — computed on paper, verified in a load test, and re-checked whenever a retry policy changes.

The lesson

Retries convert partial failure into total failure unless bounded: exponential backoff with jitter, retry budgets, circuit breakers that stop hammering a struggling dependency, and load shedding to protect the core. And "harmless" schedule/config changes are deploys (Drill 4's rule) — this one moved a stress event onto peak traffic.

Named concepts: thundering herd, backoff + jitter, circuit breakers, load shedding, cascading failure.

Real-world echoes

AWS DynamoDB, September 2015 — a brief network disruption caused storage servers to time out fetching membership metadata; they all retried at once, and the retry storm overloaded the metadata service faster than it could recover, cascading into an hours-long us-east-1 outage that took dependent AWS services down with it. A network blip supplied the spark; the retry amplifier supplied the fire.

Transfer

Trace one request path end to end: multiply the retry counts at each layer. What's our worst-case amplification factor?

Make it a ticket: Audit retry policy on our top three request paths — record per-layer retry counts, add full jitter and a retry budget wherever they're missing, and write the resulting worst-case amplification factor into the on-call runbook.

Drill 24 — The Autoscaler That Fed the Fire — the same positive-feedback shape, with an autoscaler in the role the retry policy plays here.
Drill 22 — The Quota Wall — the other way peak traffic finds a limit nobody modeled.
Drill 4 — The Stale Score (Theme 1 — Silent Failures): the "config changes are deploys" rule this lesson leans on.