Pattern — Retry Budget

The problem it solves

A retry is a good idea in isolation and a bomb in aggregate. When one shared dependency hiccups — a 2-second GC pause on a ledger service called by 400 pods (Drill 80 — The Retry Storm) — every in-flight request times out at the same instant, every client retries immediately and in lockstep, and the recovering service is hit with the backlog plus new traffic plus every retry: real demand multiplied roughly five-fold. That self-generated load induces more timeouts, which schedule the next synchronized wave — a metastable failure that outlives its trigger by twenty minutes and survives restarts. Retries with no ceiling turn a momentary blip into a latched outage. The missing primitive is a hard cap on what fraction of your traffic is allowed to be retries.

The pattern

A retry budget caps retries as a small fraction of successful requests, enforced by a token bucket — not by a per-call retry count. Every logical call may still retry, but only while the bucket has tokens: a failed call spends a token, a successful call earns a fraction of one back, and when the bucket runs dry the client stops retrying and fails fast. This bounds the amplification: a blip can add at most ~10% load, never 400%. The budget is paired with exponential backoff with full jitter so the retries that do fire are spread across time instead of arriving as a phase-locked wave. Together they change retries from an uncapped positive-feedback loop into a bounded, de-synchronized trickle that decays on its own.

How to apply it

1. Full-jitter exponential backoff on every retry. delay = random_between(0, min(cap, base * 2^attempt)) with base = 100ms, cap ≈ 10s. The random_between(0, …) is load-bearing: picking the delay uniformly in [0, backoff] means two clients that failed at the same instant almost never retry at the same instant. Plain exponential backoff without jitter still phase-locks — the jitter is what breaks the wave.
2. A token-bucket retry budget, sized as a ratio not a count. Model it the way gRPC does: maxTokens = 1000, tokenRatio = 0.1. Each failed RPC removes 1 token; each successful RPC adds tokenRatio (0.1) back. Retries are allowed only while tokens > maxTokens / 2. At steady state this holds retries to about 10% of successful traffic — so when a dependency is fully down, retries starve within a few seconds and every extra call fails fast instead of piling on. Envoy exposes the same idea as a retry_budget with budget_percent (default 20%) and min_retry_concurrency (default 3), capping concurrent retries as a percentage of active requests.
3. Cap attempts too, but let the budget win. Keep a small per-call ceiling (max 3 attempts) as a backstop, but the budget is the fleet-level guard — three attempts × 400 clients is still a storm if nothing caps the aggregate.
4. Add the two companion guardrails. A circuit breaker per dependency (open at >50% errors over a 10s window, min ~20 requests, ~5s cooldown, single half-open probe) so a dead dependency gets zero retries, not budgeted ones. And server-side load shedding on the dependency (shed low-priority / already-timed-out work above ~80% CPU, or via a bounded concurrency limit / LIFO queue) so the recovering service protects a healthy core instead of collapsing under the wave.

flowchart TD
    A["Call fails / times out"] --> B{"Budget tokens<br/>> maxTokens / 2?"}
    B -->|"yes"| C["Spend 1 token, wait<br/>random(0, min(10s, 100ms x 2^n))"]
    B -->|"no (bucket drained)"| D["Fail fast — no retry"]
    C --> E["Retry"]
    F["Call succeeds"] -->|"add tokenRatio 0.1"| G["Refill bucket"]
    D --> H["Retry load capped at ~10%<br/>storm cannot form"]
    E --> H

When to use it — and when not

Reach for a retry budget on any client that retries a shared, hot-path dependency at fleet scale — the exact setup where synchronized retries can multiply load. It is not a substitute for the other guardrails: a budget bounds the multiplier but a circuit breaker is what takes retries to zero against a fully-dead dependency, and load shedding is what saves the server — ship them together. Skip the budget where it earns nothing: a single client / low-concurrency job can't create a storm, so a plain capped-retry-with-backoff is enough. Don't set the budget so loose it never bites (a 90% budget is theatre) nor so tight it starves legitimate transient retries during normal variance — 10–20% is the tested band. And a budget is a client-side control: it caps what you emit, so it does nothing for a storm arriving from clients you don't own — that case needs server-side shedding and admission control.

Signals it's working

• A retry-amplification SLI: alarm when retries / total_requests exceeds ~20% for 1 minute — that ratio climbing is the earliest sign a storm is forming, and a healthy budget holds it near the 10% steady state.
• retry_budget_exhausted / throttled-retry counter is zero in calm periods and spikes then self-clears during a blip — proof the bucket drained, fast-failed the excess, and refilled on recovery without a human.
• Load at the dependency during a stall stays within a bounded multiple of real demand (≈1.1×, not 5×), and the fleet recovers on its own after the trigger clears — no restart needed.
• Game-day proof: inject a 2–5s stall into a hot dependency in staging and assert retry load returns to baseline within the budget and the fleet self-recovers after the stall clears.

Seen in the wild

• gRPC ships a first-class retry throttling token bucket (retryThrottling with maxTokens / tokenRatio in the service config) — the canonical implementation of this pattern (gRFC A6).
• Envoy exposes a retry_budget (budget_percent, min_retry_concurrency) that caps active retries as a percentage of concurrent requests, at the proxy/mesh layer.
• AWS Architecture Blog ("Exponential Backoff And Jitter") and the Google SRE book both prescribe full-jitter backoff plus a per-client retry budget precisely to prevent retry-storm / metastable failures.

See it

CompareScene and CardsScene carry the point: an uncapped retry stack drives load to 5× and pins the service, while a token-bucket budget (maxTokens 1000, tokenRatio 0.1 → ≤10%) plus full-jitter backoff (100ms base, 10s cap) holds the multiplier near 1.1× and lets the fleet self-recover.

Named concepts: retry budget, token bucket, exponential backoff with jitter, retry amplification. </content> </invoke>