Pattern — Circuit Breaker

The problem it solves

A dependency you don't control goes slow or down, and your callers keep marching straight into it — parking threads on a dead-slow vendor until the pool starves (Drill 76 — The Timeout That Wasn't Set), or retrying a stalled service in lockstep until the retries themselves become the load (Drill 80 — The Retry Storm). In both, the fault is long over but the caller keeps hammering, so a momentary blip latches into a sustained outage. The missing primitive is something that notices the dependency is unhealthy and stops calling it.

The pattern

A circuit breaker is a stateful proxy around one downstream that counts failures and short-circuits calls when they cross a threshold. It has three states: closed (calls flow, failures counted), open (calls fail instantly without touching the dependency — for a fixed cooldown), and half-open (after the cooldown, a single probe is allowed; success closes the breaker, failure re-opens it). Timeouts and errors both count as failures. The point is to convert "wait 30 seconds then fail" into "fail in one millisecond," so a sick dependency degrades one feature instead of consuming the caller.

How to apply it

Wrap each downstream in its own breaker instance (never share one across dependencies) and configure four numbers:

• Trip condition: open when the error rate exceeds 50% over a rolling window — require a minimum of 20 requests in the window (a 10-second sliding window works well) so one unlucky call can't trip it. Count timeouts as failures, not just 5xx / connection-refused — a slow-but-200 dependency is the case that hurts most.
• Cooldown (open duration): stay open 5 to 30 seconds before probing. Drill 76 used 30s; Drill 80 used ~5s. Longer cooldown = less load on a struggling dependency but slower recovery; pick by how expensive the dependency's recovery is.
• Half-open probe: allow exactly one in-flight request. One success → close; one failure → re-open and restart the cooldown. Do not let the whole fleet rush the dependency the instant the timer expires — that just rebuilds the storm.
• Fail-fast behavior while open: return a fallback immediately — a cached value, a "verification pending" default, or a fast error — instead of queuing. Wire the breaker to fire on timeouts, so pair it with an explicit client read timeout (e.g. connect 2s / read 5s) — the breaker can only count a failure the timeout declares.

The breaker is one guardrail in a set: put the dependency behind a bulkhead (its own bounded pool) so its breaker's blast radius is capped, and if you retry, retry inside the breaker with jittered backoff and a retry budget so a re-opened breaker gets zero retries, not three.

When to use it — and when not

Use it for calls to something you don't control on a hot path — an external vendor, an internal service on the request path, anything where "slow" or "down" can starve the caller. It is the wrong tool when there is no fallback and the call is mandatory (a payment authorization that must complete can't fail-fast into a lie — there you need queueing and idempotent retry, not a breaker), for a single non-critical background job (a breaker's bookkeeping isn't worth it), or when the real defect is a persistently overloaded dependency — a breaker masks the symptom and buys time, it does not add capacity. A breaker set too tight (low window, low threshold) becomes its own outage, flapping open on normal variance.

Signals it's working

• Breaker-state-change events are logged and alarmed: every open/half-open/close transition is visible, so you can see it trip during a blip and auto-close on recovery.
• open_state_seconds per dependency — nonzero exactly when the dependency is unhealthy, back to zero when it recovers, with no manual restart.
• Caller latency stays bounded during a downstream stall: p99 pins near the timeout, not at 30–60s, because open calls return in ~1ms instead of parking.
• Game-day proof: inject a 5-10s stall into the dependency in staging and assert the breaker opens, the rest of the service stays green, and the breaker half-opens and closes on its own after the stall clears.

Seen in the wild

Netflix Hystrix popularized the pattern (breaker + timeout + bulkhead in one library); Resilience4j and Polly (.NET) are the current standard implementations; Envoy and Istio ship outlier-detection circuit breaking at the service-mesh layer; AWS and Google SRE post-mortems of retry-storm / metastable failures prescribe exactly this — a breaker so a dead dependency gets zero retries — as the fix.

See it

CompareScene carries the point: without the breaker, caller p99 climbs to 30s and the pool starves; with it, the breaker trips at 50% errors and p99 pins at the 5s timeout — same outage, one feature degraded instead of the whole service.

Named concepts: circuit breaker, fail-fast, half-open probe.