Pattern — Load Shedding

The problem it solves

An overloaded system that accepts every request doesn't slow down gracefully — it collapses to zero goodput. In Drill 84 a brokerage's order gateway keeps returning 202 Accepted while a spike triples arrivals to 15,000/s against a tier that drains 4,000/s; the queue balloons past hundreds of thousands, end-to-end latency climbs from 200 ms to minutes, and every accepted order routes against a stale price. In Drill 24 the "add capacity" reflex feeds a positive-feedback loop until the database dies. The common shape: acceptance is not completion — when demand exceeds capacity, absorbing the excess is a promise the system can't keep, and unbounded absorption converts a spike into a total outage.

The pattern

Load shedding is the deliberate, fast rejection of excess work at the edge so the work you do admit actually completes. When you can't serve everyone, serving a bounded subset well beats serving everyone badly. It is the action half of the backpressure contract: backpressure is the return-channel signal ("I'm saturated"); load shedding is what the front door does when it hears it — say no cheaply (HTTP 429 / 503), before the work is even created, instead of accept-then-drown. Done well it keeps goodput flat under overload instead of letting it crater, and it prefers your most valuable traffic when it has to choose.

How to apply it

1. Size the admission control to measured capacity, not hope. Put a token bucket at the edge: refill = the tier's measured drain rate (4,000 tokens/s in Drill 84), burst ≈ 2× (8,000). No token → immediate reject. Behind it, a bounded queue with a hard cap sized to a few seconds of drain — 30 s × 4,000/s = 120,000 — that fails closed at capacity. No queue is ever unbounded.
2. Reject fast and cheap, with an honest signal. Over-rate or at-capacity → HTTP 429 + Retry-After (or 503 for infra overload), returned in under 5 ms, before risk checks or downstream calls. Retry-After plus client-side jittered backoff stops rejected clients from becoming a retry storm. Never accept-then-drop — a dropped 202 is a silent lie.
3. Shed by priority, not at random. Tag traffic with a criticality tier (Google's scheme: CRITICAL_PLUS > CRITICAL > SHEDDABLE_PLUS > SHEDDABLE) and drop the cheapest, most-retryable classes first — background sync, speculative prefetch, retries — while paying-customer orders get the last tokens. Uniform random shedding taxes your best traffic as hard as your worst.
4. Make the limit adaptive, and cap fan-out. Give each worker a concurrency limit (e.g. 50 in-flight) and let it self-tune with AIMD on measured latency (Netflix concurrency-limits): RTT climbs → shrink the limit → shed sooner. Trigger the shed when queue depth > 70 % of cap or in-flight hits the limit. And bound total downstream demand: max_pods × pool_size ≤ 80 % of DB max_connections (Drill 24) — so scaling out can't strangle a shared dependency the way "just autoscale" did.

flowchart TD
    A["Incoming work<br/>(3x spike)"] --> B["Token bucket<br/>(refill 4000/s, burst 8000)"]
    B -->|"no token / over rate"| R["Reject in under 5 ms<br/>429 + Retry-After"]
    B -->|"admitted"| Q["Bounded queue<br/>(cap 120,000 = 30s drain)"]
    Q -->|"depth > 70% or at cap"| R
    Q --> P["Worker<br/>(concurrency limit 50, AIMD)"]
    P --> G["Goodput stays flat<br/>admitted work finishes in ~300 ms"]
    R -.->|"shed SHEDDABLE first"| G

When to use it — and when not

Reach for it on any edge that fronts a slower, capacity-bounded tier it can't infinitely scale — a shared pricing service, an execution venue, a database. Load shedding is the right tool when failing fast is better than failing slow: retryable, idempotent, or low-value requests where an honest 429 lets the client back off. Don't shed work you cannot afford to lose — a payment you already took money for belongs in a durable queue with backpressure, not the reject pile; shed the admission, never the committed effect. Don't let shedding hide under-provisioning: shedding during a spike is health, but shedding at steady state means you're simply too small — fix capacity, not the reflex. And on a positive-feedback loop (Drill 24), shedding at the front door protects the constraint but doesn't fix the mis-wired controller — you still have to stop the autoscaler and scale down; shedding buys time, it isn't the whole cure.

Signals it's working

Treat the 429/503 rate as a first-class SLI, not an error: rising rejections under a 3× spike are healthy shedding; flat-zero 429s under overload means shedding isn't wired and you're heading for accept-until-collapse. The proof metric is goodput — successfully completed work per second — which should hold flat (or rise) under overload instead of cratering to zero, with p99 latency for admitted work staying stable (~300 ms, not minutes). Alarm when queue depth ≥ 70 % of cap. Prove it in a game day: hold arrivals at 3× drain for 10 minutes and assert a bounded queue, honest 429s, and stable admitted-p99 — degradation, not collapse.

Seen in the wild

• Netflix concurrency-limits — each service measures its own latency, infers saturation, and sheds with a fast rejection via an AIMD/gradient limit rather than queueing unboundedly.
• Google SRE ("Handling Overload") — per-request criticality tiers plus adaptive throttling; clients self-throttle so overloaded backends shed the least-important traffic first.
• Envoy / Istio — the adaptive-concurrency filter and circuit-breaker max_pending_requests shed at the proxy; AWS S3 503 Slow Down, API Gateway 429, and DynamoDB throttling are the same reflex at cloud scale.
• Facebook CoDel — controlled-delay queue management drops requests that have already waited too long, so stale work is shed instead of served.

See it

The ChainScene carries the mechanism — token bucket (4,000/s refill, 8,000 burst) → over-rate → 429 + Retry-After in under 5 ms → goodput stays flat; the TilesScene and CardsScene show the load-bearing numbers (120,000 queue cap, 50-concurrency AIMD limit, max_pods × pool ≤ 80 % of DB max_connections, and the SHEDDABLE-first priority tiers).

Named concepts: load shedding, admission control, priority-based shedding, adaptive concurrency limits.