Drill 84 — The Missing Backpressure

Theme: Queues & Backpressure · Format: 20-min drill · Difficulty: core

The story

A brokerage's order-intake service sits on the hot path. The front-end is a stateless order gateway: it accepts an order over HTTP, checks the shape of it, returns 202 Accepted, and drops the order onto an internal work queue. A slower processing tier pulls from that queue — running risk checks, verifying buying power, calling a pricing service, then routing to the execution venue. The gateway is fast and cheap; the processing tier does real work and drains at roughly four thousand orders a second. The queue between them has always hovered near empty, so nobody has ever thought about it.

Then the market opens on a volatile morning. Submission volume triples to about fifteen thousand orders a second. The gateway does exactly what it is built to do: it accepts all of them, returns 202 to every client, and enqueues. The processing tier keeps draining at four thousand a second. The queue now grows by eleven thousand orders every second — tens of thousands deep, then hundreds of thousands. End-to-end time from submit to executed climbs from two hundred milliseconds to seconds, then to minutes. Orders finally route against prices that have long since moved. Clients who see no fill resubmit, which adds more load, which pushes the drain further behind.

The whole pipeline grinds to a halt — and the tell is that the gateway is still returning 202. Every request at the edge succeeds. The edge dashboards are green. Nothing in the front-end knows the processing tier is drowning, because nothing downstream can reach back and say stop.

⏸ Stop here and ask the room: the front-end was fast and healthy the whole time — it returned 202 to every order. So why did the pipeline collapse?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Why did a fast, healthy front-end make things worse, not better?

Listen for: with no flow control, a fast front-end just fills the queue faster than the back-end can ever drain it — acceptance is not completion. The 202 is a promise of work that won't happen for minutes; speed at the edge without backpressure is a liability, not a feature. Scaling the gateway would accelerate the collapse.
Weak signal: "add more front-end capacity," or "the front-end was fine" — missing that accepting unservable work is the failure.

What signal is missing between the two tiers?

Listen for: backpressure — the queue or the processing tier must be able to say "I'm saturated, slow down," and the front-end must act on it by rejecting (HTTP 429 with Retry-After) or blocking. Load currently flows one way only; there is no return channel from the slow stage to the fast one.
Weak signal: "make the queue bigger" or "add memory" — unbounded absorption just buys a few more minutes before the same collapse.

When the front-end must reject, how does it decide — and stay fair?

Listen for: admission control at the edge — a token bucket sized to the tier's measured drain rate (about four thousand a second, plus a burst), a bounded queue with a hard capacity, and a fast 429 + Retry-After when either is exceeded. Back it with a concurrency limit on each processor and autoscaling on queue depth. Fail fast and cheap; never accept-then-drop.
Weak signal: "drop random requests" with no rate basis, or "just autoscale it" — you cannot scale a shared pricing service or execution venue infinitely, and scaling lags the spike by minutes.

The reveal

They put a return channel between the tiers. The processing tier now publishes its saturation — queue depth and in-flight concurrency — and the gateway runs a token-bucket admission control sized to the real drain rate. The internal queue is bounded; over-rate or at-capacity, the gateway returns 429 with Retry-After instead of 202. Each processor carries a hard concurrency limit and the fleet autoscales on queue depth. On the next volatile open the gateway sheds the excess at the edge: clients get an honest 429 and back off, the queue stays bounded, the tier stays busy but never buried, and accepted orders execute in about three hundred milliseconds. Rejection is now visible and finite; before, the failure was invisible and total.

Why this happens — the mechanics

A pipeline is only as fast as its slowest stage, and a fast front-end feeding a slow tier with no backpressure converts a spike directly into an unbounded backlog. Returning 202 without downstream capacity is a promise the system cannot keep — acceptance is not completion. By Little's Law the backlog is (arrival rate − service rate) × time: if arrivals hold above the service rate, queue length grows without limit until it hits a hard wall (memory, broker cap), and every accepted-but-unservable order is wasted work that makes the drain slower, which grows the backlog faster.

The deeper root is an ownership seam. The front-end team owns "accept fast." The processing team owns "drain." Nobody owns the contract between them — the channel that carries "I'm full, stop." Backpressure is precisely that channel, and load-shedding is what the fast stage does when it hears it. Without them, "absorb the spike" silently becomes "accept until collapse," and the only honest thing the edge can do — say no — is the one thing it was never wired to do.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant C as Client
    participant G as Gateway
    participant Q as Queue
    participant P as Processor
    C->>G: submit order<br/>(3x spike)
    G->>Q: enqueue work
    G-->>C: 202 Accepted
    Note over G: keeps accepting<br/>no flow control
    Q->>P: dequeue order
    Note over Q: depth balloons<br/>unbounded (silent)
    P--xC: executes too late<br/>stale price
    Note over G,P: pipeline collapses<br/>no backpressure

Solution — where the guardrails go:

flowchart TD
    A["Incoming orders<br/>(market-open spike)"] --> B["Token-bucket<br/>admission control"]
    B -->|"over rate"| R["Reject fast · 429<br/>Retry-After"]
    B -->|"admitted"| Q["Bounded queue<br/>(max depth N)"]
    Q -->|"at capacity"| R
    Q --> P["Processor<br/>(concurrency-limited)"]
    P --> S["Autoscale on<br/>queue depth"]
    Q -.-> AL["Depth SLI alarm<br/>at 70% capacity"]
    R -.-> AL

What good looks like

Backpressure signal: the processing tier publishes saturation (queue depth, in-flight count); when depth exceeds 70% of capacity or in-flight hits the concurrency limit, the gateway returns HTTP 429 + Retry-After instead of 202. Reject fast (under 5 ms) — never accept-then-drop.
Bounded queue: a hard cap sized to a few seconds of drain (e.g. 30 s × 4k/s = 120,000); at capacity the enqueue fails closed and the request gets a 429. No queue is ever unbounded.
Token-bucket admission control at the edge: refill = the measured drain rate (4,000/s), burst ≈ 2× (8,000); no token means an immediate 429 with Retry-After. Shed at the cheapest point — before the work is even created.
Concurrency limits + autoscale: cap each processor at, say, 50 in-flight; autoscale on the queue-depth SLI — add workers when depth > 70% for 2 min, remove when < 30% for 10 min — and cap max workers to what the slowest dependency (pricing, venue) can actually serve.
Queue-depth SLI + 429-rate SLI: page on depth ≥ 70% of capacity; treat the 429 rate as a first-class metric — rising 429s are healthy shedding, not an outage, and flat-zero 429s under a 3× spike means backpressure isn't wired.
Game-day load test: hold arrivals at 3× the drain rate for 10 minutes and assert a bounded queue, honest 429s, and stable p99 for admitted work — not collapse.

The lesson

A fast front-end without backpressure is just a faster way to fall over. A pipeline needs a return channel that says "slow down," and the edge must reject work it cannot finish before it accepts it — because acceptance is not completion.

Named concepts: backpressure signalling, bounded queues, token-bucket admission control, concurrency limits.

Real-world echoes

Netflix's concurrency-limits library productizes exactly this reflex: each service continuously measures its own latency, infers when it is saturated, and sheds load with a fast rejection rather than queueing unboundedly — adaptive backpressure at the edge instead of accept-until-death. The same idea shows up as AWS S3's 503 Slow Down and API gateways' 429 Too Many Requests: a downstream that can say stop is what keeps a spike from becoming an outage.

Transfer

Which of your edge services returns a fast 200 or 202 for work that a slower tier actually has to finish — and what, today, tells that edge to stop when the tier is buried?

Make it a ticket: put a token-bucket admission control sized to the measured drain rate on your busiest intake endpoint, bound its internal queue, return 429 + Retry-After at capacity, and add a queue-depth alarm at 70% plus a 429-rate SLI — then load-test at 3× drain and prove it degrades, not collapses.

Drill 81 — The Unbounded Queue (the sibling: the buffer that grows until OOM; this is the missing return channel that lets it grow).
Drill 21 — The Thundering Herd (the arrival-side shape of the same overload).
Drill 24 — The Autoscaler That Fed the Fire (why "just autoscale" can amplify, not absorb, the stress).
Pattern #111 — Backpressure, and Pattern #129 — Load Shedding (the reusable fixes).