Drill 81 — The Unbounded Queue

Theme: Queues & Backpressure · Format: 20-min drill · Difficulty: core

The story

A transaction-enrichment service consumes events, calls an enrichment API, and forwards the result. To smooth out spikes, it buffers incoming events in an in-memory queue. Normally that queue hovers near empty; the buffer has never mattered.

One afternoon the enrichment API slows — a downstream of its own degrades and per-call latency jumps from twenty milliseconds to two seconds. Producers keep arriving at the same rate; consumers now drain a hundred times slower. The in-memory queue starts to grow: thousands of events, then hundreds of thousands. The heap climbs, GC pauses lengthen — which makes the consumers slower still, which grows the queue faster. Latency across the whole service balloons.

Then the process is OOM-killed. Everything buffered in memory is gone, and now there's a data gap and a backlog. It restarts, the queue refills within minutes, and it dies again — a crash loop that only ends when the enrichment API finally recovers on its own.

⏸ Stop here and ask the room: the queue existed to absorb spikes. Why did the spike-absorber become the thing that killed the service?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

What turned an in-memory buffer into a liability?

Listen for: an unbounded queue is unbounded memory. A buffer only helps if producers eventually slow or the queue is bounded; a persistent slowdown with steady arrivals converts directly into unbounded growth and OOM. The GC feedback loop accelerates the collapse.
Weak signal: "add more memory" — that just delays the OOM; the queue is still unbounded.

What should happen when the queue fills?

Listen for: a bounded queue with an explicit full-policy — block the producer (backpressure), reject / return 429, or shed low-priority work. A durable broker (disk-backed) changes the failure from lossy-OOM to a bounded, recoverable backlog.
Weak signal: no policy named, or "just drop everything" without priority.

How would you catch this before the OOM?

Listen for: queue depth as an SLI with an alarm at a fraction of capacity, plus producer/consumer rate and consumer-lag monitoring; alert on the depth trend, not just memory.
Weak signal: "monitor memory" alone — that's the symptom at the very end, not the cause.

The reveal

They bounded the queue, moved buffering to a durable broker so a slowdown yields a bounded, recoverable backlog instead of an in-memory OOM, and applied backpressure: at capacity the service returns 429 and slows its own intake, which propagates upstream instead of silently swallowing work until death. A queue-depth alarm now fires at 70% of capacity, long before memory pressure. During the next enrichment slowdown, depth rose, the alarm paged, backpressure engaged — and the service degraded gracefully instead of crashing.

Why this happens — the mechanics

A queue is a shock absorber only if it's bounded and something pushes back when it's full. By Little's Law, queue length equals arrival rate times time-in-system: if service time jumps ten- or a-hundred-fold and the arrival rate holds, length grows without limit until it hits a hard resource wall — memory here, disk elsewhere. In-memory buffers make that wall both fatal (OOM) and lossy (buffered work vanishes), and the garbage-collection feedback loop — more heap, longer pauses, slower consumers, more heap — turns a slow bleed into a fast collapse.

Backpressure is the missing signal: the system must be able to say "slow down" and have that propagate to the source. Without it, "absorb the spike" quietly becomes "absorb until death." An unbounded queue isn't resilience — it's a crash with a configurable delay.

Diagrams

Problem — how the failure unfolds:

timeline
    title The buffer that became the bomb
    section Enrichment slows
        T+0 : Enrichment latency 20ms to 2s : consumers drain 100x slower
        T+5m : Queue grows to hundreds of thousands : heap climbs
        T+12m : GC pauses lengthen : consumers slow further (feedback loop)
    section Collapse
        T+18m : OOM-kill : buffered events lost
        T+19m : Restart, queue refills, dies again : crash loop until API recovers

Solution — where the guardrails go:

flowchart TD
    A["Incoming events"] --> B["Bounded queue (max N)"]
    B -->|"at capacity"| C["Backpressure: 429 /<br/>slow intake, propagate upstream"]
    B -->|"buffer"| D["Durable broker (disk-backed):<br/>bounded, recoverable backlog"]
    B --> E["Queue-depth SLI · alarm at 70%"]
    C --> F["Graceful degradation, not OOM"]
    D --> F
    E --> F

What good looks like

Every queue is bounded with an explicit full-policy: block (backpressure), reject (429), or shed low-priority work.
Prefer a durable, disk-backed broker for buffering so a slowdown yields a bounded, recoverable backlog — not in-memory loss.
A queue-depth SLI with an alarm at a fraction of capacity (e.g., 70%), plus producer/consumer rate and consumer-lag monitoring.
Backpressure propagates: when full, the service signals upstream to slow rather than accepting unboundedly.
Load-shedding for low-priority work under pressure, with a documented drop policy.
A load test that holds producer rate while injecting a downstream slowdown and asserts graceful degradation, not OOM.

The lesson

An unbounded queue is a crash with a delay; a buffer only helps if it's bounded and pushes back when it's full. Meter queue depth as an SLI and alarm before the resource wall, not after it.

Named concepts: bounded queues, backpressure, load shedding, queue-depth SLI (Little's Law).

Real-world echoes

In-memory buffer OOM incidents are a staple of post-incident reviews; it's precisely why production messaging systems (Kafka, SQS, RabbitMQ with limits) are disk-backed and bounded rather than unbounded in-process queues.

Transfer

Which in-memory queues in your services are unbounded — and what happens to each if its consumer slows a hundredfold for an hour?

Make it a ticket: bound every queue with an explicit full-policy, add a depth SLI + alarm at 70%, move critical buffering to a durable broker, and load-test the persistent-slowdown case.

Drill 21 — The Thundering Herd (the arrival-side version of overload).
Drill 24 — The Autoscaler That Fed the Fire (a feedback loop that amplifies stress).
Patterns #111 Backpressure and #129 Load Shedding.