Drill 82 — The Poison Message

Theme: Queues & Backpressure · Format: 20-min drill · Difficulty: core

The story

A bank's payment-processing service has one job: consume credit-transfer instructions from an ordered queue, parse each ISO 20022 message, and post it to the ledger. The queue is a single partition so payments settle in the order they arrive. The consumer commits its offset only after a message posts successfully — that way a crash never loses a payment. It has run clean for two years.

At 09:14 one message arrives with a malformed amount field — a corporate client's new integration serializes a decimal as an empty string. The consumer pulls it, tries to parse, and throws. Because the post never succeeds, the consumer does not commit the offset; the broker treats the message as un-acknowledged and redelivers it. The consumer pulls the same message, throws again, and the broker redelivers again. The consumer is now spinning on one offset, forever, at the head of the queue.

Behind that one message, every valid payment waits. Throughput to the ledger drops to zero. But nothing pages. The consumer process is up — CPU is busy, no crash, no restart. The health check is green because the consumer is answering. Error logs scroll past at thousands of lines a minute, but there is no alarm on "the same offset retried a thousand times," and no alarm on "the offset hasn't advanced." The queue depth climbs — ten thousand, fifty thousand held payments — and the settlement window ticks down while a treasury analyst is the first human to notice, at 10:40, that nothing has settled since nine.

⏸ Stop here and ask the room: the consumer never crashed and the health check stayed green. So why did every payment behind one message stop settling for an hour and a half?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

One malformed message stalled the whole queue — why the whole queue, not just that one?

Listen for: head-of-line blocking on an ordered queue. The consumer commits the offset only after success, so a message it can never process pins the offset in place; the broker's at-least-once redelivery hands back the same message indefinitely, and every message behind it is stuck by ordering. One un-processable message = a stalled partition.
Weak signal: "the consumer crashed" (it didn't) or "add more consumers" — more consumers on one partition still block on the same head message.

The process was up and the health check was green — what monitor was missing?

Listen for: liveness is not progress. You need a progress signal: consumer-lag / offset-advance ("offset unchanged for N minutes while depth > 0"), a per-message delivery-count metric, and a DLQ-depth alarm. A busy-but-stuck consumer must page.
Weak signal: "monitor CPU / is the process alive" — that is exactly the signal that lied here.

Where does an un-processable message belong, and who decides it's un-processable?

Listen for: after a bounded number of retries it is a poison message — move it to a dead-letter queue, commit the offset, and let good traffic flow. Validate the schema at enqueue so the malformed message never enters the ordered stream in the first place. The retry cap decides; the DLQ contains.
Weak signal: "retry until it works" (it never will) or "drop it silently" (a lost payment is worse than a stalled one).

The reveal

The fix is not "parse more carefully" — you cannot parse your way out of a message that is structurally wrong. They capped retries at five attempts; on the sixth the consumer routes the message to a dead-letter queue, commits the offset, and moves on. The instant that shipped, the stuck offset advanced and fifty thousand backed-up payments drained in minutes. They added schema validation at enqueue so a malformed message is rejected at the producer and never enters the ordered queue, and a DLQ-depth alarm that pages on-call the moment one message lands in the dead-letter queue. The poison message went to the DLQ, an engineer read it in thirty seconds, and the client fixed their serializer — while payments kept settling the whole time.

Why this happens — the mechanics

Ordered, at-least-once delivery makes correctness and liveness pull against each other. Committing the offset only after success is the right call for not losing a payment — but it means a message the consumer can never process becomes a permanent barricade: the offset can't advance past it, redelivery keeps handing it back, and ordering freezes everything behind it. The retry that protects you against a transient failure (a brief network blip, a momentary lock) is a trap against a deterministic one (a schema that will fail identically on every attempt). Retrying a deterministic failure is an infinite loop with extra steps.

The reason nobody saw it is that every classic health signal measures the wrong thing. The process is alive, the CPU is busy, the consumer is polling and answering — all green — because effort is not progress. A stuck consumer looks identical to a working one from the outside; the only signals that tell them apart are the ones that watch the offset move and the backlog grow. Without a retry cap the loop is infinite; without a DLQ the poison has nowhere to go; without a progress alarm the stall is silent until a human downstream notices the money stopped moving.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant P as Producer
    participant Q as Payment Queue
    participant C as Consumer
    participant L as Ledger
    P->>Q: enqueue valid<br/>payments
    Q->>C: deliver msg 1<br/>(malformed amount)
    C--xL: post fails<br/>parse error
    C->>Q: no offset commit<br/>redeliver
    Q->>C: deliver msg 1<br/>again
    C--xL: fails again<br/>retry forever
    Note over Q,C: valid payments wait<br/>behind the poison msg
    Note over C: process up, health green<br/>throughput drops to zero

Solution — where the guardrails go:

flowchart TD
    A["Incoming payment message"] --> V["Schema validation<br/>at enqueue"]
    V -->|"invalid"| R["Reject at source<br/>never enqueued"]
    V -->|"valid"| C["Consumer · retry cap<br/>N=5 then move on"]
    C -->|"succeeds"| L["Post to ledger<br/>commit offset"]
    C -->|"fails N times"| PZ["Poison detected<br/>route to DLQ"]
    PZ --> D["Dead-letter queue<br/>advance offset"]
    D -.-> AL["Alarm · DLQ depth > 0<br/>pages on-call in 5 min"]
    C -.-> AL

What good looks like

Retry cap with move-on: bound redelivery at N = 5 attempts per message; on the last failure, stop retrying and route the message onward — never retry a deterministic failure indefinitely.
A dead-letter queue: the poison message goes to a DLQ (SQS redrive with maxReceiveCount=5, or an explicit dead-letter topic), the offset commits, and good traffic flows past. The bad message is preserved for inspection, not lost.
Schema validation at enqueue: validate against the message contract at the producer / gateway so a structurally invalid message is rejected with a 400 and never enters the ordered queue. Back it with a contract test in CI.
Poison-message detection: track a per-message delivery-count and an offset-advance signal; alarm when any single offset is redelivered more than 5 times, or when the committed offset is unchanged for 2 minutes while queue depth > 0.
DLQ-depth alarm: page on-call when DLQ depth > 0 (target: within 5 minutes of the first dead-lettered message) — a message in the DLQ is a defect that needs a human, not a graph nobody reads.
A game-day: inject one malformed message ahead of a valid batch and assert the valid batch still settles and the bad one lands in the DLQ within the retry budget.

The lesson

An ordered queue moves at the speed of its slowest message, and a message that can never succeed will retry forever unless you cap it and give it somewhere to go. Cap retries, dead-letter the poison, and alarm on progress — because a busy consumer is not a working one.

Named concepts: poison-message handling, dead-letter queue (DLQ), retry caps, head-of-line blocking, schema validation at enqueue.

Real-world echoes

Kafka consumer poison-pill stalls are a well-known operational hazard: a single un-deserializable record on a partition, combined with commit-after-success and infinite retry, halts the entire partition until the record is skipped or dead-lettered. It is exactly why managed queues ship redrive / dead-letter policies and why "skip and dead-letter after N" is the standard consumer pattern.

Transfer

Which of your consumers would stall the entire queue if the very next message were structurally un-processable — and how long before anyone noticed, given that the process would still look healthy?

Make it a ticket: for every ordered consumer, add a retry cap (N = 5) plus a dead-letter queue, add schema validation at enqueue, and wire a DLQ-depth alarm and an offset-stall alarm; prove it with a game-day that injects one poison message and asserts good traffic keeps flowing.

Drill 81 — The Unbounded Queue (sibling): the backlog-growth side of the same theme — here the queue stalls, there it grows without bound.
Drill 20 — The Agent That Wouldn't Stop: unbounded retry of an operation that can never succeed.
Drill 13 — idempotent side effects: dead-lettering and redelivery both assume replay-safe processing.
Pattern #128 — Dead-Letter Queue (the reusable fix).