Pattern — Dead-Letter Queue

The problem it solves

One un-processable message can stall an entire ordered queue. Drill 82 is the canonical shape: a payment consumer commits its offset only after a successful post, so a message with a malformed amount can never be acked — the broker redelivers it forever, the offset never advances, and fifty thousand valid payments pile up behind one poison message while the process stays green and CPU-busy. Retrying is the right move for a transient failure (a lock, a network blip) and a trap for a deterministic one (a schema that fails identically every time). The same trap appears anywhere a pipeline processes a stream of records it can't all parse — including the malformed feature rows that quietly corrupt a scoring/eval batch (Drill 96): without a place to set the bad record aside, you either loop on it forever or drop it silently, and a lost record is worse than a stalled one.

The pattern

Cap redeliveries, then route the poison message to a separate dead-letter queue (DLQ) and commit past it. After a bounded number of failed receives the broker (or the consumer) stops handing the message back, moves it to a side channel that preserves it for a human, and lets the offset advance so good traffic flows. The DLQ turns an infinite retry loop into a bounded one with an escape hatch: the retry cap decides a message is un-processable, the DLQ contains it, and an alarm on DLQ depth turns a silent stall into a page. Validate the schema at enqueue so the structurally-broken message never enters the ordered stream in the first place.

How to apply it

Cap redeliveries at a small integer — 5 is the standard default. On Amazon SQS this is a redrive policy on the source queue: {"deadLetterTargetArn": "...:payments-dlq", "maxReceiveCount": 5}. On the 6th receive SQS moves the message to the DLQ automatically. Pick the cap high enough to ride out transient failures, low enough that a deterministic failure fails fast — for a single-partition ordered queue, 5 attempts × your visibility timeout is the worst-case head-of-line stall you're signing up for, so keep it tight.
Set the visibility timeout above p99 processing time. If work takes 8s at p99, a 30s visibility timeout keeps a slow-but-alive message from being redelivered as a false poison; setting it too low inflates receiveCount on healthy work and dead-letters good messages. Rule of thumb: visibility timeout ≥ 6× p99, capped at the SQS max of 12 hours.
Give the DLQ a long retention so nothing is lost while a human triages. Set the DLQ's MessageRetentionPeriod to the SQS maximum — 1,209,600 seconds (14 days) — so a message dead-lettered on a Friday survives the weekend. The DLQ is a durable inbox of defects, not a graph nobody reads.
On brokers without native redrive, dead-letter explicitly. Kafka has no built-in DLQ, so use a dead-letter topic: Kafka Connect ships errors.tolerance=all, errors.deadletterqueue.topic.name=payments.dlq, and errors.deadletterqueue.context.headers.enable=true (which stamps the failing offset, partition, and exception onto the record). Hand-rolled consumers do the same by hand — try N times, then producer.send("payments.dlq", record) and commit the offset.
Validate the schema at enqueue so the poison never enters. Reject a structurally-invalid message at the producer/gateway with a 400 and a contract-test in CI, so a malformed ISO 20022 amount is caught at the door, not fifty thousand payments deep in an ordered partition.
Alarm on DLQ depth and on offset stall. Page on-call when DLQ depth > 0 (target: within 5 minutes of the first dead-lettered message) and when the committed offset is unchanged for 2 minutes while queue depth > 0 — because a busy consumer looks identical to a working one, and only a progress signal tells them apart.

flowchart TD
    A["Incoming message"] --> V["Schema validation at enqueue"]
    V -->|"invalid"| R["Reject at source (400)<br/>never enqueued"]
    V -->|"valid"| C["Consumer · retry cap<br/>maxReceiveCount = 5"]
    C -->|"succeeds"| L["Process + commit offset"]
    C -->|"fails 5x (deterministic)"| PZ["Poison detected<br/>route to DLQ, advance offset"]
    PZ --> D["Dead-letter queue<br/>retention 14 days"]
    D -.-> AL["Alarm · DLQ depth > 0<br/>pages on-call in 5 min"]
    C -.-> AL

When to use it — and when not

Reach for a DLQ on any consumer that can hit a message it can never process — an ordered payment stream, a webhook worker, a feature-ingestion job. It is the difference between a bounded incident and a silent all-stop. But know its limits. A DLQ isn't a substitute for idempotency — because dead-lettering happens after N receives, a message can be processed successfully on attempt 3 and still get redelivered on a timeout, so the consumer must be replay-safe (Pattern 01) or you'll double-apply the effect you were trying to protect. Don't dead-letter transient failures — a downstream outage will drain your whole queue into the DLQ in minutes; pair the cap with backoff, and for outages use a retry queue with delay, not a dead-letter one. And a DLQ that nobody drains is just a slower data-loss — without a triage runbook and a depth alarm, messages age out at 14 days and vanish. Skip the DLQ only where a dropped message is genuinely acceptable (best-effort telemetry), where you'd rather block than lose (then you need a human circuit-breaker instead), or on a naturally-unordered queue where one bad message blocks nothing and you can just ack-and-log it.

Signals it's working

The load-bearing signal is DLQ depth: healthy is 0, and any non-zero value should page within 5 minutes and get a human reading the message. Track per-message receiveCount / delivery-count and alarm when a single message crosses the cap (5), and watch offset-advance — alarm when the committed offset is flat for 2 minutes while depth > 0. The proof it's wired correctly is a game-day: inject one malformed message ahead of a valid batch and assert the valid batch still drains, the bad one lands in the DLQ within the retry budget, and on-call pages. In Drill 82 the tell was brutal — throughput to the ledger at zero while every classic health check read green; the DLQ plus a progress alarm is exactly what turns that invisible stall into a 5-minute page.

Seen in the wild

Amazon SQS ships redrive natively: a source queue's RedrivePolicy with maxReceiveCount, plus a redrive-allow-policy on the DLQ, and a one-click "redrive to source" once the bug is fixed.
Kafka / Kafka Connect has no native DLQ but standardizes the dead-letter topic via errors.tolerance=all and errors.deadletterqueue.topic.name; the poison-pill partition stall is the textbook reason it exists.
RabbitMQ implements it as a dead-letter exchange (x-dead-letter-exchange), routing messages that are rejected, expire, or exceed a length limit.
Google Pub/Sub and Azure Service Bus both expose a dead-letter queue with a maxDeliveryAttempts / max-delivery-count cutoff.

See it

The ChainScene carries the mechanism — retry cap 5 → poison detected → route to DLQ → offset advances, payments drain — and the CardsScene shows the load-bearing config (maxReceiveCount=5, visibility ≥ 6× p99, 14-day retention, DLQ-depth > 0 pages in 5 min).

Named concepts: dead-letter queue (DLQ), retry caps, poison-message handling, redrive policy, head-of-line blocking.