Drill 85 — The Duplicate Delivery

Theme: Queues & Backpressure · Format: 20-min drill · Difficulty: core

The story

A payments platform runs a settlement consumer: it reads payment instructions off a broker, posts a double-entry row to the ledger, and hands the outbound wire to the rail. The broker is a standard managed queue with the standard managed guarantee — at-least-once delivery. The team has read that phrase a hundred times. Nobody has ever sat with what the least in "at least" costs them.

The consumer's loop is the obvious one: pull a message, post the ledger entry, ship the wire, then acknowledge the message so the broker drops it. It has settled millions of instructions this way. Throughput is fine, the ledger balances every night, and the runbook is one page long.

One Tuesday a settlement pod is evicted mid-batch — a node drains for a kernel patch, right in the window after the consumer has posted the ledger entry and shipped the wire but before it sends the ack. From the broker's point of view, message 42 was delivered and never confirmed. So the broker does exactly what at-least-once promises: when the consumer restarts, it redelivers message 42. The fresh consumer has no memory of the first attempt. It posts the ledger entry again. It ships the wire again. One instruction, two ledger rows, two wires out the door.

By mid-morning the reconciliation team is staring at duplicate debits, a ledger that no longer balances against the rail, and a customer who was paid twice. The consumer code is correct. The broker is behaving exactly as documented.

⏸ Stop here and ask the room: the broker did precisely what "at-least-once" guarantees, and the consumer code has no bug. So where is the defect?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

What does "at-least-once" actually oblige the consumer to do?

Listen for: the guarantee is a contract, and it delivers work onto the consumer: at-least-once means "you will sometimes see the same message twice, by design — a crash, a rebalance, a redelivery after a missed ack — so your handler must produce the same end state whether it runs once or five times." The consumer must be idempotent; that is the price of the guarantee, not an edge case.
Weak signal: "the broker should deduplicate" or "turn on exactly-once" — treating redelivery as a broker bug rather than a documented promise the consumer has to absorb.

Name the exact window — what is un-acked but already done?

Listen for: the gap between effect and ack. The consumer performs side effects (ledger insert, wire) and only then acks; a crash inside that window leaves the effect applied but the message un-acked, so it redelivers and the effect repeats. The fix is not "ack sooner" (that risks losing work) — it is making the effect itself replay-safe.
Weak signal: "ack first, then do the work" — which converts double-processing into lost processing the moment a crash lands after the ack.

Ledger post and outbound wire — same fix, or two problems?

Listen for: the ledger is your own database, so a unique key / upsert on the natural key (payment id) makes the second post a no-op. The wire crosses a system boundary you don't own, so it needs an idempotency key on the rail plus a transactional outbox so "record the intent" and "post the ledger" commit atomically and a relay ships each intent once. One principle — idempotent effects — two implementations.
Weak signal: solving only the ledger (the easy, same-database half) and leaving the money-moving side effect to fire twice.

The reveal

There was no code bug and no broker bug — there was an unmet obligation. "At-least-once" is a promise the consumer pays for by being idempotent, and this consumer was not. The team added a processed-messages store keyed on the broker message id with a unique constraint: a redelivered id is recognized and the handler short-circuits to just re-acking. The ledger post became an upsert on the natural key (payment_id, account). The wire moved behind a transactional outbox — the ledger row and the outbox intent commit in one transaction, and a relay ships each intent to the rail exactly once using the payment id as the rail's idempotency key. Duplicate settlements went to zero; the duplicate-delivery counter, correctly, kept climbing — because redelivery is normal and now it is simply absorbed.

Why this happens — the mechanics

At-least-once is not a defect; it is the only guarantee a distributed broker can cheaply make. To promise "exactly once" end to end, the broker would need a distributed transaction spanning its own commit and your side effects — the ledger, the payment rail — which it cannot see. So it makes the honest promise it can keep: it will hold a message until you acknowledge, and if the ack never comes, it will deliver again. Every missed ack — a crash, a pod eviction, a consumer-group rebalance, a visibility-timeout expiry under load — becomes a redelivery. Redelivery is not the exception; it is a routine event you are guaranteed to hit.

The bug, then, lives in the seam between the effect and the ack. A non-idempotent handler treats "I ran the effect" and "the broker knows I ran it" as one atomic step, but they are two, and any crash between them replays the effect. Exactly-once delivery is a fantasy across a boundary the broker can't transact over; exactly-once effect is achievable, and it lives entirely on your side — dedupe the message id, upsert on the natural key, and route external side effects through a transactional outbox so intent and state commit together. The organizational root is that "at-least-once" reads like fine print, so the idempotency obligation it hands the consumer never becomes anyone's explicit task.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant B as Broker
    participant Co as Consumer
    participant L as Ledger DB
    participant R as Payment rail
    B->>Co: deliver payment 42
    Co->>L: post ledger entry
    Co->>R: ship the wire
    Co--xB: ack never sent
    Note over Co: pod evicted<br/>before the ack
    Note over B: no ack —<br/>at-least-once redelivers
    B->>Co: redeliver payment 42
    Co->>L: post ledger entry again
    Co->>R: ship the wire again
    Note over Co,R: one instruction —<br/>two posts, two wires

Solution — where the guardrails go:

flowchart TD
    A["Redelivered message<br/>(at-least-once)"] --> B["Dedupe by message id<br/>unique processed-store"]
    B -->|"id seen before"| C["Short-circuit — effect<br/>already applied, re-ack"]
    B -->|"new id"| D["Transactional outbox:<br/>ledger row + outbox<br/>intent in one commit"]
    D --> E["Upsert on payment id<br/>INSERT ON CONFLICT"]
    E --> F["Ack only after the<br/>effect is durably committed"]
    F -.-> AL["Alarm: redelivery rate<br/>over one percent or DLQ depth over 0"]
    C -.-> AL

What good looks like

Idempotent consumer, keyed on message id. A processed_messages store with a unique constraint on the broker message id; the handler checks-or-inserts before doing work, and a redelivered id short-circuits to a re-ack. TTL on the store ≥ the maximum redelivery window (e.g., 7 days), so no id is forgotten while it can still return.
Upserts, not inserts, on the natural key. The ledger post is INSERT … ON CONFLICT (payment_id, account) DO NOTHING — the second attempt is a structural no-op, not a duplicate row, regardless of how many times the message replays.
Transactional outbox for cross-boundary effects. The ledger row and the outbound-wire intent commit in one database transaction; a relay ships each outbox row to the rail exactly once, passing the payment id as the rail's idempotency key so the rail itself dedupes. Intent and state can never disagree.
Understood ack semantics. Ack only after the effect is durably committed (never before). Set the visibility timeout above p99 processing time (e.g., 30 s over an 8 s p99) so slow-but-alive work isn't redelivered, and cap redeliveries (e.g., max 5 receives → dead-letter queue) so a poison message can't loop forever.
The signal that proves it's working. Track duplicate deliveries suppressed (healthy and rising = redelivery happening and being absorbed) separately from effects applied twice (must be 0). Alarm on redelivery rate > 1% or DLQ depth > 0.
A redelivery contract test in CI. Deliver the same message twice — once with an injected crash between effect and ack — and assert exactly one ledger row and exactly one wire.

The lesson

At-least-once delivery is a promise the consumer pays for: the broker will sometimes hand you the same message twice, so your handler must make repeating a message a no-op. You cannot buy exactly-once delivery across a boundary the broker can't transact over — but you can build exactly-once effect with a dedupe store, an upsert, and a transactional outbox.

Named concepts: at-least-once delivery, idempotent consumers, transactional outbox, ack semantics.

Real-world echoes

At-least-once queues (SQS, Kafka, RabbitMQ) double-processing. Every major managed broker documents at-least-once as its default and tells you, in the same paragraph, that consumers must be idempotent. Double-charges and duplicate side effects after a rebalance or a visibility-timeout expiry are a recurring post-incident theme precisely because teams read the guarantee and skip the obligation it hands them.

Transfer

Which of your queue consumers would apply a side effect twice if the broker redelivered the current in-flight message right now — a rebalance, a pod eviction, a missed ack? "It probably wouldn't happen" is not an answer; name the dedupe key or the upsert that makes the replay a no-op.

Make it a ticket: inventory every consumer that performs a side effect; for each, add a message-id dedupe store, convert own-database writes to upserts on the natural key, route external effects through a transactional outbox with a rail idempotency key, and add a redelivery contract test — starting with anything that moves money.

Drill 81 — The Unbounded Queue: the other half of queue discipline, on the producer/backpressure side of the same broker.
Drill 69 — The Double-Charge Race: the same idempotency lesson under concurrency instead of redelivery.
Drill 13 — idempotent side effects; the general principle this drill applies to message consumers.
Pattern #109 — Idempotency Keys, and Pattern #130 — Transactional Outbox (the reusable fixes).