Pattern — Idempotency Keys

The problem it solves

The same failure shows up in three costumes: a double-tapped payment charges a card twice (Drill 69), a redelivered webhook overwrites fresh state with a stale one (Drill 74), and a redelivered queue message posts a ledger row twice (Drill 85). The common shape: any network with retries replays writes — a dropped response, an at-least-once broker, a pod evicted between effect and ack — so an operation you ran once runs again, and a non-idempotent handler applies the effect twice.

The pattern

Give every logical operation a stable identifier — an idempotency key — that travels with the request and all its retries. The server records key → result exactly once, backed by a database uniqueness invariant; a repeat is recognized structurally and replays the first result instead of re-running the effect. You cannot buy exactly-once delivery across a boundary the broker can't transact over, but you can build exactly-once effect on your side: the client names the work, the database guarantees it happens once.

How to apply it

Client generates one key per operation. A random UUID v4, sent in an Idempotency-Key request header (Stripe's exact convention). Every retry of the same logical action reuses the same key; a genuinely new action gets a new one. Never derive the key server-side from arrival time — that defeats retry dedup.
Put the guarantee in the database, not the app. A UNIQUE constraint on the natural key plus the key — e.g. (loan_id, installment_id, idempotency_key) for a payment, or (payment_id, account) for a ledger post. An app-level "did we already do this?" check is only a fast-path hint; under concurrency only the DB invariant makes at-most-once true.
Write atomically, and treat a conflict as success. Use INSERT … ON CONFLICT DO NOTHING (or SELECT … FOR UPDATE in one transaction) — never a separate check-then-act. On a conflict, don't throw: look up and return the original result (receipt / 200), so a duplicate collapses to a no-op replay.
Bound the state with real numbers. Keep the key → result store for at least the maximum redelivery window — 7 days is a safe default — so no in-flight retry is ever forgotten. On a queue, set the visibility timeout above p99 processing time (e.g. 30s over an 8s p99) so slow-but-alive work isn't redelivered, and cap redeliveries at 5 before routing to a dead-letter queue so a poison message can't loop forever. For external side effects, route through a transactional outbox and pass the same key as the downstream rail's idempotency key.

flowchart TD
    A["Request + Idempotency-Key"] --> B["INSERT ... ON CONFLICT<br/>on (natural_key, idem_key)"]
    B -->|"new row"| C["Run the effect once,<br/>store key to result"]
    B -->|"conflict (seen before)"| D["Replay the ORIGINAL<br/>result, not an error"]
    C --> E["Exactly-once effect"]
    D --> E

When to use it — and when not

Reach for it on every mutating endpoint or consumer that moves money or state and can be retried. Skip it where it earns nothing: read-only or naturally idempotent operations (a pure PUT that sets an absolute value) need no key. And know its limit — a key stops repeats, not reordering. A stale but different event (Drill 74's late authorized landing after captured) is not a duplicate, so idempotency alone won't catch it; pair the key with a monotonic version guard (WHERE seq < :seq) and a legal-transition check. The cost is a durable store and a key on every write, so don't sprinkle it on high-cardinality, low-stakes traffic where the storage and the discipline aren't paid back.

Signals it's working

Track two counters separately: duplicate deliveries suppressed (healthy and rising — redelivery is normal and now absorbed) versus effects applied twice (must hold flat at 0). Alarm when the duplicate/redelivery rate crosses 1% for 15 min (an upstream retry storm) or DLQ depth goes above 0. Prove it in CI with a redelivery contract test: fire the same request twice in parallel — once with an injected crash between effect and ack — and assert exactly one row and one side effect.

Seen in the wild

Stripe made the Idempotency-Key header a first-class part of its payments API precisely so network retries can't double-charge.
At-least-once brokers — Amazon SQS, Apache Kafka, RabbitMQ — all document at-least-once as the default and, in the same paragraph, tell you consumers must be idempotent.
PayPal, Adyen, and most payment rails expose a request-id / idempotency-key so a redelivered instruction settles once.

See it

The ChainScene carries the mechanism: INSERT … ON CONFLICT → conflict → return the original receipt → double-charges = 0; the CardsScene shows the load-bearing config (7-day store TTL, 30s visibility over 8s p99, max 5 → DLQ).

Named concepts: idempotency keys, unique constraints, idempotent consumers, exactly-once effect.