BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 74 — The Out-of-Order Callback

Theme: Concurrency & Race Conditions · Format: 20-min drill · Difficulty: core

The story

Your payments platform sits between merchants and a card processor. The processor doesn't answer inline — it sends asynchronous webhook callbacks as a charge moves through its lifecycle: authorized, then captured, then settled. A small consumer reads each callback off the queue and updates the order's state column. Authorize the card, the order goes to authorized; capture the funds, it goes to captured. Downstream, settlement and the merchant payout both key off that one column. It has run clean for two years.

One afternoon the processor has a brief blip delivering the authorized callback for order A-8842 — the delivery times out, so the processor does the correct thing and queues it for retry. Meanwhile the charge keeps moving: the captured callback for the same order is generated a few seconds later and delivered on the first try. Your consumer receives captured, writes state = captured, and the order is, correctly, captured. The money is on its way.

Ninety seconds later the retried authorized callback finally lands. The consumer does exactly what it has always done: it reads the event, sees authorized, and writes state = authorized. The order that was captured is now, according to your database, merely authorized — waiting on a capture that already happened. Nothing throws. The callback returns 200 OK. The consumer's dashboards are green. But settlement now believes A-8842 was never captured, the merchant payout for it is silently held, and the processor's ledger and yours disagree about real money.

⏸ Stop here and ask the room: every callback was processed successfully and returned 200 OK. So how is the order in the wrong state?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

The consumer processed every event without error. Where's the bug?

What does the consumer need to know about an event besides its type?

Is rejecting the stale event enough, or does the update logic itself need to change?

The reveal

The consumer had one line that mattered: UPDATE orders SET state = :event_type WHERE id = :order_id. It trusted arrival order as event order. The processor's own docs said, in a sentence nobody had internalized, that callbacks are not guaranteed to arrive in the order the events occurred, and that retries can redeliver an older event at any time.

The fix was to stop treating state as a value you overwrite and start treating it as a versioned state machine. Each event carries a per-order sequence number and an event timestamp; the consumer applies an event only if its sequence is greater than the last applied, and only if the transition is legal for the current state. The late authorized now fails the sequence check and the transition check, so it's logged and dropped. A one-off reconciliation job re-derived state from the processor's ordered event log and healed the orders that had already been corrupted. State regressions went to zero.

Why this happens — the mechanics

Asynchronous callbacks give you at-least-once, unordered delivery. The network can reorder, the queue can reorder across partitions, and — the part teams forget — retries reorder by construction: a message that failed and is redelivered arrives after messages that were generated later but succeeded on the first try. So "the last event I received" is not "the latest event that happened." Any consumer that does last-write-wins keyed on arrival is one delivery blip away from writing a stale value over a fresh one.

The organizational seam is that the processor owns ordering (it knows the true sequence) while your consumer owns state (it decides what the order column says), and nothing in between carried the ordering information across. The event had a type but no version. Once state is just a mutable value with no notion of "how new is this," out-of-order delivery is indistinguishable from in-order delivery — until money moves. This is Drill 69's lesson turned inside out: there the danger was the same operation running twice; here it's a stale operation running last. Both are cured by making the write express an invariant — a unique key there, a monotonic version and a legal-transition guard here — instead of blindly obeying whatever arrived.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant P as Processor
    participant Cn as Consumer
    participant DB as Order DB
    P--xCn: authorized seq 1<br/>delivery times out
    Note over P,Cn: retry queued,<br/>redelivered later
    P->>Cn: captured seq 2
    Cn->>DB: set state<br/>captured
    P->>Cn: authorized seq 1<br/>redelivered late
    Cn->>DB: overwrite to<br/>authorized, stale
    Note over DB: captured silently lost,<br/>200 OK, no error

Solution — where the guardrail goes:

flowchart TD
    A["Callback arrives<br/>(any order)"] --> B["Read event seq<br/>and timestamp"]
    B --> C{"seq greater than<br/>stored seq?"}
    C -->|"no, older"| D["Drop as stale,<br/>state unchanged"]
    C -->|"yes"| E{"legal state<br/>transition?"}
    E -->|"no"| D
    E -->|"yes"| F["Apply update,<br/>store new seq"]
    D -.-> AL["Alert on invalid or<br/>stale callback rate"]
    F --> G["Order state<br/>converges"]

What good looks like

The lesson

Arrival order is not event order — async delivery reorders, and retries reorder by construction. Don't overwrite state with whatever message arrived last; version it, and let a state machine reject anything that would move it backward.

Named concepts: message ordering, event versioning, idempotency, state machines.

Real-world echoes

Transfer

Which of your event or webhook consumers would corrupt state if two callbacks for the same entity arrived in the wrong order right now? "They usually arrive in order" is not an answer — name the version or sequence field that would stop the stale one.

Make it a ticket: pick your highest-stakes async consumer; add a per-entity sequence/version column, guard every state write with a monotonic WHERE seq < :seq check plus a legal-transition guard, and add a shuffled-redelivery test that asserts the final state is correct.

Related