Drill 86 — The Ordering That Wasn't Guaranteed

Theme: Queues & Backpressure · Format: 20-min drill · Difficulty: core

The story

A broker-dealer's post-trade settlement pipeline consumes a stream of lifecycle events for every trade: SETTLEMENT_CREATED, then zero or more SETTLEMENT_AMENDED, and sometimes a SETTLEMENT_CANCELLED. A trading system publishes those events to a message log; a fleet of consumers reads them and drives each trade through a small state machine — created → amended → settled, or created → cancelled. It has run clean for two years. The team's mental model is simple and, they believe, obviously true: events arrive in the order they happened.

To handle volume, the log is split across twelve partitions and eight consumer instances read them in parallel. Nobody thinks hard about which event lands on which partition — the producer was configured long ago to spread load evenly, round-robin, so no single partition gets hot. And because the network occasionally hiccups, the producer retries any publish it isn't sure landed, and consumers retry any event they fail to process.

One Tuesday a trader books a trade and cancels it four hundred milliseconds later — a fat-finger, caught fast. Two events, create then cancel, four hundred milliseconds apart. But round-robin put the create on partition 7 and the cancel on partition 2, and partition 2's consumer happens to be idle while partition 7's is chewing through a backlog. The cancel is processed first. The state machine gets SETTLEMENT_CANCELLED for a trade it has never heard of. A moment later the SETTLEMENT_CREATED arrives and the trade is dutifully created — live, open, settling — a trade the desk cancelled, now marching toward the settlement window.

⏸ Stop here and ask the room: the events were published in the right order, four hundred milliseconds apart. So how did the cancel get processed before the create?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

The producer published create-then-cancel. Where did the order get lost?

• Listen for: a partitioned log only guarantees order within a partition, not across the topic. Round-robin scatters two events for the same trade onto different partitions, read by different consumers at different speeds — so global "happened-before" order is simply not preserved. Parallel consumers plus retries make relative delivery order effectively arbitrary.
• Weak signal: "the queue must have reordered them" or "it's a race in the consumer" — without naming that ordering was never guaranteed across partitions in the first place.

What makes this worse than a transient glitch — why does the state stick?

• Listen for: the state machine accepted an impossible transition. A CANCELLED for a non-existent trade should be rejected, but the code auto-created or upserted whatever it saw, so a late CREATED resurrected a cancelled trade into a live one. The bad state is durable and silent — no error, just a wrong row heading to settlement.
• Weak signal: "just add a retry" or "eventually it'll be consistent" — eventual consistency of the wrong terminal state is still wrong.

Where should the ordering guarantee actually live?

• Listen for: two complementary places. First, make same-entity events land in order — partition by the trade/settlement key so all of one trade's events go to one partition, consumed in sequence. Second, make the consumer robust to disorder anyway — sequence numbers + a reorder buffer, commutative/idempotent operations, and a state machine that rejects invalid transitions. You need both: routing for the common case, defensive logic for retries and edge cases.
• Weak signal: "add a global lock" or "process everything on one consumer" — throwing away the parallelism instead of scoping ordering to the key that needs it.

The reveal

Ordering was never guaranteed — the team had confused "published in order" with "delivered in order," and a partitioned log only ever promised the former within a single partition. The fix was layered. They partitioned by settlement_id so every event for one trade is totally ordered on one partition. They added monotonic sequence numbers per key and a small reorder buffer so a consumer applies events in sequence and holds a gap briefly rather than applying out of order. They made the write commutative (version-stamped, last-writer-wins) so a replay can't clobber newer state. And they hardened the state machine to reject invalid transitions — a CANCELLED for an unknown trade is no longer auto-created; it is rejected and dead-lettered. Resurrected trades went to zero.

Why this happens — the mechanics

A partitioned message log gives you exactly one ordering guarantee: total order within a partition, no order across partitions. That is the whole deal — it is how the log scales, by letting independent partitions be read in parallel. The moment you route two causally-related events (a create and its cancel) to different partitions, you have thrown away their relative order, because two consumers reading two partitions run at independent speeds. Add retries — a redelivered or reprocessed event can land arbitrarily late — and "the order things happened" and "the order the consumer sees them" become different sequences. Round-robin partitioning is the trap: it optimizes for even load and silently sacrifices per-entity ordering, which for most events is fine and for lifecycle events is catastrophic.

The second half of the failure is the state machine trusting its input. A create-then-cancel that arrives cancel-then-create is only dangerous if the consumer accepts a cancel for a trade it has never seen. A strict transition guard — "you may only cancel a trade that exists and is in a cancellable state" — turns an ordering bug into a rejected, dead-lettered, observable event instead of a silent corruption. Ordering guarantees at the transport and defensive logic at the consumer are not alternatives; the transport reduces how often disorder happens, and the consumer makes disorder harmless when it does.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant PR as Producer
    participant CA as Consumer A
    participant CB as Consumer B
    participant SM as State machine
    PR->>CA: CREATE trade 4821<br/>(partition 7)
    PR->>CB: CANCEL trade 4821<br/>(partition 2)
    Note over CA,CB: two partitions,<br/>no shared order
    CB->>SM: apply CANCEL first
    Note over SM: no such trade -<br/>auto-created instead
    CA->>SM: apply CREATE later
    SM->>SM: trade now OPEN
    Note over PR,SM: a cancelled trade<br/>is live and settling

Solution — where the guardrails go:

flowchart TD
    A["Lifecycle events<br/>(create, amend, cancel)"] --> B["Partition by<br/>settlement_id key"]
    B --> C["Reorder buffer:<br/>apply in sequence order"]
    C --> D["State machine:<br/>guard each transition"]
    D -->|"valid"| E["Apply · advance state"]
    D -->|"invalid"| F["Reject to D.L.Q."]
    E --> G["Commutative write:<br/>version wins, replay-safe"]
    C -.->|"gap over 30s"| AL["Alarm: out-of-order<br/>or missing sequence"]
    F -.-> AL

What good looks like

• Partition by the entity key, never round-robin, for any event whose order matters: hash(settlement_id) % partitions so every event for one trade is totally ordered on one partition. Ship a guardrail metric — count of entity keys observed on more than one partition — that must be 0.
• Monotonic sequence numbers per key, stamped by the producer, plus a bounded reorder buffer on the consumer: apply strictly in sequence, hold an out-of-order event for a gap window (e.g., 30s or 1,000 events), and page if the gap doesn't close.
• Commutative / idempotent writes: version-stamp every state write and apply last-writer-wins by version so a replayed or late event can never overwrite newer state; reprocessing the same event is a no-op.
• A state machine that rejects invalid transitions: an explicit transition table where CANCEL requires an existing trade in {CREATED, AMENDED}; anything else is rejected and dead-lettered, never auto-created. Alarm if invalid-transition rate rises above 0 per hour.
• An out-of-order SLI: fraction of events applied out of sequence, and D.L.Q. depth; alarm on the first non-zero reading, because ordering bugs are silent until they aren't.
• A chaos test in CI: publish create-then-cancel for one key onto two partitions with an injected consumer delay and assert the trade ends cancelled, not open.

The lesson

A partitioned log orders events within a partition, never across the topic — so "published in order" is not "delivered in order." Route same-entity events to one partition by key, and make the consumer robust to disorder anyway with sequence numbers, commutative writes, and a state machine that rejects impossible transitions.

Named concepts: partition-key ordering, reorder buffer with sequence numbers, commutative and idempotent operations, state-machine transition guards.

Real-world echoes

• Kafka's per-partition ordering guarantee. Kafka is explicit that messages are ordered only within a partition and that ordering across partitions is not guaranteed — which is exactly why keyed partitioning exists. Teams that partition by key preserve per-entity order; teams that round-robin for even load quietly forfeit it, and lifecycle streams are where that bites.

Transfer

Which of your streams carry causally-ordered events for the same entity — and are they partitioned by that entity's key, or round-robined for load? "It's probably fine" doesn't count — name the key that keeps one entity's events in order, and the transition guard that catches it if they aren't.

Make it a ticket: inventory every topic that carries lifecycle/state-change events; for each, partition by the entity key, add a state-machine transition guard that dead-letters invalid transitions, and add a create-then-cancel-out-of-order chaos test — starting with anything that moves money.

Related

• Drill 81 — The Unbounded Queue (sibling in this theme; the other way a queue's hidden assumption turns fatal).
• Drill 69 — The Double-Charge Race (idempotency and commutativity as the defense against replays).
• Drill 13 — the backfill that rewrote history (order and idempotency of writes under reprocessing).
• Pattern #109 — Idempotency Keys (the reusable replay-safety fix).