Drill 69 — The Double-Charge Race

Theme: Concurrency & Race Conditions · Format: 20-min drill · Difficulty: core

The story

A consumer lender's payment service has one job on the hot path: when a borrower taps Make Payment, it checks whether that installment has already been paid, and if not, it charges the card processor and writes a row to the payments table. It has worked, unremarkably, for three years.

One evening a borrower on a flaky train connection taps Pay. The spinner hangs. They tap again. Two requests reach the service about eighty milliseconds apart — and the service runs two instances behind a load balancer, so the requests land on different boxes. Both read the payments table. Both see no payment recorded for this installment. Both call the processor. Both succeed. Both insert a row. The borrower has been charged twice for one payment.

The same shape arrives a different way in the logs the next morning: a mobile client whose request succeeded but whose response was lost to a dropped connection, and which dutifully retried — charging a second time for a request that had already worked.

⏸ Stop here and ask the room: the code checked whether the payment existed before charging. So how did it charge twice?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

What exactly is the race — name the window.

Listen for: check-then-act / TOCTOU. The gap between reading "not paid" and writing the payment is not atomic; under READ COMMITTED, a second transaction runs the same read and sees the same "not paid" because the first hasn't committed. Two requests both pass the check before either writes.
Weak signal: "add a mutex" with no mention of the window, or "the database should just handle it" without saying how.

Where should the guarantee live — the app or the database?

Listen for: the database is the single source of truth; a unique constraint on the natural key (loan_id, installment_id) makes the second write impossible no matter how the app races or how many instances run. The app-level check is advisory — a fast path, not a guarantee. An idempotency key dedupes retries.
Weak signal: "lock in the application" (doesn't survive multiple instances/restarts) or "check more carefully" (you cannot check your way out of a race).

The double-click and the lost-response retry — same bug or two bugs?

Listen for: one bug — "the same operation executed more than once." An idempotency key sent by the client covers both; the unique constraint is the backstop. Any network with timeouts will replay requests, so every mutating endpoint must be idempotent.
Weak signal: handling only the double-click, or treating retries as an edge case rather than a guarantee to design for.

The reveal

The fix was not "check better" — you cannot win a race with a check. They added a unique index on (loan_id, installment_id, idempotency_key); the second insert now fails as a constraint violation, which the code catches and treats as already done, returning the original receipt. Double-charges went to zero. The app-level pre-check stayed — but only as a latency optimization, never again as the guarantee.

Why this happens — the mechanics

Check-then-act is not atomic. Between the SELECT that returns "no payment" and the INSERT that records one, another transaction can run the identical SELECT and see the identical "no payment," because neither has committed and READ COMMITTED isolation hides uncommitted writes. The application-level guard is a suggestion; only a database invariant makes "at most once" true under concurrency — a unique constraint, or SELECT … FOR UPDATE holding a row lock, or an atomic INSERT … ON CONFLICT.

Retries make it inevitable rather than unlucky. Any network with timeouts replays requests: a lost response looks identical to a failure, so a correct client retries, and a non-idempotent endpoint double-executes. This is Drill 13's idempotency lesson wearing a concurrency costume — and it is why the guarantee has to live at the layer that sees all the writes.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant C as Client
    participant A1 as API (inst. 1)
    participant A2 as API (inst. 2)
    participant DB as Payments DB
    participant P as Processor
    C->>A1: Pay installment (tap 1)
    C->>A2: Pay installment (tap 2, ~80ms later)
    A1->>DB: SELECT — already paid?
    A2->>DB: SELECT — already paid?
    DB-->>A1: no payment
    DB-->>A2: no payment
    A1->>P: charge card
    A2->>P: charge card
    A1->>DB: INSERT payment
    A2->>DB: INSERT payment (double charge)
    Note over A1,A2: both passed the check before either wrote

Solution — where the guardrail goes:

flowchart TD
    A["Pay request + idempotency key"] --> B["INSERT ... ON CONFLICT<br/>on (loan_id, installment_id, key)"]
    B -->|"new row"| C["Charge processor · return receipt"]
    B -->|"conflict"| D["Already done — return the<br/>ORIGINAL receipt, not a 500"]
    C --> E["Unique constraint = the guarantee"]
    D --> E

What good looks like

A unique constraint on the natural key (loan_id, installment_id) (plus an idempotency-key column) — the DB enforces once-only regardless of app races or instance count.
Idempotency keys on every mutating request; the server stores key → result with a TTL and replays the first result on any repeat.
The write is atomic: INSERT … ON CONFLICT DO NOTHING / upsert, or a SELECT … FOR UPDATE inside one transaction — never a separate check then act.
The constraint violation is caught and treated as success (return the original receipt), not surfaced as an error.
A metric: duplicate attempts caught by the constraint (healthy and rising = clients retrying, which is fine) vs charges actually double-applied (must be zero).
A concurrent-duplicate contract test in CI: fire the same request twice in parallel and assert exactly one charge.

The lesson

A check before an action is a suggestion; only a database invariant is a guarantee. Anything that can run twice — a double-click, a lost-response retry, a redelivered message — must be idempotent, and the enforcement belongs at the layer that sees every write.

Named concepts: check-then-act (TOCTOU), idempotency keys, unique constraints, optimistic concurrency.

Real-world echoes

Stripe's idempotency-key API. Stripe made idempotency keys a first-class part of its payments API precisely because network retries would otherwise double-charge; the client sends a key and Stripe guarantees the operation runs at most once. It is the productized version of this drill's fix.

Transfer

Which of your mutating endpoints would double-execute if the client retried the request right now? "It probably wouldn't" doesn't count — name the invariant that stops it.

Make it a ticket: inventory every write endpoint; for each, add an idempotency key plus a unique constraint on its natural key, and add a concurrent-duplicate test — starting with anything that moves money.

Drill 13 — idempotent side effects; this is that lesson under concurrency.
Drill 20 — the retry loop; unbounded repetition of a non-idempotent write.
Pattern #109 — Idempotency Keys (the reusable fix).