BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 72 — The Check-Then-Act Gap

Theme: Concurrency & Race Conditions · Format: 20-min drill · Difficulty: core

The story

A lending desk funds its originations from a revolving warehouse facility — a $50M line from a bank that the platform draws against every time it disburses a loan. The disbursement service has one rule it must never break: never draw more than the facility has left. So on the hot path it does the obvious, careful thing. It reads the facility's remaining headroom, checks that the loan amount fits, and only then wires the money and decrements the line. It has funded thousands of loans this way without a scratch.

One Tuesday afternoon two large draws arrive within the same second — a $6M commercial loan and a $6M bridge loan, approved by two different underwriters, submitted from two different queues. The facility has exactly $10M of headroom at that moment. Both requests hit the disbursement service, which runs four replicas behind a load balancer, so the two draws land on two different boxes. Both replicas read the ledger. Both see $10M available. Both check 6M <= 10M — true. Both wire the money. Both decrement the line. The facility, which had a hard $50M ceiling, is now drawn to $52M.

Nobody gets an error. Both loans disburse cleanly, both borrowers get their funds, both API calls return 200 OK. The overdraw surfaces the next morning when the bank's reconciliation file rejects the facility position for breaching its covenant — a $2M breach that took two correct-looking requests and one shared second to create.

⏸ Stop here and ask the room: every draw checked that the money was there before spending it. The check passed. So how is the facility $2M over its limit?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Name the window — where exactly is the race?

Where must the guarantee live — the app, or the database?

Two draws, four replicas — would one replica have saved you?

The reveal

The fix was not a better check — it was to stop checking and acting as two steps. They replaced the read-then-decrement with a single atomic statement: UPDATE facility SET available = available - :amt WHERE facility_id = :id AND available >= :amt. If it updates one row, the draw won the line and disbursement proceeds; if it updates zero rows, the line is gone and the draw is rejected — no wire sent. They added a CHECK (available >= 0) constraint as a hard backstop so no code path, ever, can push the facility negative. Two concurrent $6M draws against $10M now resolve to exactly one winner and one clean rejection. Overdraws went to zero.

Why this happens — the mechanics

Check-then-act is not atomic. Between the SELECT that returns "$10M available" and the UPDATE that spends $6M of it, another transaction can run the identical SELECT and see the identical "$10M," because neither has committed and READ COMMITTED isolation hides uncommitted writes. The application-level guard is a suggestion evaluated against a snapshot that is already stale by the time you act on it. Only an operation that reads and writes as one indivisible step — a compare-and-swap — makes the decision and the mutation agree under concurrency: the conditional UPDATE, a SELECT ... FOR UPDATE that holds a row lock across the read and the write, or a database CHECK constraint that refuses the illegal end state outright.

The organizational reason competent teams ship this is that the two-step version reads more carefully. "Check that there's room, then spend" looks like the responsible thing to do; the atomic one-liner looks almost reckless by comparison because the safety is invisible, delegated to the WHERE clause and the constraint. So the careful-looking code is the buggy code, and it passes every single-threaded test. The race only appears when two draws share a second — which, at any real volume, they eventually will.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant C as Borrowers
    participant D1 as Disburse r1
    participant D2 as Disburse r2
    participant L as Facility ledger
    participant B as Bank rail
    C->>D1: Draw 6M<br/>(req A)
    C->>D2: Draw 6M<br/>(req B)
    D1->>L: read headroom
    D2->>L: read headroom
    L-->>D1: 10M available
    L-->>D2: 10M available
    Note over D1,D2: both pass<br/>6M <= 10M
    D1->>B: wire 6M
    D2->>B: wire 6M
    D1->>L: available -= 6M
    D2->>L: available -= 6M
    Note over L: facility now 52M<br/>over the 50M limit

Solution — where the guardrail goes:

flowchart TD
    A["Draw request<br/>(amount = 6M)"] --> B["UPDATE facility SET<br/>available = available - amt<br/>WHERE available >= amt"]
    B -->|"1 row updated"| C["Commit +<br/>wire the funds"]
    B -->|"0 rows updated"| D["Reject —<br/>line exhausted"]
    L["Pessimistic lock<br/>SELECT ... FOR UPDATE"] -.-> B
    K["CHECK<br/>(available >= 0)"] -.-> AL(["Alarm — negative<br/>balance blocked"])
    C --> K
    D -.-> AL

What good looks like

The lesson

A check and the action it guards must be a single atomic step, or a second actor will slip through the gap between them. Enforce the invariant where the state lives — a compare-and-swap update and a CHECK constraint — never with a read followed by a hopeful write.

Named concepts: TOCTOU, compare-and-swap, atomic reservation, invariants.

Real-world echoes

Transfer

Where in your stack do you read a balance, a quota, a seat count, or a rate limit, then act on it in a separate step? "The window is tiny" is not an answer — name the atomic operation or the constraint that makes the overshoot impossible.

Make it a ticket: find every read-then-decrement on a limited resource; convert the hottest one to an atomic UPDATE ... WHERE remaining >= :amt plus a CHECK (remaining >= 0) constraint, and add a two-concurrent-draws test that asserts the resource never goes negative.

Related