BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 70 — The Lost Update

Theme: Concurrency & Race Conditions · Format: 20-min drill · Difficulty: core

The story

A retail bank's back-office team fixes account balances by hand every day. When a payment posts wrong, or a fee is reversed, or a wire needs a correction, an operator opens the customer's account in the ops console, reads the current balance, types the new figure, and saves. The console does exactly what it looks like it does: it SELECTs the balance into a form, the operator edits the number, and on Save it UPDATEs the row to whatever is now in the field. Simple, auditable, three years without a complaint.

This morning two corrections land on the same account within the same minute. Account 4471 shows a balance of 100. Priya opens it to apply a +50 credit — a mistakenly-clawed-back rewards payment. Marcus, two desks over, opens the same account to apply a −30 debit — a late-arriving card fee. Both consoles read 100. Priya types 150 and saves at 09:14:02. Marcus types 70 and saves at 09:14:05. Each save is a clean, successful UPDATE; each operator sees a green toast and moves on. The account now reads 70.

The correct answer was 120 — a hundred, plus fifty, minus thirty. Priya's +50 is simply gone. Not rejected, not queued, not flagged — overwritten, because Marcus's Save wrote 70 on top of it a heartbeat later. Nothing errored. Both tickets are marked done. The daily reconciliation totals still balance against themselves, because the ledger's own arithmetic is internally consistent — it just no longer matches reality. Three weeks later a customer calls asking where their fifty pounds went, and nobody can find a failure to point at.

⏸ Stop here and ask the room: both saves succeeded and both operators did their jobs correctly. So where did the fifty pounds go?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Both writes succeeded — so what exactly was lost, and when?

The console reads, the human edits, the console writes the absolute value. Where's the design flaw?

How would you make the second save either merge correctly or refuse — never silently win?

The reveal

The fix was not "let operators refresh before saving." They changed the write. Corrections now post as deltas through an atomic statement — UPDATE accounts SET balance = balance + :delta WHERE id = :id — so Priya's +50 and Marcus's −30 both land and the order stops mattering; the account settles to 120. For the edit-a-known-value cases that must stay absolute, the console added an optimistic version check: every row carries a version integer, Save issues UPDATE ... SET balance = :new, version = version + 1 WHERE id = :id AND version = :seen, and if it updates zero rows the UI shows "this changed under you" and reloads instead of overwriting. Underneath both, corrections are now written to an append-only ledger table — one immutable row per adjustment — and the balance is a materialized sum, so no adjustment can ever be silently lost again. The recovered audit found nine other quiet lost updates over the prior year.

Why this happens — the mechanics

A read-modify-write is three steps — read the balance, compute a new one, write it back — and under READ COMMITTED isolation nothing stops a second transaction from reading the same starting value before the first commits. Both operators read 100. Both compute a fresh absolute number from that 100. Whichever UPDATE commits last wins the row outright, and the earlier commit's effect is erased with no error, because the database was asked to set a value, not to conditionally adjust one. This is the textbook lost update: two updates based on the same read, one silently overwritten. It is the same TOCTOU window as Drill 69's double-charge — but where a double-charge does the operation twice, a lost update does it zero-net times for one of the writers.

The organizational reason it hid for three years is that every individual action looks correct. There is no exception, no failed transaction, no orphaned row — the pathology is an absence, a write that should have survived and didn't. Internal reconciliation passes because the ledger is arithmetically consistent with itself; only a customer or an external statement reveals the drift. Absolute-value writes are the trap: balance = 70 bakes in an assumption ("the balance is still 100") that the database is never asked to verify. Deltas and compare-and-swap both make that assumption explicit — either by commuting (+ delta applies no matter the current value) or by failing loudly when the assumption is stale (WHERE version = :seen updates nothing).

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant P as Priya
    participant M as Marcus
    participant DB as Accounts DB
    P->>DB: read balance
    M->>DB: read balance
    DB-->>P: balance = 100
    DB-->>M: balance = 100
    Note over P: computes 100 + 50<br/>types 150
    Note over M: computes 100 - 30<br/>types 70
    P->>DB: SET balance = 150
    M->>DB: SET balance = 70
    Note over DB: last write wins<br/>Priya's +50 vanishes
    M--xP: no error, no flag<br/>ledger now drifts

Solution — where the guardrail goes:

flowchart TD
    A["Correction request<br/>(id, delta, seen version)"] --> B{"Absolute edit or<br/>an adjustment?"}
    B -->|adjustment| C["UPDATE SET balance =<br/>balance + delta (atomic)"]
    B -->|absolute| D["UPDATE ... WHERE<br/>version = seen (CAS)"]
    D -->|"0 rows updated"| E["Stale — reload and retry"]
    C --> L["Append-only ledger<br/>one row per adjustment"]
    D --> L
    E -.-> AL["Alert: lost-update<br/>retries per minute"]
    L -.-> AL

What good looks like

The lesson

A read-modify-write that writes an absolute value assumes nothing changed since you read — and under concurrency that assumption is a silent bug. Express changes as commuting deltas, or make the write conditional on the version you read, so a stale write either merges or fails loudly but never wins by default.

Named concepts: lost update, optimistic locking, atomic increments, append-only ledger.

Real-world echoes

Transfer

Where in your stack does a human or a service read a value, compute a new absolute figure, and write it back — a balance, a counter, an inventory count, a status field? For each, ask: if two of those happen at once, does the second merge, refuse, or silently win? "It probably won't happen at once" doesn't count — name the mechanism that makes the loser fail loudly.

Make it a ticket: find every read-modify-write on a shared mutable field; convert additive ones to atomic balance + :delta increments and edit-forms to version-checked compare-and-swap, and add a parallel-write test asserting both updates survive — starting with anything that touches money.

Related