BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Pattern — Saga

The problem it solves

You have to change state in more than one place — two databases during a migration, or inventory + payments + shipping across three services — and there is no single transaction that spans them. The naive answer is a dual write: write store A, then write store B, and hope both land. They don't. In Drill 55 — The Dual-Write Drift, a months-long migration wrote every record to the old store and the new one, with the new-store write wrapped in a catch block "so the migration can never hurt production." A serialization bug silently failed about 2% of new-store writes; the old store served all reads and looked perfect, so five months of divergence accumulated invisibly and surfaced as customers missing payment history. The deeper failure shape: a multi-step distributed write has no atomicity, so a partial completion is a legal, silent, unrecoverable outcome — step 1 commits, step 2 fails, and nothing on the system knows the operation is half-done.

The pattern

A saga models a distributed operation as a sequence of local ACID transactions, where each step has a matching compensating transaction that semantically undoes it. A durable saga log records which steps have committed; a coordinator drives every saga to a terminal state — either all steps commit (forward completion) or, on failure, the already-committed steps are undone in reverse (backward recovery). You give up cross-store isolation — a saga is ACD without the I, so intermediate states are briefly visible — in exchange for the one guarantee dual-write can't give: no operation is left half-applied and unaccounted for. The swallowed catch block becomes a persisted PENDING state that a recovery process is obligated to resolve.

How to apply it

Grounding the mechanics in Drill 55 (model "write old store, then new store" as a two-step saga; the general form uses reserve-inventory → charge-payment → create-shipment):

  1. Persist saga state before every step, not after. Write a row to saga_instances(saga_id, type, current_step, status, payload) and flip status and current_step inside the same local transaction as each step's business write. Status is a strict machine: STARTED → PENDING(step_i) → COMPLETED or → COMPENSATING → COMPENSATED / FAILED. If the process dies mid-step, the log — not memory — is the source of truth on restart.
  2. Prefer orchestration for anything over ~3 steps. A central orchestrator (Temporal, Step Functions) owns the state machine, calls each step, and is trivially auditable — you can query "how many sagas are stuck at step 2." Choreography (each service reacts to the previous one's event, no central brain) has lower coupling but the workflow is emergent and hard to trace; reserve it for 2-participant flows.
  3. Bound each step and retry the retriable ones. Give each step a 30 s timeout and retry transient failures with exponential backoff — base 1 s, ×2, ±20% jitter, max 5 attempts. Steps must be idempotent (carry a step key) because a retry may replay a write that actually succeeded.
  4. On exhausted retries, run backward recovery. Execute the compensations for every committed step in reverse order (release-inventory ← refund-payment ← cancel-shipment). Compensations must themselves be idempotent and retried effectively forever — a compensation that gives up leaves exactly the orphaned half-state you were trying to prevent. Retry a failing compensation with backoff up to 24 h, then route to a DLQ and page a human.
  5. Name the pivot transaction. The step after which the saga must only go forward (a compensation is impossible or undesirable — an email is already sent, funds already captured). Before the pivot, failure ⇒ compensate; after it, failure ⇒ retry-forward until it completes.
  6. Add a countermeasure for the missing isolation. Because other readers can see a saga's intermediate state, apply a semantic lock: mark touched records status = PENDING so readers skip or wait on them, and clear it at terminal state. Alternatives: commutative updates, or re-read-then-check (optimistic version guard).
  7. Run a reaper. A sweeper scans every 60 s for sagas in a non-terminal state older than their step SLA and resumes or compensates them — this is the standing job that turns "silently stuck" into "detected and driven to done," the direct analog of Drill 55's reconciliation-until-zero.
flowchart TD
    A["Start saga · persist STARTED"] --> B["Step 1: reserve inventory<br/>(local txn + log)"]
    B --> C["Step 2: charge payment<br/>(local txn + log)"]
    C -->|"success"| D["Step 3: create shipment<br/>= pivot, go forward only"]
    C -->|"fail after 5 retries"| E["COMPENSATING: refund,<br/>then release inventory"]
    E --> F["COMPENSATED (terminal)"]
    D --> G["COMPLETED (terminal)"]
    H["Reaper every 60s: resume or<br/>compensate stuck sagas"] -.-> C

When to use it — and when not

Reach for a saga when a business operation must span stores or services that no single ACID transaction covers, and correctness depends on the whole thing either completing or unwinding. Do not reach for it when:

The cost is real: you write and test a compensation for every step, you live with eventual consistency and its visible intermediate states, and you operate a coordinator and a reaper.

Signals it's working

Seen in the wild

See it

The ChainScene carries the mechanism — reserve → charge → (fail) → compensate back in reverse to a COMPENSATED terminal state — and the TilesScene shows the load-bearing config (30 s step timeout, max 5 retries with backoff, 60 s reaper sweep, compensations retried to 24 h → DLQ).

Named concepts: saga pattern, compensating transactions, saga orchestration, backward recovery. </content> </invoke>