BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Pattern — Canary Release

The problem it solves

A change that passed every offline check still ships broken on live traffic, because the eval only saw the past. A retrained fraud model looks great on the holdout, then declines double the small-ticket purchases the moment it owns 100% of authorizations — a shifted feature that only exists in production (Drill 91). A big-bang system cutover swaps the old decisioning engine for a new one over a weekend, and a quiet DTI rounding change flips hundreds of borderline loans with no old system left to compare against (Drill 54). The common shape: a change reaches 100% of a hot-path decision in one irreversible step, so the first time anyone learns what it does on live traffic, the whole book is already exposed.

The pattern

Never let a new version own all the traffic at once. Route a small, bounded slice of live traffic to the new version, hold it there long enough to gather real signal, and gate every promotion on live metrics cut by segment — with an automatic halt-and-rollback if a gate fails. The exposure a new version may own before its gates pass is an explicit input you choose (the blast radius), defaulting to 1%, never 100%. Optionally run a shadow stage first: score live traffic with the new version but keep acting on the old one, so you compare the two streams before any real decision rides on the change. The offline score says the change is good; only staged, gated exposure proves it is safe.

How to apply it

  1. Shadow first (24h). Route 100% of live traffic to the new version but keep the champion making the actual decisions. Compare the two output streams on the same requests. Promote out of shadow only if the decision delta stays inside a band — e.g. < 0.5pp overall and < 1pp on any top-20 segment — for at least 24 hours, so a full daily traffic mix (including overnight and peak) flows through both.
  2. Canary in fixed stages. Take real traffic in steps — 1% → 5% → 25% → 100% — and hold each stage until it has enough signal to trust: ≥ 30 minutes or ≥ 10,000 decisions, whichever is later. Skipping straight to 25% defeats the point; the 1% stage is where you pay the smallest price to learn.
  3. Gate each promotion on segmented live metrics. Define the gate as a specific metric · segment cut · threshold · automatic action, e.g. decline_rate cut by {transaction_size, channel, merchant_category, issuer_bin}, promote only if every top-20 segment is within ±1pp of the champion on the same window. A failing gate stops promotion automatically — no human in the loop, no override without sign-off. A blended number hides the incident: a 2% network-wide move was a 40% move in one slice.
  4. Add a business-metric guardrail with auto-halt. Wire the deploy directly to the number that pays the bills — approval rate / authorization value — with a hard band such as approval rate within ±1pp of the trailing 7-day baseline. On breach, auto-halt and roll back, don't just page. This is the backstop of last resort, not a proxy.
  5. Make blast radius an explicit deploy input. A required parameter — "max share of decisions this version may own before its gates pass" — default 1%, raising it requires sign-off. For a cutover with no percentage router (Drill 54), the equivalent is a parallel run: both systems decide every request, a nightly diff report targets zero, and the old path stays executable behind a router flag until the new one survives a full business cycle (month-end included).
  6. Keep rollback instant and rehearsed. Keep the champion warm so rollback is one click and takes seconds. Game-day a deliberately bad canary quarterly and prove the gate halts promotion before it reaches 5% and the rollback actually fires.
flowchart TD
    A["New version (blast radius = 1%)"] --> B["Shadow 24h: score live,<br/>champion still decides"]
    B --> C["Canary 1% of traffic<br/>(hold 30 min or 10k decisions)"]
    C --> G1["Gate: decline rate by segment<br/>within 1pp of champion"]
    G1 -->|"pass"| D["Promote 1% to 5% to<br/>25% to 100%"]
    G1 -.->|"fail"| HALT["Auto-halt + rollback<br/>(champion kept warm)"]
    D --> G2["Guardrail: approval rate<br/>within 1pp of 7-day baseline"]
    G2 -.->|"breach"| HALT
    HALT --> H["Rollback in seconds,<br/>1% of book touched"]

When to use it — and when not

Reach for it on any change to a hot-path decision system that can't be fully verified offline — a model on the authorization path, a rules engine, a system cutover. It costs you: a traffic router or shadow harness, per-segment metrics, and a rollout that takes hours-to-days instead of one deploy, so don't impose it on genuinely reversible, low-blast-radius changes (a copy tweak behind a flag, a config you can flip back in seconds) where the machinery costs more than the risk it retires. Two limits to respect. First, a canary needs enough traffic per segment to reach significance — for a rare event on a thin slice, 1% for 30 minutes may show nothing, so size the stage to the base rate or you'll promote a regression you never had the sample to see. Second, a canary bounds exposure, it doesn't diagnose — a passing gate says "no worse on the metrics we watch," not "correct"; pair it with the segment cuts and the business guardrail so a regression that hides in a blended average still trips a gate.

Signals it's working

Every promotion carries a machine-readable gate result (metric, segment, value, threshold, pass/fail) in the deploy log — not a human eyeballing a dashboard. Track blast radius at first minute of every deploy (must be ≤ 1%, alarm if any version ever starts above it) and time-to-rollback (seconds, because the champion is warm). The proof it's real is the quarterly game-day: inject a deliberately bad canary and assert the gate stops promotion before 5% exposure and the rollback fires automatically. A blended metric that moved but didn't halt anything is the anti-signal — that's a dashboard, not a gate.

Seen in the wild

See it

The ChainScene carries shadow → 1% canary → segmented gate → auto-halt/rollback; the CardsScene and TilesScene show the load-bearing config (stages 1/5/25/100%, hold ≥30 min or ≥10k, segment band ±1pp, approval-rate guardrail ±1pp of 7-day baseline, blast radius default 1%).

Named concepts: canary release, progressive delivery, automated rollout gates, blast-radius control.