Pattern — Feature-Flag Kill Switch

The problem it solves

A feature flag is a runtime switch that changes production behavior without a deploy — which is exactly why it turns dangerous when nobody owns its off-position. The failure shows up in two costumes: a test-only flag (verification_enabled: false) rides a Friday diff into prod and silently disables a safety check all weekend (Drill 28), and a "temporary" migration flag stuck at 50% forks one product into two divergent code paths nobody fully understands (Drill 58). Same shape both times: a flag with no owner, no expiry, and no fast way to turn it back off becomes a permanent, unwatched change to how production behaves.

The pattern

Treat every flag as a kill switch backed by a registry, not a config value backed by memory. Two halves make it work. First, the switch: the flag's dangerous position must be reversible in seconds from a runtime control plane, with no redeploy — that's what makes it a kill switch instead of a landmine. Second, the hygiene: every flag is born with an owner, an expiry (TTL), and a deletion ticket, recorded in a registry that CI enforces; safety-relevant flags are typed so they can't reach the dangerous position in prod without a second approval; and rollout ratchets through gates (1% → 25% → 50% → 100%) with a diff monitor watching the two paths. A flag you can't instantly flip off, can't find in a registry, and can't tell is safety-relevant is technical debt wearing a switch's clothing.

How to apply it

1. Register every flag at birth — five fields, enforced by CI. Each flag records owner (a named human, not a team), created, expiry (default 90 days), rollout_pct, and deletion_ticket (the removal ticket, pre-filled the day the flag is created). A CI check rejects any PR that adds a flag not in the registry, and fails the build once a flag is past its expiry — expired flags are deleted, not extended by default.
2. Type the safety-relevant flags. Any flag that disables verification, limits, or auditing carries safety: true in the config schema. The config system then enforces the invariant: it cannot be set to the dangerous value in the prod environment without a second approval — a one-reviewer PR physically cannot merge it. Keep staging and prod config in separate files (or explicit overlays) so a staging experiment can't ride a shared 40-line diff into a prod release.
3. Roll out on a ratchet, not a leap. Advance through 1% → 25% → 50% → 100%, holding each step (e.g. 24 h, or until a diff monitor confirms the new and old paths agree on outputs, error rate, and p99 latency). Divergence pauses the ratchet; agreement advances it. Log path=old|new on every request so "which path did the affected traffic take?" is a one-line query in every incident, not archaeology.
4. Make off-instant and verify on-arrival. The runtime control plane must propagate a flip in under 30 s with no deploy — that is the kill switch. After every rollout, a post-deploy smoke test runs within 5 minutes, asserts behavior ("verification ran on the last 100 decisions"), and pages on failure; the deploy ticket doesn't close until it passes.

flowchart TD
    A["New flag proposed"] --> B["Registry entry required:<br/>owner, expiry 90d,<br/>rollout %, deletion ticket"]
    B --> C{"safety: true?"}
    C -->|"yes"| D["Dangerous value in prod<br/>needs a second approval"]
    C -->|"no"| E["Ratchet: 1% to 25% to<br/>50% to 100%, diff monitor<br/>gates each step"]
    D --> E
    E --> F["Runtime kill switch:<br/>flip off in under 30s,<br/>no redeploy"]
    F --> G["Post-deploy check:<br/>verification ran on the<br/>last 100 decisions"]

When to use it — and when not

Reach for it on any flag that changes production behavior and needs to be reversible faster than a deploy — release toggles, migration routers, and every safety/kill switch. Skip the heavy machinery where it isn't paid back: a short-lived experiment flag with a hard end date and no safety impact doesn't need typed guards or a ratchet, just an owner and a TTL. And know the cost — the kill switch is a lever, not a policy. The switch decouples "turn it off" from "ship a deploy"; it does not decide when to pull it. If you add the runtime switch but skip the registry, the expiry, and the typed guard, you've built a faster way to reach the dangerous state, not a safer one — that's exactly how a flag reaches month six at 50% or rides a Friday diff into a weekend outage.

Signals it's working

Two counters and one clock. Flag age: zero flags past their 90-day expiry (CI is red if any exist); alarm when a flag crosses 60 days with no ratchet date. Kill-switch latency: measured flip-to-effect stays under 30 s — game-day it quarterly, because an untested off-switch is a decoration. Post-deploy coverage: the behavioral smoke test passes on 100% of rollouts and the deploy ticket blocks on it. Prove the guard in CI: a test that a safety: true flag set dangerously in the prod overlay fails the build with one approval — if that test ever goes green, the guard is off.

Seen in the wild

• LaunchDarkly, Statsig, Unleash, and Flagsmith all ship runtime flag flips (no redeploy), flag ownership/expiry, and percentage rollouts as first-class features — the kill switch and the hygiene, productized.
• Facebook's Gatekeeper and Google's flag frameworks pioneered percentage-based progressive rollout with instant runtime kill switches at scale.
• Knight Capital (2012) is the price tag on skipping it: a deploy reached only 7 of 8 servers and a reused flag activated eight-year-old dormant code on the eighth — roughly $440 million lost in ~45 minutes, with no runtime kill switch and no post-deploy behavioral check to catch what production was actually doing.

See it

CardsScene shows the load-bearing config (90-day expiry · ratchet 1→25→50→100% · flip < 30 s · last-100-decisions check), and ChainScene carries the mechanism: register → type the safety flag → ratchet with diff monitor → runtime kill switch → post-deploy verify.

Named concepts: kill switch, feature-flag hygiene, typed safety flags, progressive delivery.