Pattern — Chaos Testing

The problem it solves

A resilience claim you have never exercised is a rumor with a diagram. Drill 23's system was "multi-AZ everything" on paper, yet a single availability-zone event took it fully down: the message broker was one un-clustered instance and the NAT gateway lived in the failed zone — two single points of failure that every component owner's local beliefs declared fine. The recurring shape: redundancy fails at the seams between teams, and no ordinary test probes the seams. Staging is single-zone, unit tests mock the failure away, and nobody encounters a zone loss by accident — someone has to choose to inflict it. Chaos testing is how you inflict it on purpose, before the outage does.

The pattern

Chaos engineering turns each resilience claim into a falsifiable experiment. You write down a steady-state hypothesis — a measurable "the system stays healthy" prediction — then deliberately inject a real fault (kill an AZ, drop a dependency, add latency) while a controlled slice of traffic runs, and watch whether the hypothesis holds. Two rails make it safe: blast-radius control (start tiny, escalate only after each stage passes) and an automatic abort switch (a stop condition wired to a live metric that halts and rolls back the instant real users start hurting). Run as a scheduled, staffed game day, every discovered failure becomes a ticket with an owner instead of a 2 a.m. surprise.

How to apply it

1. Write the steady-state hypothesis as numbers, not adjectives. Pick 3–4 service-level indicators and their thresholds before the experiment: e.g. checkout success ≥ 99.9%, p99 latency < 800 ms, order-throughput within 5% of baseline. The hypothesis for Drill 23's zone kill: "when AZ-b dies, customer-facing success stays ≥ 99.9% and p99 stays < 800 ms." No baseline metric, no experiment.
2. Bound the blast radius and escalate in stages. Never start in production at 100%. Ramp: staging first → 1 instance → 1 AZ → 1% of prod traffic → 5% → 100%, and only promote a stage after the previous one held the hypothesis. Run in business hours with the most hands on deck — that is when you want to discover the failure.
3. Wire an automatic abort (the load-bearing safety rail). Attach a stop condition to a live alarm: if the guardrail metric breaches — e.g. success < 99% for 2 minutes or error rate > 5% — the experiment auto-halts and rolls back the fault within 60 seconds. In AWS FIS this is a stopConditions block pointing at a CloudWatch alarm; in Gremlin it is the "Halt" button plus a status-check monitor. The abort is what lets you run in production at all.
4. Assign game-day roles and inject a real fault. Named incident commander, scribe, and comms; a written script with hypothesis, scope, and abort criteria. Kill the zone the way reality would — deny the AZ at the network layer or drain it — not a mocked flag. Netflix's Chaos Monkey does the minimal version continuously: terminate one random instance, but only 09:00–15:00 so engineers are watching.
5. Close the loop. File every discovered SPOF as a ticket with an owner and a fix date within 48 hours; re-run the SPOF audit (scripted cloud-API queries flagging any component with < 2 zones) on a schedule so the map can't go stale like the diagram did. Put the next game day on the calendar — quarterly in staging, at least annually in production — not in a someday file.

flowchart TD
    A["Steady-state hypothesis<br/>(success >= 99.9%, p99 < 800ms)"] --> B["Blast radius: staging -> 1 AZ<br/>-> 1% -> 5% -> 100%"]
    B --> C["Inject real fault<br/>(deny AZ-b at network layer)"]
    C --> D{"Guardrail metric<br/>within threshold?"}
    D -->|"yes"| E["Hypothesis holds -><br/>promote to next stage"]
    D -->|"no (breach 2 min)"| F["Auto-abort + rollback<br/>within 60s"]
    F --> G["File each SPOF as a ticket,<br/>owner + date, within 48h"]
    E --> G

When to use it — and when not

Reach for it whenever a resilience property is claimed but never exercised — failover, multi-AZ, auto-scaling, DR runbooks, retry/backoff behavior — especially at the seams between teams and dependencies where no single owner sees the whole path. It is the only way to find the SPOFs nobody knows they know. Do not run it when the prerequisites are missing: no observability to measure steady state, no automatic abort, or a known-broken component you haven't fixed yet — that isn't an experiment, it's a self-inflicted outage. Skip production chaos while you're mid-migration or already in an incident, and don't chaos-test a system with no redundancy to test — fix the single instance first, then verify it survives. Paper audits (reading Terraform) are cheaper but only find the SPOFs someone already suspects; use them to complement, never to replace, induced failure.

Signals it's working

• Steady-state held: guardrail SLIs (success ≥ 99.9%, p99 < 800 ms) stay inside threshold through the injected fault — a passing experiment.
• Abort actually fires when you inject a deliberately-too-big fault: rollback completes in < 60 s, proving the safety rail is real, not decorative.
• SPOF count trends toward zero across game days; each finding has a ticket, an owner, and a closed date within 48 h.
• Cadence is on a calendar, not a wiki backlog — quarterly staging kills, annual production kills — and client-facing resilience claims each cite the game day (with a date) that demonstrated them.

Seen in the wild

• Netflix Chaos Monkey / Simian Army — randomly terminates production instances during business hours; the origin of the discipline. Chaos Kong extends it to killing an entire AWS region.
• AWS Fault Injection Simulator (FIS) — managed fault injection with stopConditions bound to CloudWatch alarms, plus an "AZ availability: power interruption" action that simulates exactly Drill 23's zone loss.
• Gremlin — commercial chaos platform with a hard "Halt" abort and per-experiment status checks; Principles of Chaos Engineering (principlesofchaos.org) codifies the steady-state-hypothesis method.
• Google DiRT and Azure Chaos Studio — scheduled disaster-recovery game days and fault-injection experiments at platform scale.

See it

The ChainScene carries the mechanism — hypothesis → staged blast radius → inject fault → guardrail breach → auto-abort in 60s → file tickets — while the CardsScene and TilesScene pin the load-bearing numbers (success ≥ 99.9%, p99 < 800 ms, abort < 60 s, ramp 1% → 5% → 100%, Chaos Monkey 09:00–15:00, quarterly/annual cadence).

Named concepts: chaos engineering, steady-state hypothesis, blast-radius control, game days.