Drill 23 — "Multi-AZ"

Theme: Scale & Cloud · Format: 20-min drill · Difficulty: core

The story

The architecture diagram says multi-AZ everything; it's in the client security questionnaire and everyone believes it. An availability-zone event occurs. The system goes down anyway: the message broker was a single instance ("we were going to cluster it after the migration"), and the NAT gateway — through which every vendor call passed — lived in the failed AZ.

⏸ Stop here and ask the room: the diagram lied. How do you find the other lies before the next zone event?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Why do diagrams drift from reality?

Listen for: diagrams record intent at a moment in time; infrastructure records decisions made under deadline pressure ever since. Every "temporary" workaround, every "we'll cluster it after the migration," every manual hotfix widens the gap — and nothing forces the diagram to update, because the diagram isn't executable. Strong answers reach for reality-derived sources of truth: generate topology from infrastructure-as-code state and live configuration, not from a drawing tool. Sharpest observation in the room: the diagram was written for a security questionnaire — its audience was a client, not an engineer, so it recorded the aspiration.

Weak signal: "we should keep the diagram updated." A process promise with no enforcement is how the diagram got wrong in the first place — the fix is deriving the picture from reality, not asking humans to redraw it.

What's the difference between "deployed in two AZs" and "survives losing one"?

Listen for: the first is a statement about placement, the second about behavior. Surviving a zone loss requires that every dependency on the request path — broker, NAT (Network Address Translation) gateway, databases, DNS, load balancers, the failover mechanism itself — tolerates the loss, that capacity in the surviving zones can absorb the shifted traffic, and that the failover actually triggers. One single-instance component anywhere on the path converts "multi-AZ" to "multi-AZ except when it matters." Strong answers say: redundancy is a property of the path, not of the components.

Weak signal: counting instances — "we have two of everything, so we're fine." Two instances behind a single NAT gateway is one instance with extra billing.

How would you test the claim — actually?

Listen for: induced failure — a game day that really kills a zone. Staging first, then production, during business hours with everyone watching, because business hours is when you want to find out and when the most hands are available. Strong answers include a designed experiment: hypothesis written down ("we predict zero customer impact"), blast-radius controls, an abort switch, and a findings review that files every discovered SPOF (single point of failure) as a ticket with an owner.

Weak signal: "we'd review the Terraform / audit the architecture." Paper audits find the SPOFs someone already suspects; only induced failure finds the ones nobody knows they know. If the room flinches at production game days, ask them which they'd rather have — a scheduled zone kill with everyone watching, or an unscheduled one at 2 a.m.

The reveal

The postmortem finds four more single points of failure, each with its own "temporary" story. Every one was known to exactly one person.

Why this happens — the mechanics

An AZ (Availability Zone — an isolated datacenter within a cloud region) failure is a composition test: it doesn't probe any single component, it probes every dependency edge in the topology simultaneously. That's why the system failed while every component owner's beliefs were locally true — the broker team knew clustering was pending, the network team knew the NAT gateway was zonal, and no one view contained both facts. Redundancy claims fail at the seams between teams, which is exactly where diagrams are vaguest and ownership is thinnest.

The "temporary" story is the mechanism of drift. Each shortcut was rational when taken, carried a sincere intention to fix later, and became invisible the moment it stopped hurting. Knowledge of each shortcut lived with exactly one person — a bus factor of one, multiplied across four more components. Organizations don't record the gap between intended and actual architecture anywhere, so the gap compounds silently until an external event audits it all at once.

Staging never shows it because staging doesn't reproduce the topology — it's typically single-zone, or its "zones" share the underlying failure domains, and its brokers and gateways are sized and placed differently. Zone loss is also not a failure you can encounter by accident in a test environment; somebody has to choose to inflict it. That's the deeper reason game days exist: the failure mode can't be discovered incrementally, only rehearsed deliberately. A resilience property that has never been exercised isn't a property — it's a rumor with a diagram.

Diagrams

Problem — how the failure unfolds:

flowchart TD
    A["Diagram claims multi-AZ<br/>everything"] --> B["Broker is a single<br/>instance<br/>(silent)"]
    A --> C["NAT gateway lives in one<br/>AZ<br/>(silent)"]
    D["Availability Zone event<br/>hits that AZ"] --> E["Broker goes down with its AZ"]
    B --> E
    D --> F["NAT gateway unreachable"]
    C --> F
    E --> G["Message flow stops"]
    F --> H["Every vendor call fails"]
    G --> I["The multi-AZ system is<br/>fully down"]
    H --> I

Solution — where the guardrails go:

flowchart TD
    A["SPOF audit walks every<br/>component as deployed"] --> B["Broker clustered across AZs"]
    A --> C["NAT gateway deployed per AZ"]
    D["Game day actually kills a<br/>zone, staging then<br/>production"] --> E["Traffic and messaging<br/>shift to surviving AZs"]
    B --> E
    C --> E
    E --> F["System survives the next<br/>real zone event"]
    D -.-> G["Alarm fires on any single-<br/>AZ dependency found"]

What good looks like

• A SPOF audit that walks deployed reality: scripted queries against the cloud APIs and infrastructure-as-code state — every instance tagged with its zone, every component with fewer than two zones flagged automatically, re-run on a schedule so the report can't go stale like the diagram did.
• Zonal services made per-zone by default: one NAT gateway per AZ with zone-local routing, brokers clustered with quorum across three zones, load balancers with cross-zone awareness — and a lint rule in the infrastructure pipeline that blocks new single-zone singletons.
• A written game-day script: hypothesis ("zero customer impact when AZ-b dies"), scope and abort criteria, roles assigned (incident commander, scribe, comms), the kill method (deny the zone at the network layer or drain it), success metrics, and a findings-to-tickets review within 48 hours.
• Game days on a calendar, not in a someday file: quarterly in staging, at least annually in production during business hours — the scheduled kill with everyone watching beats the unscheduled one at 2 a.m.
• Capacity headroom verified, not assumed: surviving zones must absorb the failed zone's traffic, so N+1 capacity per zone is load-tested as part of the game day, not just provisioned on paper.
• "Temporary" gets an expiry date: every accepted shortcut is filed as a risk ticket with a named owner and a review date — a temporary state with no expiry is a permanent state with better marketing.
• Client-facing claims traced to evidence: every resilience statement in the security questionnaire links to the game day that demonstrated it, with a date. If there's no exercise to cite, the claim doesn't ship.

The lesson

Redundancy claims are hypotheses; only induced failure tests them. Run game days — kill an AZ in staging, then in production during business hours with everyone watching (that's when you want to find out). A resilience property you haven't exercised is a rumor.

Named concepts: chaos engineering, game days, single-point-of-failure analysis, DR validation.

Real-world echoes

• AWS S3, February 2017 — an engineer executing a documented playbook mistyped one command argument and removed far more index capacity than intended from S3 in us-east-1; the blast radius covered a huge slice of the internet. The sharpest detail: AWS's own status dashboard depended on S3, so the page stayed green while S3 was down — the recovery tooling had an undocumented dependency on the thing that failed.
• Facebook, October 2021 — a routine configuration change disconnected Facebook's backbone and withdrew its BGP routes, taking every property offline for about six hours. Internal tools, DNS, and even door-badge systems rode on the same network, so engineers were partially locked out of the buildings and systems needed for the fix — hidden dependencies that no diagram showed, discovered all at once.

Transfer

If our least-favorite AZ died at 2 p.m. today, what's our honest prediction? Now schedule the staging experiment that checks it.

Make it a ticket: Run the SPOF audit script against deployed reality — list every component with fewer than two zones, file one ticket per finding with a named owner, and put the first staging zone-kill game day on the calendar with a date.

Related

• Drill 26 — The Failback That Split the Brain — this drill's lesson applied to state: "it can't diverge" is the same rumor as "it's multi-AZ."
• Drill 27 — The Chatty Neighbors — the other drill where the topology, not any component, is the problem.
• Drill 6 — The Health Check That Checked Nothing (Theme 1 — Silent Failures): failure-domain thinking at the instance level instead of the zone level.