Theme 4 — Scale & Cloud
The reflex this theme builds: resilience mechanisms are also failure amplifiers — what does ours do under systemic stress?
Why this theme matters
Distributed systems fail by composition. In every drill in this theme, each component did exactly what it was built to do — the cache flushed on schedule, the clients retried, the autoscaler scaled, the failover fired — and the system failed as a whole anyway. Worse, the resilience mechanisms themselves are feedback loops: retries, autoscaling, failover, and health-based ejection all amplify load or spread damage when they're pointed at the wrong signal. The machinery you installed to survive stress is often the machinery that converts partial failure into total failure.
The cloud adds a second trap: invisible constraints. Quotas you never set, egress fees you never priced, zone topology you never drew — none of them appear on a green dashboard, and all of them surface only under stress, which is exactly when you can least afford the discovery. The habit these drills build is finding and rehearsing those constraints in daylight, before a peak weekend does it for you. Abbreviations are expanded on first use; the full reference is glossary.md.
The drills
| # | Drill | One-line hook | Problem diagram | ~Difficulty |
|---|---|---|---|---|
| 21 | The Thundering Herd | A one-line cron change puts a cold cache under peak traffic — retries with no jitter turn it into a 4× amplifier and a 45-minute outage. | sequence | intro |
| 22 | The Quota Wall | The promo weekend was load-tested and capacity-modeled — except for the auth-token quota nobody knew existed, whose raise waits in a support queue. | sequence | core |
| 23 | "Multi-AZ" | The diagram says multi-AZ everything; the zone event finds a single-instance broker, one NAT gateway, and four more lies. | flowchart | core |
| 24 | The Autoscaler That Fed the Fire | The autoscaler reads queuing as CPU demand and pours connection pools onto a struggling database — recovery means scaling down. | sequence | advanced |
| 25 | The Key That Expired on Saturday | A credential created 366 days ago by a departed engineer expires at 3 a.m. — the nine-hour scramble ends via a personal phone number. | timeline | intro |
| 26 | The Failback That Split the Brain | Failover was rehearsed; failback wasn't — stale DNS keeps a slice of traffic writing to DR, and two "authoritative" ledgers diverge. | sequence | advanced |
| 27 | The Chatty Neighbors | A clean microservice split makes 11 calls per request across zone boundaries — nothing is broken, and the bill grows $18k/month. | flowchart | core |
Best practices — the scale & cloud checklist
What to actually do, distilled from drills 21–27:
- Bound every retry, everywhere a retry exists: exponential backoff with jitter (randomized delays so clients don't retry in lockstep), retry budgets (a cap on the fraction of traffic that may be retries), and circuit breakers (stop calling a dependency that keeps failing). Multiply per-layer retry counts along one request path to learn your worst-case amplification factor.
- Shed load and apply backpressure at the front door: reject or queue-with-limits excess work before it reaches the struggling component, so overload degrades service instead of collapsing it.
- Point autoscaling at the true constraint, not a symptom: know your bottleneck resource under stress, give every automation a ceiling plus a human override, and cap the fleet-wide connection budget so scaling out can't strangle a shared database.
- Keep a quota and limit inventory — including every default the cloud provider set and you never changed: API (Application Programming Interface) rate limits, token issuance, account caps, connection pools, license seats, third-party rate limits. Check it against peak projections before every big event.
- Load-test in a production-shaped account — same quotas, same account tier, same auth path — because a limit you didn't hit in the test environment is a limit you'll discover at peak.
- Run game days that actually kill things: take down an Availability Zone (AZ — an isolated datacenter within a cloud region) for real, in staging and then in production during business hours. A redundancy claim you haven't exercised is a rumor; the audit finds single points of failure the diagram hides.
- Rehearse failback, not just failover: the return trip is a second failover with its own failure modes — stale DNS (Domain Name System) caches, divergent state — and it is almost never practiced.
- Enforce single-writer mechanically in every failover story: fencing (a mechanism that forces only one side to accept writes) and write rejection outside the active region. "It can't diverge" is a hope, not a property — and build the reconciliation tooling before you need it.
- Inventory everything that expires — credentials, certificates, licenses, reserved capacity — with alarms well ahead of expiry, automated rotation, and ownership assigned to a role, never to a person.
- Set request-path budgets — maximum hops, milliseconds, and cross-zone bytes per request — with topology-aware placement that keeps chatty services in the same zone, and a named owner for the end-to-end path.
- Design graceful-degradation modes in daylight and get them business-approved: what the system serves when a dependency is throttled or down is a product decision, not an improvisation at 3 a.m.
- Treat capacity planning as an ongoing practice, reviewed before every known peak (promos, seasonal events, launches) — and put big events on your provider's radar ahead of time so quota raises don't sit in a support queue.
Also see
- Facilitator guide — how to run drills, and the other ten themes.
- Glossary — every abbreviation, explained in plain language.