Resilience Drills — 68 War-Story Scenarios for Team Training
Sixty-eight realistic failure scenarios from the world of AI/ML systems, decision platforms, and cloud operations. Unlike the interview cases (which assess thinking), these drills exist to build the resilient mindset in the team you already have — the reflex to ask "how does this fail silently?", "what's the blast radius?", and "where could this happen to us?"
Why war stories
Engineers don't internalize resilience from principles; they internalize it from specific disasters that could have been theirs. A story creates the memory hook; the transfer question — "where could this exact thing happen in our stack, today?" — converts the story into an audit of your own systems. Every drill ends with it. Skipping it turns training into entertainment.
The themes
| Folder | Theme | Drills | The reflex it builds |
|---|---|---|---|
| 01-silent-failures/ | Silent failures | 1–6 | "Green dashboards prove nothing — what would tell us this is actually working?" |
| 02-data-disasters/ | Data disasters | 7–13 | "The data is guilty until proven innocent." |
| 03-llm-and-agents/ | LLM & agent incidents | 14–20 | "The model is a component, not a colleague — bound it, check it, budget it." |
| 04-scale-and-cloud/ | Scale & cloud | 21–27 | "Resilience mechanisms are also failure amplifiers. What does ours do under stress?" |
| 05-people-and-process/ | People & process | 28–34 | "The system includes the humans. Design for the 5pm Friday version of us." |
| 06-feedback-loops/ | Feedback loops & adversaries | 35–41 | "Our system changes the world it observes — and the world fights back." |
| 07-security-and-trust/ | Security & trust boundaries | 42–48 | "Every boundary you didn't draw explicitly is drawn by the attacker." |
| 08-compliance-and-governance/ | Compliance, privacy & model governance | 49–53 | "If you can't explain it, prove it, or delete it — you don't govern it." |
| 09-migrations-and-releases/ | Migrations & releases | 54–58 | "Every migration is an experiment — design the rollback before the rollout." |
| 10-vendors-and-dependencies/ | Vendors & dependencies | 59–63 | "Every dependency is a decision someone else can unmake." |
| 11-cost-and-finops/ | Cost & FinOps | 64–68 | "Every dollar of cloud spend is an engineering decision someone made — or forgot to unmake." |
Each theme folder contains a README (the theme's "why it matters" context, a drill index with difficulty ratings, and the theme's best-practices checklist) plus one document per drill, so a facilitator can print or share exactly one drill for a session.
Every drill names the formal SRE / best-practice concepts it teaches (SLOs, error budgets, blameless postmortems, circuit breakers, chaos engineering, break-glass, Goodhart's law…). sre-concepts.md is the reverse index — concept → drills — so you can also run the program concept-first ("this month is monitoring: drills 1, 2, 3, 6, 29") or grab the right drill the week after a real incident touches that concept. It ends with a short bookshelf (Google SRE, Accelerate, How Complex Systems Fail) for people who want the theory behind the stories.
Anatomy of a drill document
Each drill is a self-contained training document:
- The story — the war story, told up to a ⏸ stop blockquote where the facilitator pauses the room.
- Run it — facilitator script — timing for a 20-minute session, plus every discussion question annotated with Listen for (what a strong answer sounds like) and Weak signal (the shallow answer that means the room hasn't got it yet). A new facilitator can run the drill cold.
- The reveal — the punchline: what actually happened and how late it was discovered.
- Why this happens — the mechanics — the deeper technical and organizational explanation: why competent, busy teams hit exactly this.
- Diagrams — two mermaid diagrams. The Problem is drawn as a narrative: a sequence diagram (actors exchanging calls over time, silent gaps marked, failed calls crossed out) for incident-shaped stories, or a timeline for slow-burn failures where the duration is the point; three topology drills (23, 27, 38) keep flowcharts because the structure is the story. The Solution is always a flowchart showing where the guardrails and monitors insert.
- What good looks like — the lesson expanded into concrete implementation bullets: specific monitors, thresholds, cadences, and process mechanics.
- The lesson + named concepts — the principle, attached to the formal SRE/engineering vocabulary.
- Real-world echoes — factual public incidents of the same class (Knight Capital, GitLab 2017, Air Canada's chatbot, TSB's migration…), so the room knows this isn't hypothetical. Included only where a well-documented public incident genuinely rhymes.
- Transfer + make it a ticket — the mandatory closing question that turns the story into an audit of your own stack, and the prompt to convert the answer into a tracked action item.
- Related — links to sibling drills and cross-theme rhymes.
Abbreviations are expanded on first use in every file; glossary.md explains all of them (SLI, RTO, DLQ, SBOM, PII…) in plain language, organized by category.
How to run a drill
Format A — the weekly 20-minute drill (default). One scenario at standup+ or a dedicated slot. Facilitator reads the story up to the first stop, the team discusses each staged question, then the punchline, then the lesson, then the transfer question. Timebox hard; energy matters more than completeness.
Format B — the 60-minute tabletop. One scenario run as role-play: facilitator plays the system/stakeholders and feeds facts only when asked (like drill 16 or 21). Assign roles: incident commander, comms, investigator. Best for the incident-shaped drills (16–25).
Format C — the onboarding gauntlet. New hires run one drill per day for their first six weeks with a different teammate facilitating each time. They learn the systems and the culture of asking these questions, and they meet the whole team.
Facilitation rules
- Don't reveal the punchline early. The struggle before the reveal is the training. Read only to the marked stop, then ask.
- "What would you check first?" beats "what's the answer?" You're training moves, not trivia.
- No blame, even for fictional engineers. The moment the room mocks the character who dropped the table, people stop admitting they nearly did the same. Every story's engineer was competent and busy — that's the point.
- The transfer question is mandatory. End every drill with: "Where is the nearest cousin of this in our stack?" — and write the answers down.
- Capture actions, not just insights. If the transfer question surfaces a real gap, it becomes a ticket with an owner before the room empties. A drill that produces three tickets a month pays for itself many times over.
- Rotate the facilitator weekly. Facilitating builds the mindset faster than participating — you must anticipate every answer.
- Invite the team's own war stories after the reveal. "Anyone seen a version of this?" The day the team starts contributing their own scenarios, the culture has taken. Add them to this folder.
The universal question bank
These apply to any scenario — and to any design review. Teach them until they're reflexes:
Detection: How would we know? How long would this run before anyone noticed? What's the earliest observable symptom? Does anything monitor the business meaning, not just the process health?
Blast radius: What's the worst version of this? What does it cost per hour undetected? Is the damage reversible — and is there a point where it stops being reversible?
First moves: What do you do in the first five minutes? What do you stop first? Who needs to know before you start debugging? What would you check first, and what does each answer rule out?
Prevention vs. resilience: Could any realistic review have caught this before production? If not — what would have made it cheap instead of catastrophic? What's the guardrail that makes this class of error impossible, not just this instance?
Transfer: Where is the nearest cousin of this in our stack? Who owns that? What would it take to check this week?
Cadence and sequencing
Run one per week and the set lasts well over a year; run two per week (or a monthly double-length tabletop plus weeklies) to cover it in about eight months — by which time the early ones are worth repeating (the team will answer them differently — that difference is the progress). Sequence within a theme ramps from "spot the failure" to "handle the incident" to "redesign the system." Themes can be interleaved freely; starting with silent failures (1–5) is recommended — it recalibrates what "working" means, which every later theme builds on.
A simple maturity signal to watch for over the months: answers shift from fixes ("patch the parser") → detection ("alert on distribution shift") → design ("make this class of failure structurally impossible") → unprompted transfer ("this is basically our vendor feed — I'll check it today"). When the fourth kind dominates, the mindset has landed.