Theme 5 — People & Process
The reflex this theme builds: the system includes the humans — design for the 5 p.m. Friday version of us.
Why this theme matters
An architecture diagram that stops at the load balancer is missing half the system. On-call rotations, review habits, escalation chains, incentive structures — these are components too, with failure modes as reproducible as a memory leak. And they fail exactly where the documented process and the 5 p.m. Friday reality diverge: the forty-line config diff nobody can truly review, the rotation slot pointing at someone who left, the runbook from two reorgs ago. In every drill in this theme, the people involved were competent and busy; the system offered no seam where the error could be caught.
That's why "who screwed up?" is the wrong question — and an actively destructive one. Blaming individuals after the fact shuts down the near-miss reporting that would have prevented the next incident: humans are the only sensors that can detect near-misses, and blame turns those sensors off. Site Reliability Engineering (SRE) treats human factors as engineering material, not HR material — paging paths get tested like code, emergency access gets designed like an API, and postmortems produce guards rather than apologies. Abbreviations are expanded on first use; the full reference is glossary.md.
Drill index
| # | Drill | One-line hook | Problem diagram | ~Difficulty |
|---|---|---|---|---|
| 28 | The Friday Flag | A staging-only test flag rides a forty-line diff into Friday's 5 p.m. release — production skips verification all weekend. | sequence | core |
| 29 | Both On-Call, Neither Paged | The alert fires perfectly — into a muted channel and a rotation slot pointing at someone who left. | sequence | intro |
| 30 | The Hero's Hotfix | A 2 a.m. hand-edit saves the night, leaves no record, and detonates three weeks later. | sequence | core |
| 31 | The Postmortem That Named Names | The postmortem names the engineer; near-miss reports drop to zero — right before the big one. | timeline | advanced |
| 32 | The Backup That Wouldn't Restore | Three years of green backup checkmarks, zero restores — until the day a table drops. | timeline | core |
| 33 | The Runbook From Two Reorgs Ago | The on-call does exactly the right thing and opens the runbook — a map of a country that no longer exists. | sequence | intro |
| 34 | Locked Out at the Worst Moment | The one engineer who can fix the Sev-1 is on the bridge, ready — and locked out by her own company's automation. | sequence | advanced |
Best practices — the people & process checklist
What to actually do, distilled from drills 28–34:
- Make post-deployment behavioral verification part of "done": a deploy is finished when production behavior is confirmed (a smoke test asserting "verification ran on the last 100 decisions"), not when the pipeline turns green.
- Practice feature-flag hygiene: every flag has an owner and an expiry date, and dangerous values — anything that disables a safety check in production — carry schema-level guards requiring a second approval.
- Send synthetic test pages through the entire alerting chain — alert, routing, page, escalation on no-acknowledge — on a schedule, and re-audit rotations and escalation policies after every reorg and offboarding. Every page-severity alert must end at a human who must acknowledge.
- Run incidents with command discipline: a dedicated channel where every command is pasted as it runs, and no out-of-band changes — anything done by hand during an incident becomes a recorded artifact, or it becomes next month's "impossible state."
- Design break-glass access before you need it: grantable by an incident commander in minutes, time-boxed, auto-revoked, logged under the user's own identity, and reviewed grant-by-grant in the postmortem. If the sanctioned emergency path is slow, people will invent unsanctioned ones.
- Hold blameless postmortems that produce systemic action items — guards, checks, defaults, each with an owner. "Be more careful" is not a control; a real action item survives the engineer leaving the company.
- Protect a near-miss reporting channel and track its volume as a health metric. Zero reports means the channel is dead, not that nothing almost happened — humans are the only sensors that detect near-misses.
- Schedule restore drills that validate RTO/RPO (Recovery Time Objective / Recovery Point Objective — how long recovery may take, and how much recent data you may lose): restore to a scratch environment, validate row counts and schema, time it against the target, and treat key rotation and schema migrations as backup-breaking changes.
- Give every runbook an owner and a last-tested date at the top, exercise it in game days (scheduled drills that run the scenario for real), and fix broken steps the moment an incident exposes them — a wrong runbook is worse than none.
- Review on-call load and alert quality monthly: pages per shift and actionability rate (the fraction of pages that actually required action). A noisy pager trains people to mute channels — and the one page that matters dies in the mute.