BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 33 — The Runbook From Two Reorgs Ago

Theme: People & Process · Format: 20-min drill · Difficulty: intro

The story

A dependable engineer is paged for a queue backup and does exactly the right thing: opens the runbook. Step 3 references a dashboard that was decommissioned; step 5's command targets a cluster that no longer exists; step 7 says "escalate to the platform team" — a team dissolved in the last reorg. Ninety minutes into following the map of a country that no longer exists, they abandon it and improvise — under more pressure, with less time, than if they'd started from scratch.

⏸ Stop here and ask the room: what's worse than no runbook?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

The stop question has a one-line answer — "a wrong one" — so the value is in the why: the wrong runbook spent ninety minutes and a tank of trust before failing, where no runbook would have forced honest improvisation at minute zero. Make sure the room says that out loud. Also underline what the engineer did right: opening the runbook was the correct instinct, and the drill must not teach people to skip runbooks.

"How do runbooks rot?"

Listen for: an inventory of decay vectors — dashboards decommissioned, clusters migrated, teams dissolved, links broken, defaults changed — and the key observation that each change was made correctly by someone who had no idea this runbook referenced their system. Documentation rots because the world changes and nothing tells the document; strong answers add that audits which check a runbook exists (rather than works) actively preserve the rot.

Weak signal: "people are lazy about documentation" — this runbook was written well; laziness isn't the mechanism. The author left, the world moved, and no process ever exercised the text.

"What makes a runbook testable?"

Listen for: concrete properties — steps that are executable commands rather than prose; assertions per step ("you should now see X") so failure is detectable; an owner; a last-tested date displayed at the top so staleness is visible before you invest trust; and a cadence (game days) that actually runs it. Bonus: the more of a runbook you can execute mechanically, the less of it can silently lie.

Weak signal: "review it annually" — reading is not testing; every broken step in this story would have read fine. Step 5's cluster doesn't fail until you run the command.

"What should the on-call do the moment they find a broken step?"

Listen for: fix-as-you-go — at minimum, mark the step broken right then (a one-line edit or a ticket linked from the step) so the next 2 a.m. reader doesn't pay the same toll, and make "runbook fixes" a standard postmortem output. Strong answers make it cheap: edit access from the incident channel, no approval gate for annotations.

Weak signal: "note it in the postmortem" alone — three weeks later, from memory, the broken step count will be wrong and the fix won't land. The repair has to happen at the point of discovery.

The reveal

The runbook was written well — three years ago, by someone who left. It had been "the documentation" in every audit since. Nobody had executed it in anger or in practice.

Why this happens — the mechanics

A runbook is a frozen snapshot of a moving system. Every referenced dashboard, cluster, and team name is a pointer into an environment that reorgs, migrations, and cleanups rewrite continuously — and none of those changes has any mechanism for discovering that a document points at what they just removed. The rot isn't negligence; it's entropy plus decoupling. The people changing the world can't see the runbook, and the people who own the runbook (if anyone still does, after its author leaves) can't see the changes.

Unrehearsed procedures rot invisibly because reading is not execution. A stale runbook reads perfectly — the prose is clear, the steps are numbered, the audit box gets ticked. The document had passed every audit precisely because audits checked its existence, not its truth: paperwork compliance standing in for operational readiness. Only execution touches reality, which is why step 5's dead cluster survived three years of reviews and zero game days.

The ninety minutes is the interesting cost. A wrong runbook fails slowly — each broken step consumes time, and worse, consumes trust: the engineer keeps going because the document is official, because step 3 might just be their mistake, because abandoning the sanctioned path feels like going rogue mid-incident. This is the sunk-cost trap wearing a process badge. No runbook forces honest improvisation at minute zero; a wrong one defers it to minute ninety, under more pressure, with the incident older and the options fewer.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant PG as Page
    participant OC as On-call
    participant RB as Runbook
    participant RE as Reality
    PG->>OC: Queue backup - severity page
    OC->>RB: Opens the runbook -<br/>exactly the right instinct
    RB-->>OC: Step 3 - check the dashboard
    OC--xRE: Dashboard was decommissioned
    RB-->>OC: Step 5 - run the command<br/>on the cluster
    OC--xRE: Cluster no longer exists
    RB-->>OC: Step 7 - escalate to the<br/>platform team
    OC--xRE: Team dissolved in the<br/>last reorg
    Note over RB: Written three years ago -<br/>never executed since (silent)
    Note over OC,RE: 90 minutes in - abandons the<br/>map and improvises under more<br/>pressure with less time

Solution — where the guardrails go:

flowchart TD
    A["Runbook carries an owner<br/>and a last-tested date at<br/>the top"] --> B["Quarterly game day:<br/>runbook executed for real"]
    B -.-> C["Dead dashboards, vanished<br/>clusters, dissolved teams<br/>found in practice"]
    C --> D["Runbook repaired before it<br/>is needed in anger"]
    D --> E["Real page: on-call opens a<br/>runbook tested this<br/>quarter"]
    E --> F["Steps work — incident<br/>resolved from the map"]
    F --> G["Broken step found mid-<br/>incident is fixed as you<br/>go"]
    G -.-> H["Every incident leaves the<br/>runbook better"]

What good looks like

The lesson

A runbook is code that humans execute: it needs an owner, a last-tested date displayed at the top, and periodic execution in game days — a runbook that hasn't been run this quarter is a liability, and a wrong runbook is worse than none because it spends trust and time. Every incident that touches a runbook should leave it better (fix-as-you-go is the maintenance model that actually works).

Named concepts: runbook hygiene, documentation rot, game-day validation, fix-forward documentation culture.

Real-world echoes

No single public incident to cite with confidence — runbook rot is rarely the headline, but it's a recurring supporting character: long public outage timelines routinely contain some version of "the documented procedure no longer matched the system." The pattern hides inside other incidents' "contributing factors" sections, which is exactly how it survives.

Transfer

Open our three most-likely-to-be-needed runbooks right now and dry-run them. How many steps still work? That percentage is our real preparedness.

Make it a ticket: add owner and last-tested headers to the top three runbooks, dry-run each one this sprint recording every broken step, and book the first quarterly game day — executed by someone who didn't write them.

Related