Drill 29 — Both On-Call, Neither Paged

Theme: People & Process · Format: 20-min drill · Difficulty: intro

The story

An alert fires at 11 p.m. — correctly, promptly, with a good runbook link. It fires into a Slack channel created during an incident two years ago, which everyone has since muted. The paging rotation would have caught it, except a reorg three months back left the secondary slot pointing at someone who no longer works there — and the escalation policy stops at the secondary. Outage: six hours. Postmortem's first draft: "monitoring worked as designed."

⏸ Stop here and ask the room: did it?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

The stop question is deliberately short — let the room argue about it. Someone will defend the draft ("the alert did fire"), and that defense is the teachable moment: press on where "monitoring" ends. If the room agrees too quickly that the draft is wrong, flip it — ask what the alerting team could point at to claim success, and why every one of those numbers is real.

"What's the difference between an alert firing and a human being engaged?"

Listen for: the last-hop framing — everything up to the notification is software, and the final hop is a person with a phone, a mute button, and an employment status. Strong answers define the real Service Level Indicator (SLI — the number that measures whether the service does its job) as "acknowledged by a human within N minutes," not "alert delivered."

Weak signal: treating delivery as the finish line — "it was sent to the channel" is the same logic as the postmortem draft, one hop earlier.

"What organizational events should trigger a paging-config review?"

Listen for: reorgs, offboarding, team renames, ownership transfers, holiday coverage changes — and the insight that these events live in HR systems and org charts, which never notify the paging tool. Someone should propose wiring the two together: offboarding checklists that include rotation audits, or automation that flags rotation members no longer in the directory.

Weak signal: "audit it quarterly" with no trigger — a calendar reminder decays exactly like the muted channel did.

"How do you test a paging path?"

Listen for: synthetic pages sent through the whole chain on a schedule — alert, routing, page, escalation on no-acknowledge — with a human at the end who must ack. The test must include the failure branch: don't ack the primary, and verify the escalation actually goes somewhere live.

Weak signal: "we test alerts in staging" — that tests the alert logic, not the routing, the rotation, or the humans. The last hop is precisely what staging doesn't have.

The reveal

The alert pipeline was perfect up to the last hop — the one that involves people. Nobody owned the seam between the alerting system and the org chart.

Why this happens — the mechanics

The paging path is the only piece of infrastructure whose components include employment status, phone settings, and attention — and it's typically the least tested. Every hop before the human is observable software with metrics; the last hop is invisible until it's exercised. An alert that reaches a muted channel and a departed engineer produces exactly the same telemetry as one that wakes someone up. Success and failure are indistinguishable from the inside, which is why the postmortem draft could honestly say "monitoring worked as designed" — every metric the alerting system exports was green.

The muted channel is alarm fatigue doing what it always does. Channels created during incidents accumulate noise; humans protect their attention by muting; and each mute is individually rational. Nobody decides to disconnect the alert from humanity — a hundred small, sensible acts of self-defense do it gradually. This is normalization of deviance in miniature: every day the muted channel causes no harm is a day the mute looks smarter.

The dead rotation slot is organizational drift. Reorgs move people; offboarding removes them; and the paging tool learns about neither, because org changes are executed in HR systems that don't talk to it. The seam between the org chart and the alerting config has no owner — the alerting team owns the rules, the manager owns the roster, and nobody owns the mapping. Unowned seams rot silently, and unrehearsed paths stay rotten: a paging chain that has never carried a synthetic page end-to-end is unverified code that happens to be load-bearing at 11 p.m.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant AL as Alert
    participant CH as Muted Slack channel
    participant RS as Dead rotation slot
    participant OUT as Outage
    Note over AL: Fires at 11 p.m. - correct,<br/>prompt, good runbook link
    AL--xCH: Posts into a channel from an<br/>incident two years ago
    Note over CH: Everyone muted it long ago<br/>(silent)
    AL--xRS: Pages the secondary slot
    Note over RS: Points at an engineer who<br/>left in the reorg (silent)
    Note over RS: Escalation policy stops<br/>here (silent)
    Note over OUT: Six hours - no human<br/>engaged
    Note over AL,OUT: Postmortem first draft -<br/>monitoring worked as designed

Solution — where the guardrails go:

flowchart TD
    S["Scheduled synthetic test page"] -.-> A["Full chain exercised:<br/>alert, routing, page,<br/>escalation on no-ack"]
    A -.-> B["Muted channel and dead<br/>rotation slot discovered<br/>in a drill"]
    R["Reorg or offboarding event"] --> C["Triggers rotation and<br/>escalation audit"]
    C -.-> B
    B --> F["Paging config fixed before<br/>a real incident"]
    G["Real alert fires"] --> H["Escalation chain<br/>terminates at a human who<br/>must acknowledge"]
    H --> I["No ack — escalates onward<br/>to incident commander"]
    I --> J["Human engaged in minutes,<br/>not hours"]

What good looks like

A synthetic test page runs monthly through the entire chain — alert fires, routes, pages the on-call, and (on a no-acknowledge branch tested at least quarterly) escalates onward. The drill isn't complete until a human acks.
Every page-severity alert terminates at a human who must acknowledge, with escalation that continues past the secondary — to the incident commander or an always-staffed rotation — instead of stopping.
Offboarding and team-transfer checklists include a paging audit: search every rotation, escalation policy, and alert route for the departing person's name before their accounts close.
Automation reconciles rotations against the directory weekly and flags slots pointing at deactivated accounts, empty slots, and rotations with a single member.
Page-severity alerts never route to chat channels as the primary path — chat is a mirror for visibility, the pager is the delivery mechanism, and the distinction is written down.
Acknowledge-latency is tracked as a metric — time from alert fire to human ack, per rotation — so a decaying paging path shows up as a trend, not a surprise.
Monthly on-call review covers pages per shift and actionability rate — a noisy pager is what created the muted channel in the first place; fixing noise is part of fixing routing.

The lesson

An alert no human receives is a log line with self-esteem. The paging path is production infrastructure: test it end-to-end with synthetic pages on a schedule, audit rotations after every reorg and offboarding, and require every page-severity alert to have an escalation chain that terminates at a human who must acknowledge.

Named concepts: alert routing, escalation policies, synthetic page testing, on-call hygiene, org-change checklists.

Real-world echoes

No single public incident to point at with confidence — dead rotation slots and muted alert channels rarely headline a public postmortem. That's part of what this drill names: the failure is invisible until the outage and embarrassing afterward, so it stays in private incident reports. Nearly every long mean-time-to-detect story includes a quieter sentence like "the alert fired, but…"

Transfer

When did we last send a test page through the whole chain — including escalation on no-ack? If the answer is "never," schedule one this week.

Make it a ticket: schedule the first synthetic end-to-end page for this week (including the no-ack escalation branch), and add "audit rotations and escalation policies" as a required line item to the offboarding and reorg checklists.

Drill 33 — The Runbook From Two Reorgs Ago — the rotation slot and step 7's dissolved team rotted in the same reorg; both are human-seam infrastructure nobody tests.
Drill 34 — Locked Out at the Worst Moment — the escalation finally reaches a human — who then can't get in.
Other themes — Drill 2 (Green Dashboards, Zero Traffic) is the monitoring-side twin: alert on the absence of success; Drill 25 (The Key That Expired on Saturday) — another quiet timer that fired when nobody was looking.