Drill 2 — Green Dashboards, Zero Traffic

Theme: Silent Failures · Format: 20-min drill · Difficulty: intro

The story

During an API-gateway migration, a routing rule quietly sends the scoring endpoint's traffic to the old, about-to-be-decommissioned deployment. The new deployment — the one with all the dashboards and alerts — serves zero requests. Error rate: 0% (of nothing). CPU: healthy. Alerts: silent.

⏸ Stop here and ask the room: what's wrong with every alert this team has?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Why did no alert fire?

• Listen for: the structural observation — every alert was conditioned on the presence of bad events: errors, slow requests, resource pressure. With zero traffic there are no events at all, so every rate is vacuously healthy. 0% errors of nothing is not health; it's absence.
• Weak signal: "the thresholds were too loose" — no error-rate threshold, however tight, catches this. Tuning thresholds on presence-of-failure alerts misses that the entire alert class is blind to silence.

What alert fires when a service receives nothing?

• Listen for: minimum-throughput alarms — "fewer than N successful requests in 10 minutes" — plus end-to-end synthetic transactions that exercise the real path continuously and assert on the answer, so success itself becomes a monitored event.
• Weak signal: "monitor the gateway instead" — that just relocates the blind spot. The gateway saw plenty of traffic; it was routing it to the wrong place. The room needs both a floor on the service and a probe through the path.

How do you verify a routing change actually took?

• Listen for: post-change verification as a required deploy step — watch traffic counters rise on the intended target and fall on the old one within a defined window, and don't decommission anything until the old deployment shows observed-zero traffic over a soak period.
• Weak signal: "check that the config was applied" — config state is not traffic behavior. The rule was applied exactly as written; it was written wrong. Only observed traffic proves a routing change.

The reveal

Thirty hours later the old deployment is decommissioned on schedule and traffic goes to a black hole. The outage that finally pages someone was detectable thirty hours earlier — as absence.

Why this happens — the mechanics

The technical root is the denominator problem. Nearly every alert a team writes is a ratio — errors over requests, slow requests over requests — and when the denominator goes to zero, the ratio either reads as perfect or stops being computed at all. Monitoring systems are event-driven: no events, no evaluation, no page. Silence is not a value most alerting pipelines can even represent, so the healthiest-looking service in the fleet is one that does nothing.

Alerts also encode history. Teams write alerts for failures they have personally seen — a spike, a crash, a timeout — which are all presence signals. Nobody has been burned by absence yet, so nobody has written the absence alert. RED metrics (Rate, Errors, Duration) name the three basics for any request-driven service, and Rate is reliably the one nobody alerts on, because "too little traffic" doesn't feel like a failure mode of our service — it feels like someone else's problem.

Migrations sharpen all of this with split ownership. The team that owns the shiny new deployment — and all its dashboards — is not the team that owns the gateway routing rule, and the decommission of the old deployment runs on a calendar, not on a verification gate. Each group's slice looks fine: the gateway is routing, the new deployment is idle-healthy, the old one is serving as it always has. The failure exists only in the joint view that nobody owns — until the calendar-driven decommission converts thirty hours of quiet misrouting into a loud black hole.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant GW as Gateway migration
    participant NEW as New deployment
    participant OLD as Old deployment
    participant DB as Dashboards
    GW->>OLD: Routing rule sends all<br/>traffic here (silent)
    Note over NEW: Zero requests arrive
    NEW->>DB: Error rate 0% - of nothing
    Note over DB: Green - no minimum-traffic<br/>alarm exists
    GW->>OLD: Decommissioned on schedule
    Note over OLD: Traffic falls into<br/>a black hole
    Note over DB: 30 hours later the outage<br/>finally pages someone

Solution — where the guardrails go:

flowchart TD
    A["API-gateway migration"] --> B["Routing change deployed"]
    B --> V["Required deploy step:<br/>verify traffic actually<br/>shifted"]
    V -.-> AL["Alarm fires in minutes"]
    B --> C["New deployment throughput<br/>drops to zero"]
    C --> T["Minimum-throughput alarm:<br/>N successes per 10 minutes"]
    T -.-> AL
    P["End-to-end synthetic<br/>transactions on the real<br/>path"] -.-> T
    AL --> R["Routing rule corrected —<br/>decommission paused"]

What good looks like

• A minimum-throughput alarm on every request-serving service: fewer than N successful responses in 10 minutes pages on-call, with N derived from the historical low-water mark (say, 20% of the quietest hour's traffic) so it survives weekends and holidays.
• Synthetic transactions through the public path: a scripted scoring request every 1–5 minutes entering via the gateway — not aimed directly at the pod — asserting on the response content, not just the 200. These make the throughput floor testable even at 3 a.m. with no real traffic.
• An automated verification gate on routing changes: the deploy pipeline checks that request counters rise on the intended target and fall on the old one within 15 minutes of the change, and rolls back automatically if they don't.
• A decommission checklist with an observed-zero requirement: the old deployment must show zero non-synthetic traffic for a defined soak period (e.g., 7 days), with the last callers identified from access logs, before anyone powers it down. Calendars don't decommission; evidence does.
• Absolute request counts on dashboards next to every rate — a zero should render as an alarm-red number, not as an empty, innocent-looking graph.
• An alert-coverage review question asked per service: "if this received zero input for six hours tonight, what fires?" Record the answer; "nothing" is a ticket.

The lesson

Alert on the absence of success, not just the presence of failure: minimum-throughput alarms, end-to-end synthetic transactions that exercise the real path, and post-change traffic verification as a required deploy step.

Named concepts: synthetic monitoring / probes, RED metrics (Rate matters!), post-deployment verification.

Real-world echoes

Rhymes with the AWS S3 outage of February 2017: the AWS status dashboard depended on S3 itself, so while S3 was down the dashboard couldn't show the failure — the monitoring surface everyone watched was structurally unable to display the problem. A green dashboard proves the dashboard is green, nothing more.

Transfer

If our main pipeline received zero input for six hours tonight, what would fire — anything?

Make it a ticket: For each tier-1 service, add a minimum-throughput alarm with a written floor and a synthetic probe through the real entry path; log every "nothing would fire" answer as its own follow-up ticket.

Related

• Drill 6 — The Health Check That Checked Nothing — the same gap between "reports healthy" and "actually serving," seen from the health-endpoint side.
• Drill 1 — The Model That Scored Everyone the Same — presence-of-failure alerting misses semantic failure; here it misses absence entirely.
• Drill 29 — Both On-Call, Neither Paged, theme 5 — the organizational version of a page that never comes.