Theme 1 — Silent Failures
The reflex this theme builds: green dashboards prove nothing — what would tell us this is actually working?
Why this theme matters
Traditional monitoring was built for systems that fail loudly — a crashed process, a 500 error, a full disk. AI/ML systems fail quietly: the pipeline runs, the API returns 200 OK, the dashboard stays green, and the answers are wrong. A model scoring garbage looks identical to a model scoring well, from the infrastructure's point of view. That's why the mean time to detect (MTTD — how long a failure goes unnoticed) for semantic failures is measured in weeks, while for crashes it's minutes.
The mindset shift: stop asking "is the service up?" and start asking "is the service right?" Every drill here is a variation on that gap. Abbreviations are expanded on first use; the full reference is glossary.md.
The drills
| # | Drill | One-line hook | Problem diagram | ~Difficulty |
|---|---|---|---|---|
| 1 | The Model That Scored Everyone the Same | Graceful degradation imputes 40% of feature vectors — approvals go near-random for 19 days while every technical signal stays green. | sequence | core |
| 2 | Green Dashboards, Zero Traffic | A routing rule sends all traffic to the old deployment — the monitored one serves zero requests, with an error rate of 0% (of nothing). | sequence | intro |
| 3 | The Average That Hid the Outage | One client is 100% down for six hours — the global error rate ticks to 4.2%, safely under the 5% threshold. | sequence | intro |
| 4 | The Stale Score | A TTL typo (21m → 21d) freezes velocity features — a fraud ring discovers it before any monitor does. | sequence | core |
| 5 | The Twice-a-Year Bug | A performance dip every March and November gets a nickname instead of a root cause — it was daylight-saving time. | timeline | core |
| 6 | The Health Check That Checked Nothing | /health returns 200 while every request fails — then the "fix" turns a 30-second database blip into a fleet-wide ejection. | sequence | advanced |
Best practices — the silent-failure checklist
What to actually do, distilled from drills 1–6:
- Define SLIs (Service Level Indicators — the numbers that measure whether the service does its job) at the semantic level, not just the infrastructure level: score-distribution stability, imputation/null rates, output length and format-validity, field-fill rates. If the service produces decisions, an SLI must describe decision quality.
- Alert on the absence of success, not just the presence of failure: minimum-throughput alarms ("fewer than N successful requests in 10 minutes"), and end-to-end synthetic transactions (scripted fake requests that exercise the real path and assert on the answer) running continuously.
- Slice every SLI by the dimensions customers experience — tenant, region, segment, channel, document type. Set per-slice thresholds. Aggregates are where minority outages hide.
- Give every derived or cached data product a freshness SLI with a staleness alarm. "How old is this value allowed to be?" must have a written answer.
- Treat config changes as deploys: reviewed, canaried (rolled to a small slice first), and verified after rollout. Most silent failures in this theme started as a "harmless" config or schedule change.
- Separate liveness (restart me), readiness (don't route to me), and deep health (page a human) — and never wire deep dependency checks into automatic fleet ejection, or a dependency blip becomes a self-inflicted outage.
- Add a business-metric backstop: approval rate, queue depth, handling time, decisions per hour — with anomaly alarms. Business metrics catch the failure modes you failed to imagine; they are the monitor of last resort.
- Keep an anomaly registry: every unexplained wobble gets an entry with a timestamp. Review quarterly for patterns (calendar periodicity is a fingerprint — DST, month-end, expiry cycles). An anomaly with a nickname is an incident on layaway.
- Test your monitors by injecting failures (empty outputs, stale data, drifted distributions) in staging on a schedule. An alert that has never fired is unverified code.
- Measure your MTTD (Mean Time To Detect — how long failures go unnoticed) for past incidents. If detection came from a human noticing rather than a monitor firing, that's the gap list to work through.
Also see
- Facilitator guide — how to run drills, and the other ten themes.
- Glossary — every abbreviation, explained in plain language.