Drill 3 — The Average That Hid the Outage
Theme: Silent Failures · Format: 20-min drill · Difficulty: intro
The story
A regional deployment issue takes one client integration to a 100% failure rate for six hours. That client is 4% of traffic. Global error rate ticks from 0.2% to 4.2% — under the 5% alert threshold. Global p99 barely moves.
⏸ Stop here and ask the room: the client's CEO calls your CEO before any alert fires. What went wrong in the monitoring design?
Run it — facilitator script
Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.
Which aggregations hide a total outage for a minority of traffic?
- Listen for: the arithmetic — a slice carrying x% of traffic, fully down, can move a global rate by at most x points, so any global threshold above x is structurally blind to it. Strong answers notice this applies to means, global error rates, and global percentiles alike.
- Weak signal: "averages lie, use p99" — percentiles are still global aggregations. A 4% slice fully failing can leave p99 nearly untouched (and did, in the story). Swapping one global statistic for another doesn't remove the blindness.
What slicing would have caught it in minutes?
- Listen for: SLIs (Service Level Indicators — the numbers that measure whether the service does its job) sliced along dimensions customers actually experience — per tenant, per region, per endpoint — where the failing client's own error rate reads a screaming 100% from the first minute.
- Weak signal: "lower the global threshold to 1%" — that trades one blindness for alert noise, and still misses any client under 1% of traffic. Threshold-tuning on aggregates is not slicing.
What thresholds make sense per-slice?
- Listen for: thresholds relative to each slice's own baseline, with a minimum-sample guard so tiny tenants don't page on noise; error-budget burn-rate alerting per tenant; and awareness that the biggest clients by contract value deserve slices regardless of traffic share.
- Weak signal: copying the global 5% onto every slice with no baseline or sample-size thinking — low-traffic tenants will flap constantly, the team will mute the alerts, and the blindness returns wearing a per-tenant costume.
The reveal
Every metric was watched — globally. The failing 4% was invisible inside healthy averages. The client experienced a total outage while the provider's dashboards showed a rounding error.
Why this happens — the mechanics
The technical root is dilution. An aggregate is a weighted sum, and a minority slice's total catastrophe arrives at the dashboard as a small perturbation — 4% of traffic failing completely is, to the global error rate, arithmetically identical to 100% of traffic failing 4% of the time. Those are wildly different events for customers, and one number cannot distinguish them. The provider watches the mean; each client lives at their own coordinate. There is no global statistic — mean, median, p99 — that preserves the experience of every slice.
The organizational root is that dashboards begin life global because global is what's cheap. Per-tenant metrics multiply cardinality (every metric × every tenant × every region), which costs real money in the metrics pipeline and real effort in threshold tuning — so slicing gets deferred as an optimization, indefinitely. And teams fear alert fatigue: a handful of global alerts feels disciplined, a thousand per-slice alerts feels like noise waiting to happen. The fear is legitimate; the resolution — baselines, burn rates, sample guards — is engineering work nobody scheduled.
There is also a quiet mismatch of perspectives: SLOs (Service Level Objectives — the targets set on SLIs) get written from the provider's seat, describing the service as a whole, while contracts and reputations are per-client. In financial services this bites hardest — a lender or processor integration that is 100% down is a client unable to do business, and the escalation path that finally fires is the CEO-to-CEO phone call, the most expensive alerting channel in existence. It fires precisely because it is the only monitor sliced the way the customer experiences the service.
Diagrams
Problem — how the failure unfolds:
sequenceDiagram
autonumber
participant CL as Client - 4% of traffic
participant SV as Service
participant DB as Global dashboard
participant CEO as Client CEO
CL--xSV: Integration fails 100%
SV->>DB: Error rate ticks 0.2% to 4.2%
Note over DB: Under the 5% threshold -<br/>no page (silent)
Note over CL: Fully down for six hours
CEO->>SV: Calls your CEO directly
Note over CEO,SV: Aggregates hid a total<br/>outage for a minority slice
Solution — where the guardrails go:
flowchart TD
A["Regional deployment issue"] --> B["One client integration<br/>fails 100%"]
B --> S["SLIs sliced per tenant,<br/>region, endpoint"]
S --> T["Per-tenant error rate<br/>reads 100%"]
T -.-> AL["Per-slice alarm fires in<br/>minutes"]
S --> U["Per-tenant error budget<br/>burns fast"]
U -.-> AL
AL --> R["Region rolled back before<br/>the client escalates"]
D["Global aggregate still<br/>shows 4.2%"] -.-> N["Kept for trends, never for<br/>paging"]
What good looks like
- Per-tenant error-rate and latency SLIs for every tenant above a traffic floor — and for the top clients by contract value unconditionally, however small their traffic.
- Per-slice thresholds anchored to the slice's own baseline: page when a tenant's error rate exceeds max(5%, 10× its trailing baseline) for 5 minutes and the window contains at least 100 requests — the sample guard is what keeps small slices from flapping.
- Per-tenant error-budget burn-rate alerts, multi-window style: page at ~14× burn over 1 hour, open a ticket at ~6× burn over 6 hours — fast detection for total outages, quiet handling of slow leaks.
- Region and endpoint dimensions alongside tenant — customers experience the intersection (this client, this region, this endpoint), and a regional deployment issue shows up there first.
- Global aggregates demoted, explicitly: kept on dashboards for trend analysis, annotated "never for paging" so nobody re-wires an alert to them in a hurry.
- Synthetic probes replaying each top client's integration path every few minutes, so a per-tenant outage is detectable even when the client's own traffic goes quiet at night.
- A quarterly blindness test: simulate one tenant at 100% failure in staging (or via shadow traffic) and verify the per-slice page fires before any human would have noticed.
The lesson
Slice SLIs along the dimensions customers actually experience: per tenant, per region, per segment, per endpoint. An SLO that your biggest client can fully breach without paging anyone is not an SLO — it's a decoration.
Named concepts: dimensional SLOs, per-tenant error budgets, aggregation bias.
Real-world echoes
This class rarely makes headlines precisely because it fails silently — the famous incidents are the loud ones. Every regional cloud outage that stayed green on a global status page is this drill wearing a bigger costume, but the per-tenant version lives in postmortems that never go public.
Transfer
Could one of our clients be 100% down right now inside our green aggregates? Which one would we notice last?
Make it a ticket: List our top five clients by revenue; for each, verify a per-tenant error-rate alarm exists with a written threshold and a named pager — file every gap as its own ticket.
Related
- Drill 1 — The Model That Scored Everyone the Same — the same lesson at the semantic layer: the metrics were watched, just not the ones (or the slices) that carried the meaning.
- Drill 2 — Green Dashboards, Zero Traffic — its sibling blindness: this drill hides a slice inside the aggregate, that one hides the whole service inside silence.
- Drill 62 — The 99.9% That Didn't Matter, theme 10 — the contractual version: an SLA aggregate that stayed green through the outage that mattered.
