BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 3 — The Average That Hid the Outage

Theme: Silent Failures · Format: 20-min drill · Difficulty: intro

The story

A regional deployment issue takes one client integration to a 100% failure rate for six hours. That client is 4% of traffic. Global error rate ticks from 0.2% to 4.2% — under the 5% alert threshold. Global p99 barely moves.

⏸ Stop here and ask the room: the client's CEO calls your CEO before any alert fires. What went wrong in the monitoring design?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Which aggregations hide a total outage for a minority of traffic?

What slicing would have caught it in minutes?

What thresholds make sense per-slice?

The reveal

Every metric was watched — globally. The failing 4% was invisible inside healthy averages. The client experienced a total outage while the provider's dashboards showed a rounding error.

Why this happens — the mechanics

The technical root is dilution. An aggregate is a weighted sum, and a minority slice's total catastrophe arrives at the dashboard as a small perturbation — 4% of traffic failing completely is, to the global error rate, arithmetically identical to 100% of traffic failing 4% of the time. Those are wildly different events for customers, and one number cannot distinguish them. The provider watches the mean; each client lives at their own coordinate. There is no global statistic — mean, median, p99 — that preserves the experience of every slice.

The organizational root is that dashboards begin life global because global is what's cheap. Per-tenant metrics multiply cardinality (every metric × every tenant × every region), which costs real money in the metrics pipeline and real effort in threshold tuning — so slicing gets deferred as an optimization, indefinitely. And teams fear alert fatigue: a handful of global alerts feels disciplined, a thousand per-slice alerts feels like noise waiting to happen. The fear is legitimate; the resolution — baselines, burn rates, sample guards — is engineering work nobody scheduled.

There is also a quiet mismatch of perspectives: SLOs (Service Level Objectives — the targets set on SLIs) get written from the provider's seat, describing the service as a whole, while contracts and reputations are per-client. In financial services this bites hardest — a lender or processor integration that is 100% down is a client unable to do business, and the escalation path that finally fires is the CEO-to-CEO phone call, the most expensive alerting channel in existence. It fires precisely because it is the only monitor sliced the way the customer experiences the service.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant CL as Client - 4% of traffic
    participant SV as Service
    participant DB as Global dashboard
    participant CEO as Client CEO
    CL--xSV: Integration fails 100%
    SV->>DB: Error rate ticks 0.2% to 4.2%
    Note over DB: Under the 5% threshold -<br/>no page (silent)
    Note over CL: Fully down for six hours
    CEO->>SV: Calls your CEO directly
    Note over CEO,SV: Aggregates hid a total<br/>outage for a minority slice

Solution — where the guardrails go:

flowchart TD
    A["Regional deployment issue"] --> B["One client integration<br/>fails 100%"]
    B --> S["SLIs sliced per tenant,<br/>region, endpoint"]
    S --> T["Per-tenant error rate<br/>reads 100%"]
    T -.-> AL["Per-slice alarm fires in<br/>minutes"]
    S --> U["Per-tenant error budget<br/>burns fast"]
    U -.-> AL
    AL --> R["Region rolled back before<br/>the client escalates"]
    D["Global aggregate still<br/>shows 4.2%"] -.-> N["Kept for trends, never for<br/>paging"]

What good looks like

The lesson

Slice SLIs along the dimensions customers actually experience: per tenant, per region, per segment, per endpoint. An SLO that your biggest client can fully breach without paging anyone is not an SLO — it's a decoration.

Named concepts: dimensional SLOs, per-tenant error budgets, aggregation bias.

Real-world echoes

This class rarely makes headlines precisely because it fails silently — the famous incidents are the loud ones. Every regional cloud outage that stayed green on a global status page is this drill wearing a bigger costume, but the per-tenant version lives in postmortems that never go public.

Transfer

Could one of our clients be 100% down right now inside our green aggregates? Which one would we notice last?

Make it a ticket: List our top five clients by revenue; for each, verify a per-tenant error-rate alarm exists with a written threshold and a named pager — file every gap as its own ticket.

Related