Pattern — SLO / Error-Budget Policy

The problem it solves

Two failure shapes share one root: nobody agreed, in advance and in numbers, on how good "good enough" is. In Drill 3 — The Average That Hid the Outage, one client integration is 100% down for six hours while the global error rate ticks from 0.2% to 4.2% — under a 5% alert threshold — so the only monitor that fires is the client CEO phoning your CEO. A single global aggregate is arithmetically blind to any slice smaller than the threshold. The mirror-image failure is the team with a green dashboard that still ships a reliability regression every sprint, because "is it healthy?" was never a measured target with a consequence. An SLO / error-budget policy turns reliability from a vibe into a contract with a number, a window, and a pre-agreed action when the number is spent.

The pattern

Define a Service Level Indicator (SLI) — a ratio of good events / valid events (e.g. requests served under 300 ms with a 2xx/3xx). Set a Service Level Objective (SLO): a target on that SLI over a rolling window — 99.9% over 28 days. The complement is your error budget: 100% − SLO of allowed badness — 0.1%, which over 28 days is about 40 minutes of downtime you are permitted to spend. You alert not on the raw SLI but on burn rate — how fast the budget is draining relative to "spend it evenly" — with fast and slow windows so a total outage pages in minutes while a slow leak opens a ticket. And you write an error-budget policy: a document, signed by product and engineering, that says what happens when the budget hits zero — typically feature launches freeze and the team works only reliability until the budget recovers. Crucially, for Drill 3, the SLI is sliced along dimensions customers actually experience — per tenant, per region, per endpoint — so a budget can burn for one client without being diluted into the global average.

How to apply it

1. Write the SLI as an event ratio, not an average. availability = good_requests / valid_requests over a window; latency as requests_faster_than_300ms / valid_requests ≥ 99%. A ratio composes and slices; a mean (Drill 3's trap) hides a total outage of a minority behind healthy bulk. Pick a rolling 28-day window (four weeks — stable across weekday/weekend cycles), not calendar month.
2. Set the target and derive the budget in real units. 99.9% over 28 days → error budget 0.1% → ≈ 40 min unavailable / 28 days; 99.95% → ≈ 20 min; 99.99% → ≈ 4 min. Put the time on the dashboard — "37 of 40 budget-minutes spent" is legible in a way "99.91%" is not. Don't set 99.999% because it sounds good — every extra nine roughly 10×'s the cost and forbids nearly all deploys.
3. Alert on multi-window, multi-burn-rate — the load-bearing config. Burn rate = (1 − SLI) / (1 − SLO); 1× spends the whole budget exactly at window's end. Use the Google SRE table: page at 14.4× burn over 1 h (short window 5 min must agree) — that consumes 2% of a 28-day budget in an hour; page at 6× over 6 h (30-min short) — 5%; ticket at 3× over 24 h (2-h short) — 10%. The short window is what stops a recovered blip from paging after the fact.
4. Slice the SLO — the fix for Drill 3. Compute the SLI per tenant, per region, per endpoint, not just globally. Page a tenant when its own burn crosses the table and the window holds ≥ 100 requests (a sample-size guard so a 12-request tenant can't flap). Cover the top clients by contract value unconditionally, however small their traffic share. Keep the global aggregate for trend lines only, annotated "never for paging."
5. Write the error-budget policy and give it teeth. In-budget → ship freely. Budget exhausted → freeze feature launches, redirect the team to reliability, and require a leadership sign-off to override. Both product and eng leaders sign it before the incident, so the freeze is a pre-agreed rule, not a fight during the outage.

flowchart TD
    A["SLI = good events / valid events<br/>(per tenant, region, endpoint)"] --> B["SLO target 99.9%<br/>over rolling 28 days"]
    B --> C["Error budget = 0.1%<br/>(about 40 min / 28d)"]
    C --> D{"Burn rate =<br/>(1 - SLI) / (1 - SLO)"}
    D -->|"14.4x over 1h (5m agrees)"| E["Page now — 2% budget/hour"]
    D -->|"3x over 24h (2h agrees)"| F["Ticket — slow leak"]
    D -->|"budget hits 0"| G["Error-budget policy:<br/>freeze launches, fix reliability"]

When to use it — and when not

Reach for it on any user-facing service where reliability trades against feature velocity and where "how down was it, for whom?" needs an agreed answer — exactly the multi-tenant, per-contract world of Drill 3. It is not free: an SLO is a promise you must staff to keep, so don't set one you won't defend — an unenforced error-budget policy (a freeze nobody ever honors) is worse than none, because it trains the org to ignore the number. Skip formal SLOs on internal batch jobs or early-stage prototypes where a simple threshold alert is enough and the ceremony isn't repaid. And know the tradeoff of slicing: per-tenant SLOs multiply cardinality (tenants × regions × endpoints) and cost real money in the metrics pipeline — so slice the dimensions customers feel and the clients that pay, not every possible cross-product. A global-only SLO is the specific anti-pattern Drill 3 punishes: it can stay green while your biggest client is 100% down.

Signals it's working

• The budget burns and the policy actually bites: in a bad month the error budget reaches zero and a feature freeze is declared and honored — the number changed a decision, which is the whole point.
• Per-slice pages beat the phone call: a single tenant at 100% failure triggers a per-tenant burn-rate page within minutes, not a CEO escalation hours later. Prove it with a quarterly blindness test — simulate one tenant fully failing in staging and assert the slice pages before a human would notice.
• Alerts are few and true: burn-rate alerting cuts pages versus a static threshold; you page on fast burn, ticket on slow leaks, and small tenants don't flap because the sample-size guard (≥ 100 req) holds.
• The dashboard reads in minutes, not decimals ("6 of 40 budget-minutes spent this window"), and the global aggregate is explicitly demoted — annotated never-for-paging so nobody rewires an alert to it.

Seen in the wild

• Google SRE originated SLI/SLO/error-budget and the multi-window, multi-burn-rate alerting recipe (SRE Workbook, "Alerting on SLOs") — the 14.4× / 6× / 3× table above is theirs.
• Google Cloud, Datadog, Nobl9, Grafana, and Prometheus (Sloth / slo-generator) all ship first-class SLO objects with burn-rate alert generation and error-budget dashboards.
• Stripe, Slack, and most SaaS platforms publish availability SLAs and run internal per-service, often per-tenant, SLOs behind them — the SLA is the customer contract, the SLO is the tighter internal target that protects it.

See it

CardsScene and TilesScene carry the load-bearing config — SLI ratio, 99.9% over 28 days → 40 budget-minutes, and the 14.4× / 6× / 3× burn-rate table; CompareScene shows the tradeoff, a global aggregate staying green at 4.2% while a per-tenant slice reads a screaming 100%.

Named concepts: service-level objectives, error budgets, burn-rate alerting, error-budget policy.