Pattern — Graceful Degradation

The problem it solves

When a dependency fails, a service has two honest choices — crash, or serve a reduced answer — and a dangerous third: serve a reduced answer while pretending nothing happened. In Drill 1 a scoring service takes the third path. Its feature service starts timing out after an upgrade, so 40% of applications arrive with mostly-null feature vectors; the scorer imputes the missing features with training-set means, exactly as designed, and keeps returning 200 OK. Scores compress into a narrow band around 0.52, approvals go nearly random — and because latency, error rate, and uptime all stay green, nobody notices for 19 days. The failure isn't the fallback. The failure is that the fallback was silent and unbounded: no signal fired when it activated, and no cap stopped it from papering over a decision it had no business making.

The pattern

Graceful degradation is serving a bounded, labeled reduced answer when a dependency fails, instead of either crashing or lying. The mechanism has three inseparable parts: a tiered fallback (full answer → degraded answer → refuse), so the choice is never just degrade-or-crash; a degradation signal emitted on every activation, so a fallback firing at scale is a visible incident; and a cap — a written threshold past which you stop pretending and fail explicitly. Degrading gracefully is a best practice. Degrading silently is the bug — a fallback without a signal converts a loud failure you'd have fixed in an hour into a quiet one you'll fix in three weeks.

How to apply it

1. Define fallback tiers, not degrade-or-crash. Rank responses: Tier 0 — the full, live answer; Tier 1 — a degraded answer (cached / last-known-good / imputed, with a confidence penalty); Tier 2 — refuse, returning a typed degraded/unavailable so the caller decides. Every tier below 0 carries a degraded: true flag and a degradation_reason in the response envelope, so downstream never mistakes a fallback for a first-class answer — the exact mistake Drill 1's approval pipeline made.
2. Emit a degradation signal on every fallback. Increment a counter labeled by tier and reason; the share of requests served below Tier 0 is your degradation SLI. On Drill 1's scorer: the fraction of requests scored with more than K% imputed features — warn at 2%, page at 5%. If the fallback path activates at scale, that is the incident; alarm in minutes, don't surface it in a weekly review.
3. Bound it with a cap — the fail-open / fail-closed line. Per response, cap how much you'll paper over. Below the cap, fail open (serve Tier 1). Above it, fail closed: e.g. if > 30% of a feature vector is imputed, or any top-20-by-importance feature is null, don't auto-decide — flag the decision as degraded and route it to manual review (Tier 2). Also alarm at the input edge: any top-20 feature's null rate > 3× its trailing 7-day baseline for 15 minutes.
4. Watch the output, not just the input. Add a population-stability monitor on the result distribution — compute PSI (Population Stability Index) or a KS-statistic hourly against a trailing baseline, warn at 0.10, page at 0.25 — so a silent compression to a narrow band (0.52) screams on day one, even when a fallback you never anticipated is the cause.
5. Bound fallback freshness. A cached / last-known-good Tier 1 gets a max staleness TTL (e.g. serve stale up to 15 min, then escalate to Tier 2). A fallback with no expiry is just a silent stale-read wearing a fresh timestamp.

flowchart TD
    A["Request"] --> B{"Dependency healthy?"}
    B -->|"yes"| T0["Tier 0 — full live answer"]
    B -->|"no"| C{"Past the cap? (> 30% imputed)"}
    C -->|"below cap"| T1["Tier 1 — degraded<br/>(cached / imputed) + degraded flag"]
    C -->|"above cap"| T2["Tier 2 — refuse / flag,<br/>route to manual review"]
    T1 --> S["Emit degradation SLI<br/>warn 2% · page 5%"]
    T2 --> S
    S --> M["PSI on output histogram<br/>warn 0.10 · page 0.25"]

When to use it — and when not

Reach for it whenever a non-critical dependency's failure shouldn't take the whole request down and a partial answer still has value — search without its personalization ranker, a feed without the recommendation row, a score served with a confidence penalty. The two hard rules: never degrade silently (Drill 1's whole lesson — a fallback without a signal is a 19-day outage waiting to happen), and never degrade where a wrong answer is worse than no answer. Moving money, authentication and authorization decisions, and eligibility calls past the cap must fail closed, not serve a confident guess. The cost is real: every fallback tier is code that must be tested (inject the dependency failure in staging quarterly and confirm the alarms fire and the cap routes to review), and every degraded answer is a small lie you're telling downstream — if you can't measure it and bound it, don't ship it.

Signals it's working

The degradation SLI exists and is nonzero-but-low in steady state — fallback activations are visible in minutes, not discovered in a weekly business review. The proof metric is MTTD (mean time to detect): for a dependency brownout it should drop from business-review cadence (days) to alarm cadence (minutes). The cap actually fires — count of requests refused or flagged above threshold is > 0 during an incident, proving it isn't dead code. The population-stability monitor holds flat in steady state and spikes when the output distribution shifts. Prove all of it in a game day: inject a synthetic null-flood / dependency timeout and assert the null-rate, degradation-rate, and PSI alarms all fire, and that above-cap requests route to manual review instead of scoring.

Seen in the wild

• Netflix (Hystrix) — when the personalized-ranking service is unavailable, the UI falls back to non-personalized / cached recommendations rather than showing an error; the fallback is explicit and instrumented.
• Cloudflare "Always Online" — serves a stale cached copy of a page when the origin is unreachable, clearly a degraded (older) version rather than a 5xx.
• DNS serve-stale (RFC 8767) — resolvers return expired-but-cached records past TTL when authoritative servers are unreachable, bounded by a max-stale window — graceful degradation with an explicit cap.
• ML serving / feature stores — impute missing features or fall back to a simpler model, attaching a confidence flag so downstream can down-weight or reject the prediction.

See it

The ChainScene carries the cap mechanism — dependency down → past the cap? → below: Tier 1 degraded + flag / above: Tier 2 refuse → route to manual review; the TilesScene shows the three fallback tiers and the MetricLineScene shows the load-bearing thresholds (degradation SLI warn 2% / page 5%, PSI 0.10 / 0.25, cap at > 30% imputed).

Named concepts: graceful degradation, fallback tiers, degradation SLI, fail-open vs fail-closed.