Drill 6 — The Health Check That Checked Nothing

Theme: Silent Failures · Format: 20-min drill · Difficulty: advanced

The story

Every service has a /health endpoint; the load balancer and the on-call dashboard both rely on it. The scoring service's health check returns 200 OK — it's a static handler that confirms the process is up. Tonight the service can't reach its database: every scoring request fails, but the process is running, so /health keeps smiling. The load balancer keeps routing traffic to it.

⏸ Stop here and ask the room: what should a health check actually assert?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

What's the difference between "the process exists" and "the service works"?

• Listen for: a static 200 asserts only that a process is resident in memory and answering one trivial route; "works" means the service can complete a representative request end to end, dependencies included. The gap between those two claims is exactly where tonight's outage lives.
• Weak signal: "if the process is up it usually works" — the drill is the counterexample. Anyone defending the static handler as "good enough" is describing the healthy case and ignoring that health checks exist for the unhealthy one.

Should a health check test dependencies — and what happens if it does and the dependency blips?

• Listen for: "it depends on who consumes the check" — the mature answer. Deep dependency checks belong in alert-only probes; wiring them into per-instance ejection means a shared dependency blip makes every instance report unhealthy simultaneously — perfectly correlated failure, fleet-wide ejection, blip becomes outage.
• Weak signal: an unqualified "yes, check everything, real health means dependencies" — that's the second half of the story. Enthusiasm for deep checks without asking what the consumer does on failure is how the fix becomes worse than the bug.

Who consumes each kind of check?

• Listen for: three consumers, three checks, three actions — the orchestrator restarts on liveness failure, the load balancer stops routing on readiness failure, a human gets paged on deep-health failure. Each check's content must be designed backward from the action its consumer takes.
• Weak signal: "one endpoint for everything, keeps it simple" — conflated semantics is the trap. When the same check drives restarts, routing, and paging, no one can safely make it deeper or shallower, and it drifts toward asserting nothing.

The reveal

Worse: when the team later "fixes" the check to ping the database, a 30-second database blip causes every instance to report unhealthy simultaneously — the load balancer ejects the entire fleet and a blip becomes an outage.

Why this happens — the mechanics

Health endpoints are day-one scaffolding. Someone needs /health to exist so the load balancer can be configured, writes return 200, and intends to come back later. It works — in the sense that deploys pass and routing flows — so it survives every subsequent review, because nothing about a passing health check invites scrutiny. The check's meaning was never designed; it accreted. And a health check is only meaningful relative to the action taken when it fails: the code says "200 OK," but the real semantics are "the load balancer will keep sending customer traffic here." Nobody wrote that sentence down, so nobody noticed it was false.

The overcorrection is just as characteristic as the original sin. After the incident, "check nothing" is obviously wrong, so the pendulum swings to "check everything" — ping the database, the cache, the feature service. But this changes the failure domain of the check without anyone deciding to: a per-instance signal now transmits the state of a shared dependency. Per-instance remediation (eject this box, restart that one) is exactly wrong for shared failures — ejecting all instances because the database blipped helps nobody and converts a 30-second degradation into a self-inflicted outage. Automation amplifies whatever signal you feed it; correlated signals produce correlated, fleet-wide actions.

The discipline that prevents both failures is asking, for every check: what does the consumer do when this fails, and is the thing being asserted in the same failure domain as that action? Restart-scoped checks (liveness) must only assert restart-fixable things. Routing-scoped checks (readiness) must only assert this-instance things. Everything deeper — real dependencies, end-to-end correctness — belongs with the one consumer that can exercise judgment about shared failures: a paged human.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant DB as Database
    participant SVC as Scoring instance
    participant HC as Health check
    participant LB as Load balancer
    Note over DB: Becomes unreachable
    SVC--xDB: Every scoring request fails
    LB->>HC: GET /health
    HC-->>LB: 200 OK - static handler<br/>checks nothing (silent)
    LB->>SVC: Keeps routing traffic
    Note over LB: Users eat failures until<br/>a human notices
    Note over HC: Later fix - health check<br/>now pings the database
    Note over DB: 30-second blip
    HC-->>LB: All instances unhealthy<br/>at once
    LB->>SVC: Ejects the entire fleet
    Note over LB,DB: A blip becomes a<br/>self-inflicted outage

Solution — where the guardrails go:

flowchart TD
    A["Database becomes unreachable"] --> B["Scoring requests start failing"]
    B --> R["Readiness check: can this<br/>instance serve right now"]
    R --> R2["LB stops routing to<br/>affected instances only"]
    L["Liveness check: is the<br/>process alive"] --> L2["Used only for restart<br/>decisions"]
    B -.-> P["Deep health: synthetic<br/>end-to-end scoring probe"]
    P -.-> AL["Alarm pages a human in minutes"]
    AL --> H["Human sees fleet-wide<br/>dependency failure"]
    H --> F["Failure domain isolated —<br/>no automatic fleet<br/>ejection"]

What good looks like

• Liveness checks that are process-internal only — event loop responsive, no deadlock, memory sane — consumed exclusively by the orchestrator for restart decisions. Never a dependency in sight: restarting a process does not fix a down database.
• Readiness checks scoped to "can this instance serve right now" — connection pool has capacity, config loaded, caches warm — consumed by the load balancer for routing only, and deliberately excluding shared dependencies so instances can't all fail the check identically at once.
• A minimum-healthy-hosts floor on the load balancer (e.g., never eject below 50% of the fleet) as the backstop for correlated readiness failures — and know your LB's fail-open behavior: many route to all instances when none report healthy. Whether yours does should be written down, not discovered.
• Deep health as a synthetic end-to-end scoring probe every 1–5 minutes, asserting on the content of the answer — wired to paging only, never to routing or restarts. Fleet-wide deep-health failure with green readiness is the signature of a shared-dependency outage, and that pattern itself is diagnostic for the on-call.
• A health-check inventory, one line per endpoint: what it asserts, who consumes it, what action failure triggers. Reviewed when any new service onboards, and whenever an LB or orchestrator config changes.
• A game day for the failure domain: block the database in staging, verify affected instances leave rotation correctly, the ejection floor holds, the deep-health page fires — and the fleet does not self-destruct. An ejection policy that has never been exercised is a guess.

The lesson

Distinguish liveness (restart me if this fails) from readiness (don't route to me right now) from deep health (synthetic end-to-end probes, alerting only). Wiring deep checks into fleet-ejection logic converts dependency blips into self-inflicted outages.

Named concepts: liveness vs. readiness probes, shallow vs. deep health checks, failure-domain isolation.

Real-world echoes

This class rarely makes headlines precisely because it fails silently — the famous incidents are the loud ones. The static-200 half of this drill lives quietly in thousands of codebases right now; the fleet-ejection half surfaces in postmortems as "our own automation amplified a transient blip," usually without a press release.

Transfer

Open our health endpoints today: what do they actually assert? Which consumer acts on each one — and can a dependency blip make them all fail at once?

Make it a ticket: Build the health-check inventory (endpoint → assertion → consumer → action), fix any deep dependency check currently wired to routing or restarts, and set a minimum-healthy-hosts floor on the load balancer.

Related

• Drill 2 — Green Dashboards, Zero Traffic — the same cure applies to both: synthetic probes that assert on real answers, because both static checks and silent dashboards report health without evidence.
• Drill 1 — The Model That Scored Everyone the Same — "the process is up" vs. "the service works," played out at the model layer instead of the endpoint layer.
• Drill 24 — The Autoscaler That Fed the Fire, theme 4 — the other face of automation amplifying a transient signal into a full outage.