Pattern — Health-Check Design

The problem it solves

A health check is only meaningful relative to the action taken when it fails — yet most are written as day-one scaffolding (return 200) and never designed. Two failures follow, and they are mirror images (Drill 6). The shallow check asserts only "a process is resident in memory," so when the database is unreachable every scoring request fails while /health keeps smiling and the load balancer keeps routing traffic into the hole. The over-deep overcorrection pings the database inside a per-instance probe, so a 30-second shared-dependency blip makes all instances fail the check at once — the load balancer ejects the entire fleet and a blip becomes a self-inflicted outage. The common shape: a check whose content is in a different failure domain than the action its consumer takes.

The pattern

Design each check backward from the action its consumer takes, and keep its content in the same failure domain as that action. That means three distinct probes for three distinct consumers, never one endpoint for everything:

• Liveness — "is this process wedged?" Consumed by the orchestrator, action = restart. Assert only restart-fixable, process-internal things (event loop responsive, no deadlock). Never a dependency: restarting a pod does not fix a down database.
• Readiness — "should traffic come to this instance right now?" Consumed by the load balancer, action = stop routing here. Assert only this-instance things (local connection pool has capacity, config loaded). Deliberately exclude shared dependencies so instances cannot all fail identically at once.
• Deep health — "can the service actually complete a real request end to end?" A synthetic probe wired to paging only — the one consumer (a human) who can exercise judgment about a shared failure.

How to apply it

1. Split the endpoints and scope each one. Liveness on /livez, readiness on /readyz, deep health as an out-of-band synthetic probe. In Kubernetes:
   • livenessProbe: httpGet /livez, periodSeconds: 10, timeoutSeconds: 1, failureThreshold: 3 (≈30s of wedge before a restart). Handler returns 200 unless an internal watchdog says the event loop is blocked — no dependency calls.
   • readinessProbe: httpGet /readyz, periodSeconds: 5, timeoutSeconds: 1, failureThreshold: 3, successThreshold: 1. Handler checks local state only — DB connection-pool has a free slot (not a live round-trip to the DB), config parsed, caches warm. On fail this one instance leaves rotation; the others keep serving.
2. Set a minimum-healthy floor so correlated readiness failures can't eject the fleet. On the load balancer / mesh, never drop below 50% healthy hosts — this is Envoy's healthy_panic_threshold (default 50%, which fails open: below it, route to all hosts) and the equivalent of an AWS target-group minimum-healthy floor. Know your LB's fail-open behavior and write it down — many route to every instance when none report healthy, which is safer than routing to none but must be a decision, not a surprise.
3. Make deep health a real end-to-end probe on a timer, paged not routed. Run a synthetic scoring request every 60 seconds (1–5 min is the usual band), assert on the content of the answer, not just a 200. Wire it to paging only (PagerDuty / on-call), never to the readiness or liveness path. The diagnostic signature to alert on: fleet-wide deep-health red while readiness stays green = a shared-dependency outage — exactly the state per-instance ejection must not react to.
4. Keep an inventory and rehearse the failure domain. One line per endpoint — what it asserts · who consumes it · what action failure triggers — reviewed on every new-service onboard and every LB/orchestrator config change. Then run a game day: block the database in staging and verify affected instances leave rotation, the 50% floor holds, the deep-health page fires, and the fleet does not self-destruct. An ejection policy that has never been exercised is a guess.

flowchart TD
    A["Shared dependency blips<br/>(e.g. 30s DB outage)"] --> L["Liveness /livez<br/>(process-internal only)"]
    A --> R["Readiness /readyz<br/>(this-instance only)"]
    A --> D["Deep health<br/>(synthetic end-to-end, 60s)"]
    L -->|"restart"| L2["Orchestrator: no restart<br/>(nothing local is broken)"]
    R -->|"stop routing here"| R2["LB ejects only affected<br/>instances, floor holds at 50%"]
    D -->|"page a human"| D2["On-call sees fleet-wide deep-red<br/>+ readiness-green = shared outage"]
    R2 --> F["Failure domain isolated —<br/>no automatic fleet ejection"]
    D2 --> F

When to use it — and when not

Use the full three-probe split on any service behind an orchestrator and a load balancer where a dependency can fail independently of the process — the normal case for a stateful request service like Drill 6's scorer. Where it earns nothing, don't pay for it: a single-instance batch job or a CLI has no routing consumer, so readiness is meaningless — liveness alone is fine. The load-bearing rule is the anti-pattern to avoid: never wire a deep dependency check into per-instance ejection or restart logic. That is precisely the overcorrection that turns a blip into an outage. Deep checks belong to a human; shared-failure remediation requires judgment automation doesn't have. And resist collapsing the three into one "simple" endpoint — conflated semantics means no one can safely make it deeper or shallower later, and it drifts back toward asserting nothing.

Signals it's working

• During a real dependency blip: readiness stays green fleet-wide and deep-health goes red fleet-wide — the LB does not eject en masse, and the on-call gets exactly one page. That divergence is the pattern in force.
• Healthy-host count never crosses the 50% floor as an automated action; any ejection below it was a human decision.
• The health-check inventory exists and each row's asserted content matches its consumer's failure domain (no shared dependency inside a per-instance probe).
• A recurring game day passes: DB blocked in staging → affected instances drain, floor holds, page fires, fleet survives.

Seen in the wild

• Kubernetes ships three separate probes — livenessProbe, readinessProbe, and startupProbe — precisely so restart and routing decisions don't share a signal.
• Envoy / Istio expose healthy_panic_threshold (default 50%) that fails open to protect against correlated health-check failure ejecting a whole cluster.
• AWS ELB/ALB target groups separate health checks from a minimum-healthy-targets posture, and many meshes document fail-open-when-none-healthy behavior for the same reason.
• Google SRE guidance is explicit: liveness/readiness are different questions, and deep dependency checks belong to alerting, not to automated per-task remediation.

See it

The LayersScene contrasts the three probes (liveness → restart, readiness → routing, deep health → paging), and the CardsScene carries the load-bearing config: /readyz local-only at periodSeconds 5 · failureThreshold 3, the 50% Envoy panic floor, and the 60-second synthetic deep-health probe paged-not-routed.

Named concepts: liveness vs. readiness probes, shallow vs. deep health checks, failure-domain isolation. </content> </invoke>