Drill 76 — The Timeout That Wasn't Set

Theme: Networking & DNS · Format: 20-min drill · Difficulty: core

The story

A digital bank's onboarding service verifies every new applicant against a third-party KYC (Know Your Customer) vendor — identity, sanctions, and politically-exposed-person screening — as a synchronous call on the signup path. The service runs a pooled HTTP client over a fixed pool of two hundred worker threads, the same pool that serves login and the account dashboard. For two years it has been dull and reliable: the vendor's ninety-ninth-percentile latency sits around four hundred milliseconds, a signup completes in under a second, and nobody thinks about it.

One Tuesday afternoon the vendor ships a bad deploy. They do not go down. Every response is still a 200 OK — but latency creeps, then climbs: median goes from three hundred milliseconds to nine seconds, and a long tail hangs for thirty, sixty seconds before answering. No errors, no 500s, no dropped connections. Just slow.

Inside the onboarding service, every request that calls the vendor holds its worker thread while it waits — and because no read timeout was ever configured, the thread waits as long as the vendor takes. Within about two minutes all two hundred threads are parked on the vendor. New requests queue and then time out at the load balancer. The health-check endpoint needs a worker thread too, can't get one, and stops answering — so the load balancer marks the box unhealthy and pulls it. Then the next box, and the next. Login stops. The dashboard stops. A service with nothing structurally wrong is fully dark — and the dependency that caused it never returned a single error.

⏸ Stop here and ask the room: the vendor kept returning 200 OK the entire time and never went down. So how did a slow dependency take down the whole service — including login and the dashboard, which never call it?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Name the resource that actually ran out.

Listen for: thread-pool (or connection-pool) exhaustion. Each in-flight call holds one worker thread; with no read timeout the hold time is unbounded, so a fixed pool fills in seconds and everything sharing it starves — unrelated endpoints and the health check included. The bottleneck is threads, not CPU or memory.
Weak signal: "the vendor was slow so we were slow" — misses that a slow dependency should degrade one feature, not the whole service. "Just add more threads" only moves the fill a few seconds later.

Why is a slow dependency worse than a down one?

Listen for: a down dependency fails fast — connection refused, an instant error — and immediately frees the thread; the failure is loud and local. A slow-but-200 dependency holds the resource and emits no error, so nothing trips. Without a timeout there is no upper bound on hold time, so slowness becomes unbounded resource consumption. Slow is worse than down.
Weak signal: assuming failures are always fast and loud; treating latency as a lesser problem than error rate.

What's the single missing line — and why isn't it enough on its own?

Listen for: an explicit client read timeout (and connect timeout) is the missing primitive. But a timeout alone still lets every request march up to the dead-slow vendor and fail after the timeout — you also need to stop calling (circuit breaker), isolate the blast radius (bulkhead), and make the timeout a budget derived from the caller's remaining deadline.
Weak signal: "just set a timeout" and stop there; or naming one guardrail as the silver bullet.

The reveal

It was a default. The HTTP client's read timeout was infinite out of the box, and no one had ever set one — so "we never configured a timeout" quietly meant "we wait forever." The vendor's slow deploy turned that default into a total outage. The fix was not one line: they set an explicit connect timeout of two seconds and read timeout of five seconds, wrapped the vendor in a circuit breaker (opens at fifty-percent failures over a rolling twenty-request window, stays open thirty seconds), moved vendor calls onto their own bulkhead pool of twenty-five threads separate from login and the dashboard, and pushed a request-level deadline down the chain so no hop waits longer than the caller has left. At the next vendor slowdown, signups degraded for a few minutes and auto-recovered; nothing else noticed.

Why this happens — the mechanics

A synchronous call over a bounded pool couples your availability to your slowest dependency's latency. The resource that runs out is threads (or pooled connections), not CPU — every in-flight call parks one, and with no read timeout the hold time is unbounded. Little's Law makes it concrete: concurrency equals arrival rate times hold time, so if hold time jumps from four hundred milliseconds to thirty seconds at a constant arrival rate, the concurrency you'd need climbs seventy-five-fold and a fixed pool empties in seconds. Once the shared pool is empty, everything on it starves — including endpoints that never touch the vendor, and the health check itself, which is exactly why one slow dependency reads as a full outage.

Slow is worse than down because of what each does to the resource. A down dependency refuses the connection and returns instantly, freeing the thread and raising a loud, local error. A slow-but-200 dependency holds the thread and emits nothing, so no error rate rises and no alarm trips — the system fails silently, by starvation. The defaults conspire: most HTTP clients ship with no read timeout, so the one dependency you control least ends up setting the ceiling on your own availability. And the organizational seam matches the technical one — the vendor is someone else's system, the pool is shared infrastructure no single feature owns, and the timeout is a client default nobody ever reviewed.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant A as Applicant
    participant O as Onboarding<br/>pool
    participant K as KYC vendor
    participant L as LB / health
    A->>O: signup - holds a thread
    O->>K: verify identity
    Note over K: bad deploy - slow,<br/>still 200 OK
    Note over O: no read timeout -<br/>thread parked
    A->>O: more signups arrive
    Note over O: all 200 threads<br/>parked on vendor
    L--xO: health check - no thread
    Note over L: box marked<br/>unhealthy, pulled
    A--xO: login - pool empty
    Note over A,L: total outage from a<br/>slow dependency

Solution — where the guardrails go:

flowchart TD
    A["Call to KYC vendor"] --> B["Connect timeout 2s +<br/>read timeout 5s"]
    B --> C["Bulkhead: vendor gets its<br/>own 25-thread pool"]
    C --> D["Circuit breaker: opens at<br/>50% fails / 20 reqs, 30s"]
    D --> E["Timeout budget: deadline<br/>propagates down the chain"]
    B -.-> AL["Fail fast - free the thread,<br/>degrade one feature"]
    C -.-> AL
    D -.-> AL
    E --> F["Slow dependency degrades<br/>one feature, not the service"]

What good looks like

Explicit connect + read timeouts on every outbound client: connect two seconds, read five seconds — a small multiple of the vendor's four-hundred-millisecond p99, never the library default (usually infinite). A call that can't finish inside the caller's budget is abandoned, not parked.
Bulkhead the dependency: the KYC vendor gets its own bounded pool (twenty-five threads / a dedicated connection pool), separate from the pool serving login and the dashboard. Vendor slowness can exhaust at most that pool; the rest of the service keeps serving.
Circuit breaker with timeouts counted as failures: trip open at fifty-percent failures over a rolling twenty-request window, stay open thirty seconds, then half-open with a single probe. While open, fail fast or return a "verification pending" fallback instead of queuing behind a dead-slow vendor.
Timeout budget / deadline propagation: the inbound request carries a deadline (say eight seconds); each downstream hop gets min(remaining budget, its own cap), so no single call can wait longer than the caller has left. Deadlines shrink as they propagate.
Saturation and latency alarms: page when pool utilisation (active ÷ pool size) exceeds eighty percent for one minute, when the dependency's p99 exceeds three seconds for two minutes, and on every circuit-breaker state change. A saturation alarm would have fired minutes before the outage.
A latency game day: inject ten seconds of vendor latency in staging and assert the rest of the service stays green, the breaker opens, and only signups degrade.

The lesson

Every call to something you don't control needs a deadline and a blast-radius limit: a timeout so a slow dependency can't hold your resources forever, and a bulkhead plus a circuit breaker so its slowness degrades one feature instead of the whole service. A missing timeout is an unbounded resource leak that just hasn't met a slow day yet.

Named concepts: client-side timeouts, circuit breaker, bulkhead isolation, timeout budget.

Real-world echoes

Missing-timeout cascading outages are among the most common post-incident findings in service architectures: a dependency goes slow rather than down, in-flight calls with no read timeout pile up on a shared thread pool, and an unrelated service goes dark by starvation. It is the entire reason resilience libraries such as Netflix's Hystrix bundled timeouts, bulkheads, and circuit breakers into one package — to bound the blast radius of a slow dependency before it can consume the caller.

Transfer

Pick your single most-called third-party dependency. What is the read timeout on that client right now — the actual configured value, not "there's probably one"? And if that dependency went slow (not down) for ten minutes, which of your own endpoints would go down with it?

Make it a ticket: audit every outbound client for an explicit connect + read timeout, put your top external dependency behind its own bulkhead pool and a circuit breaker, add a pool-saturation alarm at eighty percent, and run a staging game day that injects ten seconds of vendor latency and proves the rest of the service stays up.

Drill 75 — The TTL That Was Too Long (sibling in this theme; the other way a networking default silently bites you).
Drill 21 — The Thundering Herd (load exhausting a shared resource; same starvation shape from the other direction).
Drill 27 — The Chatty Neighbors (noisy-neighbor resource starvation; bulkheads are how you isolate it).
Pattern #124 — Timeouts, Bulkheads & Circuit Breakers (blast-radius isolation for calls you don't control).