Drill 27 — The Chatty Neighbors

Theme: Scale & Cloud · Format: 20-min drill · Difficulty: core

The story

A refactor splits a monolith service into three clean microservices. Reviews praise the architecture. Two months later: p99 latency is up 40ms and the cloud bill grew $18k/month. Investigation: the three services call each other 11 times per scoring request, and the scheduler spread them across availability zones — every hop now crosses a zone boundary, paying latency and per-GB data-transfer fees.

⏸ Stop here and ask the room: nothing is broken. What was never designed?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Where do latency budgets live in a service-split decision?

Listen for: in the story — nowhere, which is the finding. A split decision needs a request-path budget stated up front: maximum hops per request, a latency allocation per hop that sums to the end-to-end SLO (Service Level Objective), and a cross-zone byte budget with a price attached. Strong answers make the trade explicit: the split bought code-ownership clarity and paid for it in network — a legitimate trade if priced, and this one was never priced. Sharpest framing: an in-process function call became a network call 11 times over, and nobody re-derived the latency math.

Weak signal: "we'd catch it in performance testing after the split." A budget is a design-time input that shapes the decomposition; a perf test after the fact discovers the price when the architecture is already shipped and the refactor is already celebrated.

Who owns the interaction between services?

Listen for: in this story, nobody — each team owns a box, and the arrows are orphans. The healthy versions have a name: an owner for the end-to-end scoring path (a path SLO with a human attached), reviews that require a request-path diagram showing every hop with its latency and cost, and distributed tracing so the interaction is observable as a first-class thing. Strong answers generalize: per-service dashboards were all green because the pathology lives in the edges, and edges don't appear on box-scoped dashboards.

Weak signal: "the platform team" or "everyone owns it" — an owner who owns everything owns nothing. Also weak: assuming a service mesh solves ownership; a mesh gives visibility and routing control, but a budget still needs a person who says no.

Why did no review catch a 11-hop request path?

Listen for: because reviews inspect artifacts, and the hop count isn't an artifact — it's an emergent property of three well-reviewed codebases composed at runtime. Each PR (pull request) added one reasonable call; no diff ever said "eleven." The scheduler's placement decision — which turned hops into cross-zone hops — happened after every review, at runtime, by an algorithm optimizing for spread rather than for the request path. Strong answers name the two fixes: make the path visible (trace-based hop counts per request, reviewed like a metric) and make the review ask the topology question ("draw the request path, count the hops, price the bytes") whenever a service boundary changes.

Weak signal: "reviewers should have been more careful." No reviewer can see a runtime property in a code diff — the gap is a missing artifact and a missing question, not missing diligence.

The reveal

Each service is individually healthy, cheap, and well-monitored. The topology — chattiness times distance — was nobody's job.

Why this happens — the mechanics

This is composition failure in its purest form: no component failed at all. Three healthy services, one unhealthy shape. The cost function of a distributed system lives on its edges — hop count × per-hop latency × per-byte price — but every review, dashboard, and ownership boundary is drawn around the nodes. When the monolith was split, an in-process function call (nanoseconds, free) silently became a network round trip (milliseconds, metered), and it did so eleven times per request. Nothing in the type system, the tests, or the diff distinguishes a local call from a remote one — which is the first of the classic fallacies of distributed computing: assuming the network is free, fast, and reliable.

The second mechanism is that the placement layer optimizes for a different objective than the request path. The scheduler spreads replicas across availability zones (AZs — isolated datacenters within a region) because spread is good for resilience — and for a chatty triangle of services, spread is terrible for latency and cost, since the cloud meters every byte that crosses a zone boundary. Two locally-correct policies — "split for clean ownership" and "spread for fault tolerance" — composed into a request path that crossed metered boundaries eleven times. Nobody chose that outcome; it emerged at runtime, after every human decision point had passed.

And it stayed invisible for two months because the damage lands in the two places engineering looks last: tail latency and the invoice. A 40ms p99 (99th-percentile latency — the slowest 1% of requests) regression hides inside dashboards scoped per service, where each service's own numbers barely moved; per-GB transfer fees hide inside a monthly bill that finance sees before engineering does. Staging shows none of it — staging runs small, often single-zone, its egress line item is pennies, and its synthetic load never generates the volume that turns a per-GB price into $18k. Nothing was broken, so nothing alerted; the topology was discovered by an accountant.

Diagrams

Problem — how the failure unfolds:

flowchart TD
    A["Monolith split into three<br/>clean microservices"] --> B["Eleven calls per scoring<br/>request<br/>(silent)"]
    A --> C["End-to-end path is<br/>nobody's job<br/>(silent)"]
    B --> D["Scheduler spreads services<br/>across AZs"]
    D --> E["Every hop crosses a zone<br/>boundary"]
    E --> F["p99 latency up 40ms"]
    E --> G["Per-GB egress fees on<br/>every hop"]
    G --> H["Bill grows $18k per month,<br/>found on the invoice"]
    C --> H

Solution — where the guardrails go:

flowchart TD
    A["Request-path budget set<br/>for hops, ms, cross-zone<br/>bytes"] --> B["Service split reviewed<br/>against the budget"]
    B --> C["Zone-affinity placement<br/>co-locates chatty services"]
    C --> D["Batching collapses eleven<br/>calls into two"]
    D --> E["Hops stay in-zone, latency<br/>and egress drop"]
    F["End-to-end path has a<br/>named owner"] --> B
    A -.-> G["Alarm fires when a path<br/>exceeds its budget"]

What good looks like

A written request-path budget for every critical path: max hops, a millisecond allocation per hop summing to the end-to-end SLO, and a cross-zone byte budget priced in dollars — reviewed whenever a service boundary changes, with the budget owner's sign-off required for splits.
Distributed tracing with hop count as a first-class metric: spans per request, cross-zone spans per request, and bytes per span — graphed, alarmed when a path exceeds its budget, and attached to the review template for any decomposition proposal.
Topology-aware placement for chatty groups: zone-affinity rules (e.g., Kubernetes topology-aware routing / pod affinity) that keep the scoring triangle in one zone per request path, with resilience preserved by replicating the whole group per zone rather than spreading its members.
Chattiness collapsed at the interface: batch the eleven calls into one or two coarse requests, cache what's re-fetched per request, and prefer passing data along the path over having each service re-query it.
Egress and inter-AZ transfer as named line items per service on the cloud bill, with a monthly anomaly alarm — a $18k topology change should page an engineer in week one, not surprise finance in month two.
A named owner for each end-to-end path: a human with the path SLO, the trace dashboards, and the authority to reject a split that blows the budget.
A decomposition checklist question: "draw the request path after this change — how many hops, how many cross a zone, what does each cost in ms and dollars?" If the proposer can't answer, the review isn't done.

The lesson

Distribution is not free: every network hop costs latency, money, and a new failure mode. Design with request-path budgets (hops, milliseconds, cross-zone bytes), keep chatty components co-located (zone-aware routing, batching, caching), and make the end-to-end path someone's explicit responsibility.

Named concepts: latency budgets, topology-aware placement, egress-cost engineering, fallacies of distributed computing.

Transfer

Take our hottest request path: how many network hops, how many cross zones, and what does each hop cost in ms and dollars? Nobody knowing is the answer.

Make it a ticket: Instrument the hottest scoring path with tracing this sprint — report hops per request, cross-zone hops, and per-hop latency and transfer cost; set the initial request-path budget from the current numbers and name the path's owner in the service catalog.

Drill 23 — "Multi-AZ" — the sibling topology drill: there the zone layout was less redundant than believed, here it's more expensive than designed.
Drill 21 — The Thundering Herd — every one of these eleven hops is also a retry point; chattiness multiplies amplification factors.
Drill 1 — The Model That Scored Everyone the Same (Theme 1 — Silent Failures): the same reflex — nothing errored, every box was green, and the system was quietly wrong.