Drill 83 — The Consumer That Fell Behind

Theme: Queues & Backpressure · Format: 20-min drill · Difficulty: core

The story

A payments platform runs a Kafka consumer group that reads every card transaction off a transactions topic, enriches each one with a synchronous call to a merchant-risk service, and writes the result to the store that feeds the fraud engine and the operations dashboards. Producers push a steady eight thousand transactions a second all day. The consumer group — three instances across twelve partitions — normally sits within a second of the tip of the log. The lag chart has been flat for a year, so nobody looks at it.

At 11 p.m. the merchant-risk service quietly degrades: a database behind it loses a read replica, and per-call latency drifts from fifteen milliseconds to four hundred. Nothing errors. The enrichment calls still return 200 OK — just slowly. Each consumer instance now spends most of its poll loop waiting on that call, so it commits offsets far slower than transactions arrive. A garbage-collection pause on one instance an hour later makes it worse. The gap between the tip of the log and the committed offset — the lag — starts to widen, a few thousand records a minute.

There is no alarm on lag. The dashboards are green because the records they do have look perfectly healthy; they're just old. The fraud engine keeps scoring, but it's scoring transactions from twenty minutes ago, then forty, then an hour. Every service is up. Every health check passes. The consumers are processing as fast as they can — they're simply processing the wrong hour.

By the time the first analyst arrives at 7 a.m. and notices the fraud queue is reviewing yesterday's dinner-time transactions, the group is six hours behind. Six hours of transactions have been authorized against stale risk scores, and the morning's real-time dashboards are showing a version of the business that ended before midnight.

⏸ Stop here and ask the room: every service was up, every health check green, and the consumers never stopped working. So how did the whole platform end up running six hours in the past?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

The consumers never crashed and never errored. Where is the failure hiding?

Listen for: the failure is in throughput vs. arrival rate, not availability. When consume rate drops below produce rate, lag accumulates monotonically — a healthy-looking service silently falls into the past. "Up" and "current" are different properties; liveness and health checks measure the first and say nothing about the second.
Weak signal: "but nothing was down" — treating a green health check as proof the pipeline is fine, with no notion of how far behind the data is.

What is the single number that would have caught this, and what does it measure?

Listen for: consumer-lag — records behind (offset gap) and, better, time behind (age of the last processed event). It is the direct SLI for "are we keeping up." Alarm on it well before it becomes hours: page when lag exceeds a threshold for a sustained window, not on a one-off blip.
Weak signal: "monitor CPU / memory / the enrichment latency" only — those are upstream causes or symptoms; none of them tells you the pipeline is now serving stale data.

Suppose lag alarms. Adding more consumer instances — does that always help?

Listen for: parallelism is capped by partition count: a Kafka partition is consumed by at most one instance in the group, so a twelve-partition topic can never use more than twelve consumers. Autoscaling on lag helps only up to that ceiling; past it you must repartition or fix the slow enrichment call. And a downstream slowdown won't be outrun by fan-out — you may need to make enrichment async or shed it.
Weak signal: "just add more consumers" with no mention of the partition ceiling or the root-cause dependency.

The reveal

The enrichment slowdown was real but recoverable — the read replica came back on its own around 2 a.m. The damage was that nothing measured whether the pipeline was keeping up. The team added a consumer-lag SLI (both records-behind and seconds-behind) with a page at sustained lag, wired the consumer group to autoscale on lag up to the partition ceiling, raised the partition count so parallelism could actually grow, and added an end-to-end freshness monitor on the fraud store so "the newest record is older than N minutes" pages even if lag reporting itself breaks. On the next enrichment slowdown, lag crossed the threshold in four minutes, consumers scaled out, and the group recovered a small backlog before anyone noticed — instead of after six hours.

Why this happens — the mechanics

A consumer group is a queue with the depth hidden on the broker side. Lag is exactly produced_offset − committed_offset: it grows whenever consume rate falls below produce rate, and — like any queue under a persistent slowdown (Little's Law) — it grows without bound until the rates re-cross. Because the records the consumer has already processed look completely normal, every downstream health check stays green. Staleness is invisible from inside a record; you can only see it by comparing the record's age to now, and nothing in the default health-check vocabulary does that.

The organizational root is an ownership seam: the platform team owns "are the services up," the data team owns "is the fraud store correct," and nobody owns "is the consumer keeping up with the log." Lag is a broker-side metric that neither dashboard naturally surfaces, so a best-practice, healthy-looking deployment can drift hours into the past with every signal green. It's a slow-motion version of the unbounded-queue failure: the queue lives in Kafka instead of in the heap, so it never OOMs — it just silently serves the past.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant P as Producers
    participant K as Kafka topic
    participant C as Consumer group
    participant D as Fraud + dashboards
    P->>K: 8k transactions/sec steady
    K->>C: poll batch
    Note over C: enrichment 15ms to 400ms<br/>plus a GC pause
    C--xK: commit offset slips<br/>behind the log tip
    Note over K: lag grows silently<br/>no alarm on lag
    C->>D: writes stale (old) results
    Note over D: dashboards green<br/>data is hours old
    Note over P,D: found at 7am by an analyst<br/>six hours behind

Solution — where the guardrails go:

flowchart TD
    A["Transactions to Kafka"] --> B["Consumer group<br/>processes batches"]
    B --> G1["Consumer-lag SLI<br/>records + seconds behind"]
    G1 -.-> AL["Page: lag over 60s<br/>for 5 minutes"]
    B --> G2["Autoscale on lag<br/>up to partition count"]
    B --> G3["Partition parallelism<br/>match key cardinality"]
    B --> G4["End-to-end freshness<br/>SLO on fraud store"]
    G4 -.-> AL
    AL --> H["On-call drains<br/>the backlog fast"]
    G2 --> H
    G3 --> H

What good looks like

Consumer-lag SLI, in both units. Export lag as records behind (produced − committed) and seconds behind (age of the last processed event). Warn at 10,000 records or 30 seconds behind; page at 60 seconds behind sustained for 5 minutes so a blip doesn't page but a real slowdown does.
Autoscale consumers on lag. Add an instance when lag stays above 10,000 records for 3 minutes; scale down when it drops under 1,000 for 10 minutes — capped at the partition count, because extra instances past that ceiling sit idle.
Partition for the parallelism you need. Size partitions to peak arrival rate divided by single-consumer throughput, with headroom (e.g. 24–36 partitions, not 12), so autoscaling has room to actually catch up; pick a partition key with enough cardinality to spread evenly.
End-to-end freshness SLO as the backstop. Independently of Kafka's own lag metric, alarm when the newest record in the fraud store is older than 2 minutes — the monitor of last resort that fires even if lag reporting itself is broken.
Guard the slow dependency. Put a timeout and circuit breaker on the enrichment call so one slow downstream can't drag the whole consume loop; consider making enrichment asynchronous so consume rate doesn't track it 1:1.
Game-day the slowdown. Inject 400 ms of latency into the enrichment call in staging, hold producer rate, and assert the lag alarm pages within 5 minutes and autoscaling recovers the backlog — prove the alarm fires before you need it.

The lesson

A service can be fully up and still be hours in the past: measure whether your consumers are keeping up, not just whether they're alive. Lag and end-to-end freshness are first-class SLIs — alarm on how far behind the data is, because "green" says nothing about "current."

Named concepts: consumer-lag SLI, autoscaling on lag, partition parallelism, end-to-end freshness SLO.

Real-world echoes

Kafka consumer-lag backlog incidents are a recurring theme in streaming post-mortems: a downstream slows, consume rate drops below produce rate, and lag climbs for hours while every service stays green — which is exactly why lag-exporter tooling (Burrow, the built-in consumer-lag metrics, Cruise Control) exists and why "seconds behind" is treated as a primary SLO for streaming pipelines.

Transfer

Which of your streaming consumers or queue workers has no alarm on how far behind it is — and if its slowest dependency degraded tonight, how many hours could it fall into the past before a human noticed?

Make it a ticket: for every consumer group, export lag in records and seconds, page at 60 seconds behind for 5 minutes, add autoscaling on lag capped at partition count, and add an end-to-end freshness alarm at 2 minutes on the store it feeds.

Drill 81 — The Unbounded Queue (the in-heap version of the same overload; here the queue lives on the broker and never OOMs).
Drill 4 — The Stale Score (downstream consumers running on data that quietly aged out).
Drill 24 — The Autoscaler That Fed the Fire (scaling on the wrong signal; here scaling on lag is the right one).
Pattern #111 — Backpressure, and Pattern #129 — Load Shedding (the reusable fixes for producers outrunning consumers). </content> </invoke>