BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 24 — The Autoscaler That Fed the Fire

Theme: Scale & Cloud · Format: 20-min drill · Difficulty: advanced

The story

The database slows under lock contention from a batch job. App-tier latency rises; requests queue; CPU utilization on the app tier rises with the queuing. The autoscaler does its job: adds pods. Each new pod opens its full connection pool to the struggling database. More connections, more contention. The autoscaler sees things getting worse and adds more pods.

⏸ Stop here and ask the room: draw the feedback loop. Where does it break — or does it?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

What signal was the autoscaler watching, and what was the actual constraint?

Listen for: the autoscaler watched app-tier CPU — a symptom — while the constraint was database lock contention, two layers away. The trap is that queuing itself burns CPU, so the symptom correlates with load in the healthy case and inverts under this failure: more pods → more connections → more contention → more queuing → more CPU → more pods. Strong answers name the loop explicitly and say "positive feedback": the controller's action increases the error it's trying to reduce. The strongest connect it to control theory — an autoscaler is a feedback controller, and a controller keyed to the wrong signal amplifies instead of dampens.

Weak signal: "the autoscaler was too aggressive — lower the max or slow the scaling rate." Tuning the gain of a mis-wired controller changes the speed of the fire, not its direction. The wiring (signal choice), not the tuning, is the bug.

What limits the total connections a fleet can open?

Listen for: by default — nothing. Connection budget = pods × pool size, and the autoscaler controls the pod count, so the fleet's total demand on the database is unbounded by design. Strong answers know the fix layers: a fleet-wide connection budget (max pods × pool size held under the database's connection ceiling with headroom), or a connection proxy (PgBouncer, RDS Proxy) that multiplexes many app connections onto a fixed database-side pool, so scaling out adds workers without adding database load.

Weak signal: "raise max_connections on the database." That moves the cliff, and each connection has memory and contention costs — a bigger ceiling on an unbounded demand is still an unbounded demand.

What's the manual move at minute five?

Listen for: the counter-intuitive one — pause the autoscaler and scale the app tier down, kill or deprioritize the batch job, and let the database drain its lock queue. Backpressure at the front door (shed or queue excess requests) protects the database while it recovers. The key insight: every instinct and every automation in the room says "add capacity," and the correct move is the opposite — which is why it must be written in a runbook before the incident, with an explicit human override for the autoscaler.

Weak signal: "add read replicas / more pods / failover the database." Adding capacity downstream of a lock-contention constraint feeds the loop — and anyone who can't say how to stop the autoscaler mid-incident has found a second finding.

The reveal

The database collapses fully under connection load. Recovery requires scaling the app tier down — the opposite of every instinct and every automation in the room.

Why this happens — the mechanics

An autoscaler is a feedback controller: it measures a signal, compares it to a target, and acts to close the gap. Control loops are safe only when the action actually reduces the signal. Here the causality is inverted — the CPU rise is caused by queuing on a constrained database, and the action (add pods) increases demand on that same database. The loop's sign flips from negative feedback (stabilizing) to positive feedback (amplifying), and a positive-feedback loop doesn't drift toward failure, it accelerates: each cycle's output is the next cycle's input, which is why the incident ran away in minutes while the humans were still checking dashboards.

The deeper cause is that the signal and the constraint live in different layers. CPU is measured where the autoscaler acts (the app tier); the bottleneck is where the work serializes (the database's lock manager). Under the theory of constraints, adding capacity anywhere except the constraint does nothing — and adding capacity that demands more from the constraint is actively harmful. Connection count is the multiplier: each pod arrives with a full pool, so the fleet's demand on the database scales linearly with pod count, unbounded, controlled by the very automation that's misreading the situation.

Staging never shows it because the loop needs all four ingredients at once: a real contention source (the batch job's write pattern at production data volumes), enough traffic for queues to build, an autoscaler with production thresholds and limits, and a database at production scale. Staging typically lacks three of the four — and autoscaling is often disabled there to save cost. Load tests are worse than useless here if the environment omits the batch job: they "prove" the autoscaler works, because with a healthy database, adding pods genuinely helps. The failure only exists when the controller meets the constraint — which is why it has to be found by reasoning about the loop on a whiteboard, then rehearsed in a game day.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant BJ as Batch job
    participant DB as Database
    participant AP as App tier
    participant AS as Autoscaler
    BJ->>DB: Lock contention from<br/>batch writes
    DB--xAP: Queries slow down -<br/>requests queue
    loop Positive feedback - each pass adds load
        AP->>AS: CPU rises with<br/>the queuing
        AS->>AP: Adds pods
        AP->>DB: Each new pod opens a full<br/>connection pool
        Note over DB: More connections deepen<br/>contention (silent)
    end
    Note over DB: Collapses under<br/>connection load
    Note over AP,AS: Recovery requires scaling<br/>DOWN - against every instinct

Solution — where the guardrails go:

flowchart TD
    A["Batch job causes database<br/>lock contention"] -.-> B["Alarm fires on DB lock<br/>waits, the true constraint"]
    A --> C["Autoscaler keyed to<br/>constraint signal, not CPU"]
    C --> D["No new pods aimed at a<br/>dying database"]
    A --> E["Fleet-wide connection<br/>budget holds the line"]
    E --> F["Backpressure at the front<br/>door sheds excess load"]
    F --> G["Database drains its queue<br/>and recovers"]
    C --> H["Automation ceiling plus<br/>human override available"]

What good looks like

The lesson

An autoscaler is a feedback controller; pointed at a symptom instead of the constraint, it becomes a positive-feedback amplifier. Know your system's bottleneck, cap total connection budgets fleet-wide, apply backpressure at the front door (shed load, queue with limits), and give automation a ceiling plus a human override.

Named concepts: bottleneck analysis (theory of constraints), backpressure, connection pooling budgets, control-loop design.

Real-world echoes

Transfer

What is our actual bottleneck resource under 3× load — and does any of our automation make more demands on it when stressed?

Make it a ticket: For each autoscaler and auto-remediation we run, document the signal it watches, the resource it consumes when it acts, and the current ceiling — then verify the fleet-wide connection budget inequality against the database's limit and write the pause-the-autoscaler override command into the on-call runbook.

Related