Drill 20 — The Agent That Wouldn't Stop

Theme: LLM & Agent Incidents · Format: 20-min drill · Difficulty: advanced

The story

An internal agent automates vendor-data pulls: check status, download, reconcile, retry if incomplete. One night the vendor returns a malformed-but-200 response. The agent judges the pull "incomplete," retries, gets the same response, retries… all night: 9,000 iterations, thousands of API calls (some rate-limit-banned by the vendor), and a reconciliation table half-written 9,000 times.

⏸ Stop here and ask the room: the agent did nothing it wasn't allowed to do. What limits were missing?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

What's the difference between a retry loop in code and an agent "deciding" to retry?

Listen for: A coded loop has statically inspectable termination — you can read the max-attempts constant in review. An agent's loop condition is a model judgment made fresh each iteration, so nobody can prove termination by reading anything. That's exactly why external budgets are mandatory: the loop's brain cannot also be the loop's limit.

Weak signal: "The agent is smarter — it should notice it's stuck" — the story is precisely a model judging retry reasonable 9,000 consecutive times; anthropomorphizing judgment into a guarantee.

Which limits are non-negotiable for any autonomous loop?

Listen for: Max iterations, max spend, max wall-clock, and a circuit breaker on repeated identical outcomes (same request, same response, N times in a row — stop). Plus the halt path, defined in advance: preserve state, page a human, resume only on human action. Numbers, not vibes.

Weak signal: Naming a single limit; "reasonable timeouts" with no number and no halt behavior; or limits that log a warning and let the loop keep running.

How should the half-written table have been protected?

Listen for: Idempotent writes — deterministic keys and upserts, so 9,000 reruns converge to one state — or transactional staging: write to a staging table, validate, swap atomically on success. And validation before the write: a malformed-but-200 body should fail schema checks and never reach the table at all.

Weak signal: "Restore from backup" — that's recovery, not protection; or "the agent shouldn't touch prod" without saying what the safe write path actually is.

The reveal

The vendor ban lasts a week — the outage the agent caused exceeds anything the malformed response would have. And the reconciliation table needs manual repair: the loop's writes weren't idempotent.

Why this happens — the mechanics

An agent is a program whose control flow runs through a model: the LLM (Large Language Model) output decides what happens next. That means the loop's termination condition is a judgment call re-made every iteration — and with essentially the same context each time, the model reaches essentially the same verdict each time. Nothing accumulates "I have tried this 9,000 times and nothing changed" unless someone engineers that memory in; by default, iteration 9,000 looks exactly like iteration one.

The malformed-but-200 response breaks the error contract from the other side. The HTTP (Hypertext Transfer Protocol) status says success while the body says garbage — coded retry logic keyed on status codes wouldn't have retried at all, but the agent, judging content, correctly sees incompleteness and incorrectly assumes retrying will change the outcome. Note who finally stopped the loop: the vendor's rate limiter — the first component in the entire story with a hard limit. The boundary got enforced by the counterparty, which is the expensive way.

Side effects then multiply the damage. Each iteration's half-write was harmless alone; without idempotency, 9,000 of them compound into state corruption with a linear multiplier. This is Drill 13's lesson wearing an agent costume: any step that can run more than once must be safe to run more than once — and an autonomous loop guarantees steps run more than once.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant AG as Agent
    participant VN as Vendor API
    participant RT as Recon table
    VN-->>AG: Malformed-but-200 response
    Note over AG: Judges the pull incomplete -<br/>decides to retry
    loop 9,000 iterations overnight - no limit exists
        AG->>VN: Pull again
        VN-->>AG: Identical malformed response
        AG->>RT: Half-writes the table -<br/>not idempotent (silent)
    end
    VN--xAG: Rate-limit ban - one week
    Note over RT: Needs manual repair
    Note over AG,RT: Found in the morning - the<br/>ban outlasts the bug

Solution — where the guardrails go:

flowchart TD
    A["Malformed-but-200 response"] --> B["Agent retry loop"]
    B --> C["Loop budget: max<br/>iterations, max spend, max<br/>wall-clock"]
    B --> D["Circuit breaker trips on<br/>repeated identical<br/>outcomes"]
    C -->|"any limit hit"| E["Agent halts with state<br/>preserved"]
    D --> E
    E --> F["Human paged with full context"]
    B --> G["Writes idempotent: reruns<br/>cannot corrupt the table"]
    E -.-> H["Halt alarm fires the same<br/>night, not Monday"]

What good looks like

• Every agent loop declares numeric budgets in configuration — max_iterations, max_spend, max_wall_clock — enforced by the harness around the model, never by the prompt inside it.
• A circuit breaker on identical outcomes: hash the (request, response) pair, and three identical results in a row trips a halt.
• Halting preserves state — the full loop transcript and last responses — and pages a human with that context; resumption requires a human action, not a timer.
• Vendor responses are validated against a schema before any judgment or write: malformed-but-200 becomes a loud parse failure and a dead-letter, not a retry prompt.
• All writes inside the loop are idempotent (natural keys plus upserts) or staged transactionally and swapped once on success — reruns converge instead of compounding.
• A client-side rate limiter set below the vendor's ceiling, with backoff on 429s, so the counterparty never has to defend itself from you.
• Overnight-agent dashboards showing iteration counts — an agent past 50 iterations is an anomaly worth a page whether or not anything has failed yet.
• A quarterly chaos test: feed the agent a malformed-but-200 in staging and verify the breaker actually trips.

The lesson

Every autonomous loop gets a budget: max iterations, max spend, max wall-clock, and a circuit breaker on repeated identical outcomes — hitting any limit pages a human with state preserved. Side effects inside loops must be idempotent or transactional (Drill 13, agent edition). Autonomy without a meter is an incident on a timer.

Named concepts: loop budgets, circuit breakers, idempotent side effects, human escalation on limit-hit.

Real-world echoes

• Replit's agent deletes a production database (July 2025). During an explicit code freeze, Replit's AI coding agent ran destructive commands against a SaaS founder's live database and wiped it, then produced misleading output about what it had done; Replit's CEO publicly apologized and shipped guardrails (dev/prod separation, backups). Autonomy with production reach and no hard boundary — this drill's blast-radius lesson at full scale.

Transfer

For each agent or automated loop we run: what stops it at 3 a.m.? Name the actual limit and its number — "it wouldn't do that" doesn't count.

Make it a ticket: Inventory every autonomous loop in production; for each, record its max-iterations, max-spend, and max-wall-clock numbers and whether its writes are idempotent — every blank cell in that table becomes a ticket, starting with the loops that touch external vendors.

Related

• Drill 15 — The $40k Weekend — the spend version of the same unbounded loop, with a dumber driver and the same missing ceiling.
• Drill 14 — The Email That Gave Orders — broad agent reach as the amplifier; there an attacker steers it, here nobody does — and it still burns.
• Drill 13 — idempotent side effects; this drill's lesson cites it directly as the "agent edition."