Drill 15 — The $40k Weekend

Theme: LLM & Agent Incidents · Format: 20-min drill · Difficulty: intro

The story

Friday afternoon: a new document-reprocessing job ships. Saturday 2 a.m.: a malformed batch makes the job fail — and retry. No backoff, no max attempts, and each retry reprocesses the entire batch through the LLM API. Billing data lags a day; no one set a spend alert anyway. Monday 9 a.m.: someone opens the cloud console.

⏸ Stop here and ask the room: how much, and what three missing guardrails does this story name?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Where should cost visibility come from if the provider's billing lags?

Listen for: From your own side of the wire — count tokens and requests at dispatch time, emit them as metrics like any other SLI (Service Level Indicator), and alarm on spend-rate anomalies in near-real time. The provider's invoice is reconciliation, not monitoring.

Weak signal: "Set up billing alerts in the cloud console" — which the story already tells you lag a day; or treating cost as finance's month-end problem rather than an operational signal that pages someone.

What's the difference between a retry policy and a retry budget?

Listen for: A policy says how to retry — backoff, jitter, which errors qualify. A budget says how much in total — max attempts, max spend, max wall-clock — and what happens at the ceiling: park the input in a dead-letter queue, page a human. A policy without a budget is an unbounded loop wearing a best practice as a costume.

Weak signal: "We use exponential backoff" offered as a complete answer — backoff spreads the cost over time, it doesn't bound it. The $40k arrives slightly later.

What's the kill switch — and who can pull it at 2 a.m. Sunday?

Listen for: A named mechanism — feature flag, queue pause, revocable API key, scale-to-zero — that stops spend in one action, documented in the runbook, executable by whoever is on-call without waking an owner or hunting for credentials.

Weak signal: "We'd redeploy with the job disabled" — a 40-minute pipeline is not a kill switch; or naming a person ("Dave could stop it") instead of a mechanism anyone on rotation can operate.

The reveal

$40k of identical failed work. The job never succeeded once after the first failure — forty thousand dollars of retrying the same poison batch.

Why this happens — the mechanics

LLM (Large Language Model) providers meter usage per token and aggregate it asynchronously — spend typically lands in consoles and exports hours to a day later. The natural feedback loop, someone seeing a bill, operates on a timescale of days, while the failure operates on a timescale of seconds. Unless you meter from your own side, there is no signal in between.

The amplification comes from retry granularity. The job retries at batch level, so one poison document re-buys the entire batch through a per-token-priced API on every attempt. Do the multiplication — attempts per hour × tokens per batch × price per token × 55 hours — and $40k stops being surprising. And unlike a database that would fall over under this load, the API provides no natural backpressure: it happily accepts infinite paid work, all weekend, without a single error you didn't cause yourself.

The last ingredient is the calendar. A poison input fails deterministically, so retry-until-success is a loop with no exit except a human — and the humans are gone from Friday afternoon to Monday morning. MTTD (Mean Time To Detect) for a cost incident is bounded by office hours unless a machine is watching the meter.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant JB as Job
    participant API as LLM API
    participant BIL as Billing
    participant OC as On-call
    Note over JB: Friday afternoon -<br/>reprocessing job ships
    JB--xAPI: Saturday 2 a.m. -<br/>poison batch fails
    loop No backoff, no max attempts
        JB->>API: Reprocess the entire batch
        API--xJB: Same failure, full<br/>price (silent)
    end
    Note over BIL: Spend data lags a day - no<br/>alert configured (silent)
    Note over OC: Weekend passes -<br/>nobody paged
    OC->>BIL: Monday 9 a.m. - opens<br/>the cloud console
    BIL-->>OC: $40k of identical<br/>failed work

Solution — where the guardrails go:

flowchart TD
    A["Poison batch fails the job"] --> B["Retry with exponential backoff"]
    B --> C["Retry budget: hard ceiling<br/>on attempts"]
    C -->|"budget exhausted"| D["Batch parked in dead-<br/>letter queue (DLQ)"]
    D --> E["Pipeline continues on<br/>healthy work"]
    B --> F["Self-side token meter<br/>counts tokens and requests<br/>as dispatched"]
    F --> G["Hard budget cap halts the<br/>workload automatically"]
    F -.-> H["Near-real-time spend alarm<br/>pages on-call"]
    H -.-> I["Runaway detected Saturday<br/>2:15 a.m., cost bounded"]

What good looks like

• A self-side token and request meter — a counter in the API client wrapper, labeled per job — exported to the metrics stack like any other production signal.
• A spend-rate alarm ("more than $50/hour for 15 minutes pages on-call") plus a daily hard cap that pauses the queue automatically when hit.
• Exponential backoff with jitter and a max-attempts ceiling — five, not "until it works" — per unit of work.
• Retry at document level, not batch level, so one poison input never re-buys the whole batch.
• A dead-letter queue (DLQ): inputs that exhaust their retry budget get parked with full failure context, the pipeline moves on, and DLQ depth has its own alarm.
• A kill switch on-call can pull in under two minutes — queue pause or revocable key — documented in the runbook and rehearsed quarterly.
• New jobs launch under a deliberately low budget cap that gets raised on evidence, instead of defaulting to unlimited.

The lesson

Cost is a production SLI: meter it from your side (tokens dispatched, requests counted) with near-real-time alarms and hard budget caps that stop the workload. Retries need backoff, budgets, and a dead-letter queue for poison inputs — a retry policy without a ceiling is a blank check.

Named concepts: cost SLOs, retry budgets, exponential backoff, dead-letter queues, kill switches.

Real-world echoes

• Replit's agent deletes a production database (July 2025). An AI coding agent with production access wiped a SaaS founder's live database during an explicit code freeze; Replit's CEO publicly apologized. Different damage, same lesson as this drill: the blast radius of an automated system is whatever it can do between "runaway starts" and "a human can actually stop it."

Transfer

What's the maximum our LLM/API spend could reach between "runaway starts" and "first human notices"? Compute it — the number is usually alarming.

Make it a ticket: For each LLM-calling workload, compute that worst-case number and attach it to the service's runbook; any workload where the answer exceeds a day's normal spend gets a budget cap and a kill switch ticket this sprint.

Related

• Drill 20 — The Agent That Wouldn't Stop — the agent-shaped sibling: the same unbounded loop, spending side effects instead of dollars.
• Drill 2 — Green Dashboards, Zero Traffic (Theme 1) — the same weekend-shaped gap: nothing pages when the signal you actually need was never wired up.