Theme 3 — LLM & Agent Incidents
The reflex this theme builds: the model is a component, not a colleague — bound it, check it, budget it.
Why this theme matters
An LLM (Large Language Model) is a new kind of component with properties nothing else in the stack has: it is non-deterministic, it can be instructed by anyone who manages to write into its context — a customer, an email, a PDF — it is priced per token, and it is confidently wrong in ways that never throw an exception. A function that behaved like this would never pass code review; wrapped in an API and a friendly tone, it ships to production anyway.
None of that makes classic engineering discipline obsolete — it makes it more necessary. Idempotency, retry budgets, input validation, least privilege, cost controls: every drill in this theme is one of those old disciplines reasserting itself around a new component. The common failure is always the same — treating the model as a colleague whose judgment you trust instead of a component you bound, check, and budget. Abbreviations are expanded on first use; the full reference is glossary.md.
The drills
| # | Drill | One-line hook | Problem diagram | ~Difficulty |
|---|---|---|---|---|
| 14 | The Email That Gave Orders | A customer email instructs the agent — and it obeys, leaking another customer's data. | sequence | core |
| 15 | The $40k Weekend | A retry loop with no ceiling re-buys the same poison batch all weekend. | sequence + retry loop | intro |
| 16 | The Upgrade That Broke Nothing (Visibly) | A version bump quietly nulls 30% of extractions for three weeks. | timeline | core |
| 17 | The Policy That Never Existed | The bot invents a fee waiver — and legal says the bot speaks for the company. | sequence | intro |
| 18 | The Judge That Liked Long Answers | Scores climb, complaints climb — the judge was rewarding verbosity all quarter. | sequence + optimize loop | advanced |
| 19 | The Clause on Page 47 | Silent truncation hides the clause that sinks the deal. | sequence | core |
| 20 | The Agent That Wouldn't Stop | 9,000 overnight retries, a corrupted table, and a week-long vendor ban. | sequence + retry loop | advanced |
Best practices — the LLM & agent checklist
What to actually do, distilled from drills 14–20:
- Treat all user- and document-supplied content as data, never as instructions. Anything that can write into the model's context can command it. Authorization is enforced outside the prompt — in the architecture — never by asking the model to behave.
- Scope credentials per request, least-privilege. Before the model ever runs, the agent's data access is narrowed to what the authenticated requester is entitled to see. A prompt-injected model with no reach is a nuisance; with broad credentials it's a breach.
- Filter outputs with DLP (Data Loss Prevention). Scan everything the model emits for cross-tenant data, secrets, and PII (Personally Identifiable Information) before it leaves — defense in depth for the day the layers above fail.
- Treat cost as an SLI (Service Level Indicator), metered from your side. Count tokens and requests as you dispatch them; don't wait for the provider's billing. Near-real-time spend alarms, plus hard caps and a kill switch that anyone on-call can pull at 2 a.m.
- Give every retry a budget and every poison input a dead-letter queue (DLQ). Exponential backoff, a hard ceiling on attempts, and a parking place for inputs that repeatedly fail. A retry policy without a ceiling is a blank check.
- Version prompt + model version as one artifact, gated like code. Every change goes through a regression eval suite in CI (Continuous Integration), a canary slice of live traffic, and a rollback plan. "Looks the same on five documents" is not a gate.
- Parse model output strictly and fail loud. Unexpected format, missing field, markdown-fenced JSON: reject and alarm. A permissive parser converts provider behavior changes into silent nulls in your database.
- Ground obligation-creating statements in retrieval. Policies, prices, commitments: the model retrieves and quotes from canonical sources via RAG (Retrieval-Augmented Generation), cites or declines, and anything obligation-shaped gets human review. The bot must be structurally unable to improvise a promise.
- Calibrate LLM-as-judge evals against human panels before optimizing against them, recalibrate on a schedule, and audit disagreements for known biases (length, style, self-preference). An unvalidated judge is a Goodhart machine with an API.
- Account for input completeness on every call. Token-count every input, route over-length documents explicitly (chunk-and-merge, hierarchical summarization, or refuse and route to a human), and record coverage per document as a queryable field. Silent truncation is a wrong answer with good grammar.
- Give every autonomous loop a budget and idempotent side effects. Max iterations, max spend, max wall-clock, and a circuit breaker on repeated identical outcomes — hitting any limit halts the agent, preserves state, and pages a human. Writes inside loops must be idempotent or transactional.
- Run red-team suites in CI. Hostile inputs — injection attempts in emails and documents, cross-tenant probes, oversized inputs, malformed-but-200 responses — exercised on a schedule, not once at launch.