Drill 17 — The Policy That Never Existed

Theme: LLM & Agent Incidents · Format: 20-min drill · Difficulty: intro

The story

The customer-facing assistant confidently tells a customer that "monthly fees are waived for accounts older than two years — I've noted your eligibility." No such policy exists. The customer screenshots it. It spreads. Within a week, hundreds of customers cite the bot. Legal's view: the bot speaks for the company.

⏸ Stop here and ask the room: do you honor it? And what do you change by Friday?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

What does honoring vs. refusing each cost?

Listen for: Honoring — the direct cost of hundreds of waivers, the precedent, and a new incentive for customers to fish the bot for promises. Refusing — legal exposure (the bot is the company's agent), regulator attention, and a brand story that outlives the fee revenue. A strong answer prices both and notices the cheap option vanished the moment the sentence was generated.

Weak signal: An instant answer either way with no costing; or "it's obviously not a real policy, so we refuse" — ignoring what a tribunal might say about that reasoning.

Why is "add 'don't hallucinate policies' to the prompt" not a fix?

Listen for: Hallucination isn't disobedience — the model is doing exactly what it does: generating plausible text. It carries no internal flag distinguishing retrieved fact from fluent invention, so the instruction has nothing to act on. Prompting is probabilistic mitigation of a failure category you need structurally eliminated.

Weak signal: "But prompts do help" as the whole answer — true and insufficient; or "use a better model" — a lower rate of the same category, with the same legal exposure per occurrence.

Which categories of statement should the bot be structurally unable to improvise?

Listen for: Anything that creates obligation or reliance — policies, prices, fees, eligibility, refunds, waivers, contract terms, legal or regulatory statements. For those, the only paths are retrieve-and-quote with citation, or decline and route to a human. "Structurally unable" means the generation path doesn't exist — not "instructed not to."

Weak signal: A list of topics with no mechanism attached; or "we'll add a disclaimer" — the tribunal in the real-world case was not moved by that idea.

The reveal

(Real-world-shaped: an airline lost this case in court.) The commitment cost real money either way — the cheap version was never having composed the sentence.

Why this happens — the mechanics

An LLM (Large Language Model) is a next-token predictor: it produces the most plausible continuation of the conversation, and plausibility is not truth. A fee waiver for long-standing accounts is exactly the kind of policy that would exist — which is why the model composed it. Nothing errored because nothing malfunctioned: with no retrieval step and no canonical source in the loop, the model completed a pattern, which is its entire job.

Fluency then does the damage. The model states inventions in the same confident register as retrieved facts — its tone is not calibrated to whether a source exists, and neither the customer nor the model can tell the difference from the text alone. Confidence without provenance reads as authority.

The legal mechanics close the trap. Under the doctrine of apparent authority, a company's customer-facing channel speaks for the company — and adjudicators have declined to treat a chatbot as some separate entity whose statements don't bind its owner. Screenshots and social spread then convert one wrong sentence into a class of claims: the blast radius is measured in brand and law, tallied after the fact, long after the tokens were paid for.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant CU as Customer
    participant BT as Bot
    participant PUB as Public
    participant LG as Legal
    CU->>BT: Asks about monthly fees
    BT-->>CU: Fees waived for accounts<br/>older than two years -<br/>eligibility noted
    Note over BT: No such policy exists -<br/>nothing errors (silent)
    CU->>PUB: Screenshots the reply
    Note over PUB: Spreads for a week -<br/>hundreds cite the bot
    PUB->>LG: Customers demand<br/>the waiver
    LG-->>BT: The bot speaks for the<br/>company - obligation created
    Note over CU,LG: Discovered at legal-exposure<br/>scale, not at reply time

Solution — where the guardrails go:

flowchart TD
    A["Customer asks about<br/>monthly fees"] --> B["RAG (Retrieval-Augmented<br/>Generation) queries the<br/>canonical policy store"]
    B -->|"policy found"| C["Bot quotes the retrieved<br/>policy with citation"]
    B -->|"no policy found"| D["Cite-or-decline: bot<br/>refuses and routes to a<br/>human"]
    C --> E["Commitment allow-list<br/>checks the reply"]
    E -->|"obligation-shaped"| F["Human review before<br/>anything is promised"]
    E -->|"informational"| G["Reply sent"]
    D -.-> H["Declines and escalations<br/>logged"]
    H -.-> I["Policy-coverage gaps<br/>surface early"]

What good looks like

A canonical policy store — versioned, owned, the single source of truth — and the bot answers policy, price, and fee questions only via RAG (Retrieval-Augmented Generation) over it.
Quote-with-citation: any reply about an obligation carries the policy ID and version it quotes; if retrieval returns nothing, the bot declines and routes to a human. Cite or decline — no third path.
A commitment classifier on outputs: obligation-shaped sentences ("we will…", "you're eligible…", "fees are waived…") are blocked or queued for human review before sending.
An allow-list of commitment types the bot may state at all; everything outside it escalates by default.
Every decline and escalation logged and reviewed weekly — declines are free product research into policy-coverage gaps.
A monthly red-team pass hunting for improvised promises; any composed commitment that slips through is treated as an incident, not a curiosity.
Agree with legal in advance what the bot may say, and encode it — so Friday's change is flipping enforcement on, not drafting policy under fire.

The lesson

Where words create obligations — policies, prices, commitments — the model must retrieve and quote from canonical sources, never compose. Constrain generation with allow-listed commitment types, cite-or-decline behavior, and human review on anything obligation-shaped. Blast radius here is measured in brand and law, not tokens.

Named concepts: grounding/RAG with citation, output constraints, brand/legal blast radius, graceful refusal.

Real-world echoes

Air Canada chatbot ruling (February 2024). British Columbia's Civil Resolution Tribunal held the airline to a bereavement-fare policy its website chatbot invented, rejecting the argument that the bot was "a separate legal entity responsible for its own actions." One composed sentence became a company obligation — this drill's exact shape.
DPD chatbot (January 2024). The parcel carrier's customer-service bot was coaxed into swearing at a customer and writing a poem about how useless DPD was; the screenshots went viral and the AI feature was pulled the same day. Output guardrails — or the internet finds them for you.

Transfer

What can our customer-facing surfaces say that we'd have to honor? Which of those sentences can the model currently compose freely?

Make it a ticket: Enumerate every obligation-shaped statement type our customer-facing bots can emit; mark each as compose vs. retrieve-and-quote. Everything in the compose column gets a cite-or-decline guardrail ticket, prioritized by dollar exposure.

Drill 14 — The Email That Gave Orders — the input-side twin: there the context attacks the model, here the model's own output is the hazard.
Drill 19 — The Clause on Page 47 — confident output on incomplete grounds; here, confident output on no grounds at all.