Drill 16 — The Upgrade That Broke Nothing (Visibly)

Theme: LLM & Agent Incidents · Format: 20-min drill · Difficulty: core

The story

The model provider announces a deprecation; the team bumps the version string, runs five documents by hand — "looks the same, ship it." The new model version occasionally wraps JSON in markdown fences and renames one field under long inputs. The downstream parser was written permissively: unparseable fields default to null. Extraction coverage quietly drops 30%. Dashboards track pipeline throughput, which is unchanged.

⏸ Stop here and ask the room: three weeks pass. What monitoring and what process were both missing?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

What should gate a model-version change?

Listen for: The same gates as code — a regression eval suite (golden documents with expected extractions, format assertions, per-field accuracy thresholds) running in CI (Continuous Integration); a canary slice of live traffic compared against the incumbent on semantic metrics; an explicit rollback plan. "Prompt + model version" travels as one versioned artifact.

Weak signal: "Test it more before shipping" without saying what a test asserts or what blocks the merge; or "the provider said it's backward compatible" — the provider guarantees the interface, not the behavior.

Why did permissive parsing make this worse?

Listen for: The parser converted a loud, catchable format change into silent nulls — it absorbed the failure at the last point where it could have alarmed. Postel's law ("be liberal in what you accept") backfires when nothing downstream measures what was accepted: tolerance without measurement is concealment.

Weak signal: "Permissive parsing is fine, the alerting was the problem" — missing that a strict parser is the alert; or blaming the provider for changing behavior, which providers will always do.

What's the field-level SLI that catches it in an hour?

Listen for: Field fill rate — for each extracted field, the share of documents where it arrives non-null — plus a format-validity rate (parsed clean on first attempt), tracked per field and per document type, alarmed against a trailing baseline.

Weak signal: "Monitor the error rate" — there were no errors; or a single aggregate accuracy number that averages the one broken field against nine healthy ones — Theme 1's aggregation trap, replayed.

The reveal

A reviewer finally asks why half her queue's summaries are missing amounts. The eval suite that would have caught it in CI didn't exist; the fill-rate alarm that would have caught it in production didn't either.

Why this happens — the mechanics

A model version is not an API contract. What the provider guarantees is the interface — request and response shapes; what actually changed is the behavior surface: formatting habits, markdown-fencing tendencies, field-naming choices, behavior under long inputs. Those are properties of learned weights, not code, and they shift between checkpoints in ways no changelog fully describes. A version-string bump is a behavior change of unknown extent shipped as a one-line diff.

Five hand-checked documents is a tiny, biased sample. The regression only manifests on long inputs — exactly the stratum a quick eyeball skips, because long documents are tedious to check by hand. Nobody would ship a code refactor validated on five conveniently chosen inputs; the friendly phrase "looks the same" let this one through.

Then the permissive parser absorbed the signal. Unparseable fields became nulls, and null is a legal value everywhere downstream — so the failure propagated as data, not as an error. The dashboards measured motion (documents processed) rather than meaning (fields filled), and motion was unchanged. Three weeks is what MTTD (Mean Time To Detect) looks like for a semantic failure with no semantic monitor: detection waited for a human whose queue felt wrong — the monitor of last resort.

Diagrams

Problem — how the failure unfolds:

timeline
    title Three silent weeks after a version bump
    section Week 0
        Ship day : Provider deprecation lands : Team bumps the version string : Five docs eyeballed - ship it
    section Weeks 1-2
        Silent drift : Model wraps JSON in fences : Fields renamed on long inputs : Parser nulls what it cannot read
        Green dashboards : Coverage drops 30% (silent) : Throughput metric unchanged
    section Week 3
        Discovery : Reviewer flags missing amounts : No eval gate, no fill-rate alarm : Found by a human, 3 weeks late

Solution — where the guardrails go:

flowchart TD
    A["Model version change proposed"] --> B["Regression eval suite runs<br/>as a CI gate"]
    B -->|"fail"| C["Change blocked before<br/>production"]
    B -->|"pass"| D["Canary slice: new version<br/>on a small share of live<br/>traffic"]
    D --> E["Strict parser: unexpected<br/>format fails loud, not<br/>null"]
    E --> F["Full rollout only after a<br/>clean canary"]
    D -.-> G["Field-fill-rate SLI<br/>(Service Level Indicator)<br/>per extracted field"]
    G -.-> H["Fill-rate alarm catches<br/>the drop within an hour"]

What good looks like

Pin exact model versions or snapshots; treat provider deprecations as planned migrations with a scheduled window, never a same-day version bump.
A regression eval suite in CI: a few hundred documents stratified by length, type, and language, asserting parse validity, field presence, and per-field accuracy — with thresholds that block the merge.
Canary the new version on ~5% of live traffic and compare field-fill and format-validity SLIs against the incumbent before full rollout, with one-click rollback.
Parse strictly against a schema (JSON Schema, Pydantic, or equivalent): fenced output, renamed fields, missing keys — reject, count, alarm. Never default to null.
A field-fill-rate SLI per extracted field, alarmed on drops of a few percentage points against a 7-day baseline.
Store the raw model output next to the parsed record, so diagnosing drift is a query rather than a re-run.
A standing human-review quota — say twenty documents a week — as the backstop for whatever the evals didn't think to encode.

The lesson

Prompt + model version is a versioned artifact: changes go through a regression eval suite, a canary slice, and a rollback plan — same as code. And parse strictly: a loud failure at the parser beats a silent null in the database (Drill 7's lesson, one layer up).

Named concepts: regression evals as CI gates, canary releases for models, strict validation, semantic SLIs (field-fill rates).

Real-world echoes

None cited — model-upgrade regressions of this shape are endemic but rarely make the news; they surface in postmortems, not headlines.

Transfer

If our model provider changed behavior tonight, which of our pipelines would tell us — and which would just get quieter?

Make it a ticket: Pick our highest-value extraction pipeline and add a field-fill-rate SLI with a baseline alarm; then list every pipeline whose model version can change without a CI gate, and schedule an eval suite for the top one.

Drill 19 — The Clause on Page 47 — the other silent-coverage failure: there the input was cut, here the output was nulled.
Drill 7 — permissive parsing and silent nulls, one layer down; this drill is that lesson replayed at the model boundary.
Drill 4 — The Stale Score (Theme 1) — another one-line change shipped without canary or post-change verification.