Drill 18 — The Judge That Liked Long Answers

Theme: LLM & Agent Incidents · Format: 20-min drill · Difficulty: advanced

The story

A team spends a quarter improving a summarization system, iterating against an LLM-as-judge eval. Scores climb week over week — from 6.8 to 8.9. Reviewer complaints also climb, which is confusing, but the eval says things are great.

⏸ Stop here and ask the room: both signals are real. Reconcile them.

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

What might the judge actually be rewarding?

Listen for: Named, specific judge biases — length and verbosity, elaborate structure, hedging, its own model family's style (self-preference), position effects in pairwise comparisons — anything correlated with the score that isn't quality. Bonus points for proposing the test on the spot: regress score against output length; rescore identical content padded versus trimmed.

Weak signal: "Maybe the judge is just wrong sometimes" — random noise doesn't produce a steady quarter-long climb; only a systematic bias under optimization pressure does.

How would you test a judge before trusting it?

Listen for: Calibrate it against a blind human panel — score the same outputs both ways, measure agreement quantitatively, and probe the known bias axes deliberately (same answer at two lengths, same content in two styles, swapped positions). Then recalibrate on a schedule and after any judge-model change, because the judge is itself a prompt + model version.

Weak signal: "Spot-check some of its scores" — by reading them yourself, with the same biases the judge has; or "use a bigger model as the judge" — capability doesn't remove bias, only calibration finds it.

Which is ground truth — the judge or the reviewers?

Listen for: Neither automatically — but when a proxy metric and a direct human signal disagree, the proxy is the suspect. The judge exists for exactly one purpose: to predict human judgment cheaply. Reviewer complaints are the judge failing its one job, and a blind human comparison is the experiment that settles it.

Weak signal: "The eval is objective, complaints are anecdotal" — Goodhart's law in a single sentence; or splitting the difference diplomatically without proposing the experiment that decides.

The reveal

The judge systematically prefers longer, more elaborate outputs (and its own model family's style). The team spent a quarter optimizing verbosity. Blind human comparison prefers the original system.

Why this happens — the mechanics

LLM-as-judge means using one LLM (Large Language Model) to score another's outputs — cheap, fast, and scalable, which is why teams reach for it. But judges inherit systematic preferences from their training: verbosity bias (longer answers score higher at equal content), self-preference (outputs in the judge's own family style score higher), and format and position effects. These are measured, reproducible findings in the evaluation literature — not folklore.

Goodhart's law does the rest: when a measure becomes a target, it ceases to be a good measure. Under optimization pressure, the team's weekly iterations sample precisely the direction the judge over-rewards — each "improvement" climbs the bias gradient rather than the quality gradient. The score climb from 6.8 to 8.9 was real; it just measured fit-to-judge, and fit-to-judge diverged from quality the moment the team started optimizing.

The organizational trap seals it. The eval is quantitative, dashboarded, and improving; the complaints are scattered, anecdotal, and inconvenient. Teams weight the legible signal — so the contradicting one gets explained away for a quarter. A quarter is exactly the MTTD (Mean Time To Detect) you get when no disagreement audit exists between the proxy and the humans it's supposed to predict.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant TM as Team
    participant JG as LLM judge
    participant RV as Reviewers
    Note over JG: Prefers longer, more<br/>elaborate outputs (silent)
    loop Weekly, for a quarter
        TM->>JG: New system variant
        JG-->>TM: Score climbs - 6.8<br/>toward 8.9
        Note over TM: Outputs grow more verbose,<br/>not better (silent)
    end
    RV--xTM: Complaints rise all quarter
    Note over TM: Eval says great -<br/>complaints explained away
    TM->>RV: Blind human comparison,<br/>at last
    RV-->>TM: Original system preferred -<br/>a quarter spent on verbosity

Solution — where the guardrails go:

flowchart TD
    A["Candidate LLM judge"] --> B["Calibrate against a blind<br/>human panel before any use"]
    B -->|"disagrees with humans"| C["Judge repaired or<br/>rejected: check length,<br/>style, self-preference<br/>bias"]
    B -->|"agrees with humans"| D["Judge approved for<br/>optimization"]
    D --> E["Scheduled recalibration<br/>against humans"]
    D -.-> F["Disagreement audit samples<br/>judge vs. reviewer<br/>verdicts"]
    F -.-> G["Judge drift caught in<br/>weeks, not a quarter"]

What good looks like

Before any optimization: a blind human panel scores a representative sample, and the judge must clear a stated agreement bar with them (pairwise agreement or Cohen's kappa against a written threshold) — recorded and versioned.
Bias probes in the judge's own test suite: the same answer padded and trimmed, restyled, and reordered — score deltas must be near zero before the judge is approved.
Length control on every report: judge scores are published alongside output length, with an alarm when score-length correlation appears.
The judge's prompt + model version is a versioned artifact, recalibrated quarterly and after any change — Drill 16's rule, applied to the eval itself.
A continuous disagreement audit: a fixed sample of outputs per week goes to human review, and judge-human agreement is tracked as an SLI (Service Level Indicator) with an alarm on decline.
Reviewer complaints routed into the eval backlog as data — a rising complaint rate against a rising eval score triggers an investigation, not a shrug.
A decision rule written down in advance: when the proxy and the human signal diverge, optimization freezes until they're reconciled.

The lesson

An unvalidated judge is a Goodhart machine with an API. Calibrate LLM judges against human panels before optimizing against them, re-calibrate on a schedule, and check known biases (length, style, self-preference). When a proxy metric and a human signal disagree, the proxy is the suspect.

Named concepts: eval validity, judge calibration, Goodhart's law, proxy-metric drift.

Real-world echoes

None cited — judge-bias failures live in research papers and internal postmortems rather than headlines; verbosity and self-preference bias are well documented in the evaluation literature, which is exactly why they're checkable before a quarter is spent.

Transfer

Every automated quality score we optimize against — when was it last checked against a human? Diary the next check now.

Make it a ticket: List every automated quality metric any team optimizes against; for each, record the date of its last human calibration and book the next one — any metric with no recorded calibration gets a blind-panel ticket this month.

Drill 16 — The Upgrade That Broke Nothing (Visibly) — evals as CI gates; this drill asks whether the eval itself deserves the trust.
Drill 5 — The Twice-a-Year Bug (Theme 1) — the same cultural failure: a contradicting signal explained away instead of investigated.