Drill 98 — The Judge That Drifted

Theme: Evaluation & Validation · Format: 20-min drill · Difficulty: core

The story

A retail bank ships a customer-support copilot: it drafts replies to complaint tickets — disputed fees, card fraud, hardship requests — for a human agent to approve. Nobody trusts "vibes," so quality has a number. Every month an eval oracle scores a sample of the copilot's drafts one to five against a written rubric: partly an LLM-as-judge (a frozen prompt asks a large model to grade each draft), partly a panel of human annotators who score an overlapping slice. The monthly mean lands on a leadership dashboard as Answer Quality, and it has sat around 3.8 for two quarters. Steady. Trusted.

In the background, three ordinary things happen over those months. In February the rubric gets a "clarification" — one sentence added about tone on hardship cases, because an annotator asked. In March two of the original four annotators rotate off the panel and two new hires rotate on, trained on the rubric as it reads now. In April the platform team upgrades the judge: the old model is being deprecated and the new one is cheaper and, everyone agrees, generally smarter, so the judge prompt now points at the new model. Each change ships through its own owner, each is individually reasonable, and none of them is logged as an eval change.

In May the dashboard moves. Answer Quality climbs from 3.8 to 4.1 — the biggest jump in a year. The copilot team is delighted; a quarterly review deck cites it as evidence the last prompt iteration paid off. Leadership approves more headcount for the program. Then in June the copilot team ships a prompt refactor that, unknown to anyone, actually makes hardship-case drafts a little worse — and the dashboard doesn't flinch. It stays at 4.1.

⏸ Stop here and ask the room: the copilot's Answer Quality rose 0.3 points on a stable, trusted metric — and nobody touched the copilot in a way that explains it. What did we actually measure?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

The score went up 0.3 — what are the candidate explanations, and how would you rank them?

• Listen for: the system got better is only one hypothesis, and the weakest one given nothing shipped. The strong answer separates the thing being measured (the copilot) from the instrument measuring it (the judge): the rubric changed, the annotator pool changed, and the judge model changed — any of those moves the number with the system held constant. This is eval-oracle drift: a non-stationary ruler. The way to rank them is to hold the inputs fixed and see if the score still moves.
• Weak signal: "the metric went up, so the model improved" — treating the oracle as ground truth rather than as an instrument with its own drift.

The judge is a large model that got "upgraded." Why is that a breaking change to the metric, not an improvement to it?

• Listen for: an LLM-as-judge has its own score distribution — how generous it is, how it weighs tone vs. correctness, its length and self-preference biases. A newer model is a different instrument with different units; May's 4.1 and January's 3.8 are no longer measured on the same ruler, so month-over-month comparison is meaningless. A judge upgrade is a breaking eval change that demands re-baselining, exactly like changing units on a chart mid-year.
• Weak signal: "the new model is smarter, so the scores are more accurate" — conflating a better judge with a comparable one; even a genuinely better judge invalidates the time series unless you re-score history.

Suppose you can only add ONE thing to make next quarter's number trustworthy. What is it?

• Listen for: a frozen gold set — a fixed, versioned set of drafts with locked reference judgments — re-scored by the current oracle every period. If the copilot's inputs are identical but the gold-set score moves, that delta is the drift, and you subtract it or block the comparison. Bonus points for pairing it with inter-annotator agreement so you know how much of the score is signal vs. rater.
• Weak signal: "score more samples" or "add more annotators" — more measurements on a moving ruler give you a more precise wrong answer, not a stable one.

The reveal

The copilot never improved. When the team re-scores January's frozen drafts with the June judge, January comes back at 4.1 too — the entire 0.3 gain is the new judge being more generous, compounded by two new annotators reading a rubric that had quietly gained a sentence. Worse, the inflation ran the wrong way at the worst time: June's real regression on hardship cases was masked because the more-generous judge lifted every score by roughly the same amount the regression pulled it down. The dashboard measured the judge, not the system, and the two changes happened to cancel.

The fix isn't a smarter judge. The team freezes a 300-draft gold set with locked labels and re-scores it every month; the month the judge changed, the gold set jumps 0.3 with identical inputs — drift, made visible and subtracted. They start tracking inter-annotator agreement (Krippendorff's alpha) and judge-vs-human agreement as first-class metrics, pin the judge model and rubric by version, and treat any judge upgrade as a breaking change that requires re-scoring the last two quarters before the series can continue. Answer Quality becomes comparable across months again — and June's regression, re-measured on the stable ruler, finally shows up as the drop it always was.

Why this happens — the mechanics

An eval oracle is a measurement instrument, and every property that makes a physical instrument trustworthy — a fixed unit, a known zero, a documented calibration — is implicit and unowned here. An LLM-as-judge has a score distribution baked into its weights and prompt: how strict it is, how it trades off tone against correctness, its documented biases toward longer answers and toward outputs that resemble its own. Swap the model and you swap the instrument. Human annotators are instruments too, with their own zero: rotate the panel or edit the rubric and the unit shifts under you. None of these changes touches the system under test, yet all of them move the number — and because the score is precise (4.1, to one decimal) it reads as if it were accurate and comparable, which it is not.

The organizational reason competent teams hit this is that the oracle has no single owner who treats a judge change the way a data team treats a schema change. The model deprecation is the platform team's ticket; the rubric edit is the annotation lead's; the panel rotation is a hiring event. Each is locally correct and none is filed under "we just changed how quality is measured." So the ruler drifts one reasonable edit at a time, and month-over-month comparisons — the whole reason the metric exists — silently become apples-to-oranges. The only defense is to make the instrument itself a monitored, versioned artifact: re-score a frozen reference every period so drift in the ruler is separated from change in the system, and gate any change to the judge behind a re-baseline.

Diagrams

Problem — the ruler drifts one reasonable edit at a time:

timeline
    title Answer Quality — reported vs. real
    Jan : Judge v1 scores copilot 3.8 : baseline set, trusted
    Feb : Rubric "clarified" mid-quarter : (silent eval change)
    Mar : 2 of 4 annotators rotate off : (silent unit shift)
    Apr : Judge model upgraded v1 to v2 : (silent, no re-baseline)
    May : Reported score jumps to 4.1 : leadership celebrates
    Jun : Prompt refactor regresses quality : (masked by generous judge)
    Later : Re-score Jan drafts with v2 : Jan is also 4.1 - real change is zero

Solution — make the instrument a versioned, monitored artifact:

flowchart TD
    A["New period's copilot drafts"] --> J["Eval oracle<br/>(pinned judge + rubric)"]
    J --> S["Reported Answer Quality"]
    G1["Frozen gold set<br/>(300 drafts, locked labels)"] --> J
    J --> D["Gold-set score vs.<br/>last period"]
    D -.->|"delta > 0.05"| AL["Judge-drift alarm<br/>(block the comparison)"]
    G2["Judge-vs-human agreement<br/>+ annotator kappa"] -.->|"kappa < 0.6"| AL
    G3["Judge upgrade proposed"] -.->|"breaking change:<br/>re-score last 2 quarters"| AL
    AL --> H["Re-baseline before<br/>the series continues"]

What good looks like

• A frozen gold set, re-scored every period: pin 300 representative drafts with locked reference judgments; re-score them with the current oracle each month. Inputs are identical, so any movement is pure instrument drift — alarm and investigate when the gold-set mean shifts more than 0.05 on the 1–5 scale (or 2 points on a 100-point scale) between periods. This is the single control that separates "system changed" from "judge changed."
• Inter-annotator agreement as a gate, not a footnote: compute Krippendorff's alpha (or Cohen's kappa) across the human panel every period; require alpha ≥ 0.6 (substantial agreement) before a score is publishable. Below that, the rubric is ambiguous or the panel drifted — fix the ruler before you trust the reading.
• Judge-vs-human calibration, tracked over time: on the overlapping slice, measure agreement between the LLM-judge and the human panel (correlation and mean-absolute-error in score points); require agreement to stay within a band (e.g., MAE ≤ 0.3 points). A drop means the judge and the humans have diverged — recalibrate the judge prompt against fresh human labels.
• Pin and version the judge like a production dependency: the metric records the exact judge model ID, prompt hash, and rubric version. A judge upgrade is a breaking eval change — before switching, re-score the last 2 quarters with the new judge, publish both series through an overlap window, and only then cut over. Never silently repoint the judge at a new model.
• Anchor the scale with a rubric plus few-shot examples: give the judge explicit anchor answers ("a 3 looks like this, a 5 looks like this") and 3–5 few-shot calibration examples in the prompt, so the score is a function of quality, not of the judge's mood. Anchoring measurably cuts run-to-run variance and inter-judge disagreement.
• A drift dashboard next to the quality dashboard: plot gold-set score, annotator alpha, and judge-human MAE over time. If any drifts, the Answer Quality number is not comparable across that boundary — mark the discontinuity on the chart so no deck ever compares across a ruler change.

The lesson

An eval oracle is an instrument, not an authority: if the judge, the rubric, or the raters change, the units change, and last month's score and this month's are no longer on the same ruler. Freeze a gold set and re-score it every period so drift in the ruler is separated from change in the system — and treat any judge upgrade as a breaking change that must be re-baselined before the number means anything.

Named concepts: eval-oracle drift, frozen gold set, inter-annotator agreement, LLM-as-judge versioning, rubric anchoring.

Real-world echoes

• AlpacaEval's length-controlled win rate (2024). The AlpacaEval leaderboard scores model outputs with an LLM-as-judge, and maintainers documented that the judge systematically favored longer answers — so raw win-rates reflected the judge's bias as much as answer quality, and models could climb by padding. The fix was to introduce a length-controlled win rate that anchors the comparison, making scores comparable across models again. It is the same lesson at leaderboard scale: when the judge has a drift or a bias, the number measures the judge until you control for it.

Transfer

For the most important quality number on any dashboard your team reads, ask: who or what is the judge, and has it changed — model version, rubric wording, or the people scoring — since the last two data points you just compared? If you can't answer, you don't yet know whether your trend is the system or the ruler.

Make it a ticket: stand up a frozen gold set of ~300 items with locked labels for your highest-stakes eval, re-score it every period, and alarm when the gold-set score moves more than 0.05 on a 1–5 scale (2 points on a 100-scale) with identical inputs — and add the judge model ID and rubric version to the metric's metadata this sprint.

Related

• Drill 93 — The Metric That Lied — the sibling failure: there the offline metric didn't transfer to production; here the oracle scoring the metric silently changes units underneath you.
• Drill 18 — Judge That Liked Long Answers — the same LLM-as-judge, viewed through its bias rather than its drift; both are reasons the judge's score is not the system's quality.
• Drill 4 — The Stale Score — another instrument that quietly stopped meaning what everyone assumed, measured against a world that had moved.
• Pattern #132 — SLO / Error-Budget Policy — the discipline of deciding on a metric whose definition is pinned and owned, not drifting edit by edit.