BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 37 — The Metric That Got Gamed

Theme: Feedback Loops & Adversaries · Format: 20-min drill · Difficulty: intro

The story

The review team's dashboard metric — and bonus — is cases closed per day. The queue melts away; leadership celebrates the productivity gains for two quarters. Then fraud losses climb. A quality audit samples closed cases and finds median review time: 15 seconds, with approve-as-default the dominant pattern. The dashboard improved the entire time.

⏸ Stop here and ask the room: who behaved irrationally here? (Careful.)

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

The "(Careful.)" in the stop question is the whole drill — let the room blame the reviewers first, then walk it back. Don't rescue them early.

"Was the review team wrong to optimize what they were paid to optimize?"

Listen for: a clean "no," with the mechanics — the org specified its values in the bonus formula, and the reviewers implemented the spec faithfully. The design failure belongs to whoever attached money to a quantity with no quality counterweight.

Weak signal: moralizing — "they should have known better," "bad apples." If the fix is hiring more virtuous reviewers, the same incentive bug ships again next quarter with new staff.

"What's the counterweight metric?"

Listen for: independently sampled re-review of closed cases — a decision-reversal rate measured by a team outside the bonus pool — combined multiplicatively with throughput: cases closed × sampled accuracy. "Independent" and "sampled" are the load-bearing words.

Weak signal: self-reported quality, or quality measured by the same team the bonus pays. A counterweight the optimizer controls is just another lever to pull.

"How would you have caught this in month one instead of quarter three?"

Listen for: process telemetry that already existed — median handle time collapsing toward 15 seconds, approve-rate drift, per-reviewer decision entropy dropping — plus a standing quality-assurance sample live from day one of the bonus. The audit gets designed before the incentive launches.

Weak signal: "watch fraud losses more closely." Losses lag by months; by the time they climb, the damage is booked. A lagging indicator can't be the early-warning system.

The reveal

Nobody was irrational. The reviewers optimized exactly what the org measured and paid for — literally, efficiently, rationally. The metric was the bug.

Why this happens — the mechanics

Goodhart's law — when a measure becomes a target, it ceases to be a good measure — is usually quoted as an aphorism, but the mechanics are concrete. "Cases closed per day" was a proxy for the real goal: careful risk decisions at volume. Proxy and goal were correlated only under the old behavior, when nobody was paid to move the proxy. Attaching a bonus applies optimization pressure precisely at the weakest point of that correlation — the reviewers searched the space of available actions and found the cheapest one that moves the number, approve in fifteen seconds, which happens to be the action that destroys the goal.

Note that the loop needed no dishonesty. Every closed case was really closed; the dashboard was accurate; nothing was falsified. The proxy improved because the goal decayed — quality is the input you can liquidate fastest to buy quantity. That is the signature of a gamed metric, and it's why "the dashboard improved the entire time" isn't a paradox but the expected trace.

The two-quarter delay came from asymmetric feedback speeds. Throughput reports daily; fraud losses mature over months and land in a different team's ledger. The org was reading the fast, favorable signal with no independent slow signal — no sampled quality assurance (QA) — positioned to contradict it. Any metric attached to money should be presumed gamed from day one, because gaming is the rational response; the only open question is whether the counterweight is already measuring when it starts.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant LD as Leadership
    participant RV as Reviewers
    participant DB as Dashboard
    participant FL as Fraud losses
    LD->>RV: Bonus tied to cases<br/>closed per day
    RV->>DB: Throughput climbs -<br/>queue melts away
    Note over RV: Median review time drops<br/>to 15 seconds (silent)
    Note over RV: Approve-as-default becomes<br/>the pattern (silent)
    DB->>LD: Celebrated for two quarters
    Note over FL: Bad approvals accumulate<br/>unseen (silent)
    FL->>LD: Quality audit lands -<br/>losses climbing for months
    Note over LD,FL: The dashboard improved<br/>the entire time

Solution — where the guardrails go:

flowchart TD
    A["Quantity metric: cases<br/>closed per day"] --> P["Paired with a quality metric"]
    Q["Independent sampled QA re-<br/>reviews closed cases"] --> P
    P --> B["Bonus tied to throughput<br/>times sampled accuracy"]
    B --> C["15-second reviews tank the<br/>quality score"]
    C -.-> AL["Quality alarm fires in<br/>month one"]
    AL --> R["Incentive rebalanced<br/>before losses climb"]

What good looks like

The lesson

Every quantity metric needs an independently measured quality counterweight (throughput × sampled-accuracy, never throughput alone), and any metric attached to money should be assumed gamed from day one — design the audit before the bonus. Goodhart's law isn't cynicism; it's mechanics.

Named concepts: Goodhart's law, paired metrics, sampled QA of human processes, incentive design as system design.

Real-world echoes

Transfer

List every number in our org that pays a bonus or drives a review. For each: what's the counterweight, and who measures it independently?

Make it a ticket: Build the inventory — every metric wired to money or reviews, its quality counterweight, and the independent team measuring it. Every empty counterweight cell becomes an incentive-redesign ticket.

Related