Drill 96 — The Wrong Number

Theme: Evaluation & Validation · Format: 20-min drill · Difficulty: core

The story

A card issuer's data-science team spends a quarter rebuilding the real-time fraud model that scores every transaction before it authorizes. At the model review, the headline slide is clean and confident: 97% accuracy on a held-out month of a million transactions. The room exhales. The old rules engine was a tangle nobody trusted; here is a model that gets the answer right ninety-seven times out of a hundred. It passes review, replaces the incumbent on the authorization path, and the team moves on to the next roadmap item.

Nobody in the room asks the one question that matters: how often is a transaction actually fraud? The answer, sitting in the same warehouse, is about 3% — thirty thousand fraudulent transactions in that million-row month. On a base rate like that, a model that simply stamps "never fraud" on every transaction is also right 97% of the time. The headline number the whole room celebrated is the number you get for doing nothing.

Three months later, the fraud-operations team raises a flag that has nothing to do with a dashboard. Chargeback losses are flat — not falling, the way everyone assumed they would after shipping a shiny new model. The model runs green, throws no errors, and reports the same 97% accuracy it reported at review. It is doing exactly what the metric said it would. The metric just never measured the thing the model exists to do.

⏸ Stop here and ask the room: the model is genuinely 97% accurate on a fair, held-out month. On a problem where fraud is 3% of traffic, why does that number tell you almost nothing about whether the model catches fraud?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

97% accuracy on a 3% fraud rate — what is that number actually measuring?

• Listen for: the accuracy paradox under class imbalance. Accuracy is a traffic-weighted average, so it is dominated by the 97% majority class (legitimate transactions); a model can score 97% by getting every easy "legit" right and every fraud wrong. The correct move is to name the trivial baseline out loud — "predict never-fraud" scores 97% too — so the model is being compared against doing nothing.
• Weak signal: "97% is a great score" — treating accuracy as self-evidently good without ever asking the base rate. If the room can't state what a null model would score, they can't read the metric.

If not accuracy, what should the review slide have shown?

• Listen for: metrics derived from the decision and its costs. Precision and recall at the live operating threshold (of the fraud we see, what fraction do we catch; of what we flag, how much is real), PR-AUC rather than ROC-AUC because the positive class is rare, and a cost-weighted expected loss that prices a missed fraud (full chargeback) against a false decline (customer friction). Missing fraud and blocking a good customer are wildly asymmetric costs; a single accuracy number flattens both to zero.
• Weak signal: "add F1" or "use AUC" as a reflex, with no mention of the operating threshold, the base rate, or dollars. A better single number is still the wrong shape of answer.

The scores drive an approve/decline threshold — what else has to be true?

• Listen for: calibration. Because a human or a rule sets a cutoff on the model's predicted probability, a score of 0.8 has to mean roughly an 80% chance of fraud, or the threshold is set on a lie. Discrimination (ranking) and calibration (probabilities being right) are different properties; you need a reliability curve and an expected-calibration-error check, not just a rank metric.
• Weak signal: "the probabilities look fine" with no reliability curve — assuming a model that ranks well is automatically calibrated at the cutoff where decisions actually happen.

The reveal

Pull the confusion matrix at the live threshold and the 97% evaporates. Of the 30,000 frauds that month, the model catches about 2,400 — a recall of 8%. It is precise (of the 3,000 transactions it flags, ~2,400 are real, 80% precision), so it looks disciplined, but 27,600 frauds sail straight through. Accuracy is 97.2%; the never-fraud baseline is 97.0%. The entire quarter of work bought two-tenths of a percentage point of accuracy — and, in dollars, cut monthly fraud losses from roughly $6.6M to $6.1M when a model actually targeting recall could have taken them to ~$2.4M. The team rebuilt the evaluation around expected loss and recall at the operating point, put the confusion matrix and the naive baseline on every review slide, and added a calibration gate. The next model shipped because it caught 70% of fraud, not because it was 97% accurate.

Why this happens — the mechanics

Accuracy is (TP + TN) / total — a count-weighted average over the whole population. When one class is 97% of the traffic, that class's true negatives dominate the sum, and the model's behavior on the rare, expensive class barely moves the number. This is the accuracy paradox: on an imbalanced problem, the trivial majority-class predictor is a strong accuracy baseline, so any accuracy figure has to be read relative to the base rate, never in absolute terms. 97% is meaningless until you know that 97% is also the score for doing nothing.

The costs make it worse. Accuracy treats a missed fraud and a wrongly declined customer as identically weighted single errors, when a missed fraud is a full chargeback plus fees and a false decline is a few dollars of friction. The metric that matches the decision is expected loss, computed from the confusion matrix at the actual threshold with real per-cell costs — and because the threshold is chosen on predicted probabilities, those probabilities have to be calibrated for the chosen cutoff to mean what the reviewer thinks it means. Competent teams fall for the wrong number because accuracy is the default a hundred tutorials reach for, it is a single comfortable figure, and on a balanced toy problem it is even reasonable. The discipline is to derive the metric from the decision and its costs, and to always print the naive baseline next to it.

Diagrams

Problem — how the wrong number ships and hides:

sequenceDiagram
    autonumber
    participant DS as Data Science
    participant RB as Review Board
    participant M as Fraud Model
    participant F as Fraud Ops
    DS->>RB: Report 97% accuracy
    Note over RB: Never asks base rate<br/>(fraud is 3%)
    RB->>M: Approve, ship to auth path
    M--xF: Misses 92% of fraud<br/>(8% recall)
    Note over M: No alarm — accuracy<br/>still reads 97%
    F->>RB: Chargeback losses flat,<br/>not falling
    Note over DS,F: Found in loss report,<br/>not in the eval

Solution — evaluate against the decision, not the average:

flowchart TD
    A["New fraud model"] --> G1["Decision metric:<br/>recall at threshold +<br/>expected loss (not accuracy)"]
    A --> G2["Confusion matrix at<br/>the live threshold"]
    A --> G3["Calibration check<br/>(ECE &lt; 0.03)"]
    A --> G4["Naive-baseline compare<br/>(never-fraud = 97%)"]
    G1 -.-> GATE["Ship gate: must beat<br/>the naive baseline on<br/>the decision metric"]
    G2 -.-> GATE
    G3 -.-> GATE
    G4 -.-> GATE
    GATE --> P["Promote only if it<br/>catches real fraud"]

What good looks like

• Metric derived from the decision, not the default: report recall at the live operating threshold (target ≥ 70% of fraud caught) and precision at that same point, plus PR-AUC (appropriate for a rare positive class) — never accuracy as the headline on an imbalanced problem.
• A cost-weighted expected-loss number: price each confusion-matrix cell (a missed fraud ≈ full chargeback + fees; a false decline ≈ a few dollars of friction) and report expected monthly loss at the threshold. The model is chosen to minimize dollars, not to maximize a count-based average.
• The confusion matrix at the live threshold on every review slide: raw TP / FP / FN / TN at the exact cutoff the model will run at — so "catches 2,400 of 30,000 frauds" can never hide behind a single percentage.
• A calibration gate when probabilities drive a cutoff: a reliability curve plus expected calibration error (ECE) < 0.03; if the score that trips the decision isn't calibrated, recalibrate (isotonic/Platt) before the threshold is trusted.
• A naive baseline printed next to every metric, every time: the majority-class predictor's accuracy (97%) and the base-rate PR-AUC (0.03). A model ships only if it beats the trivial baseline on the decision metric — not on accuracy, where the baseline already wins.
• A promotion rule in writing: "beats never-fraud on recall-at-threshold and expected loss," reviewed and signed off, so no future model is approved on a number that a constant would also earn.

The lesson

A metric means nothing until you know what a trivial model scores on it — and on an imbalanced problem with asymmetric costs, accuracy is a number a constant can beat you at. Choose the metric from the decision and its costs, show the confusion matrix at the threshold you'll actually run, and always print the naive baseline beside the headline.

Named concepts: class imbalance, accuracy paradox, precision/recall at the operating point, PR-AUC, cost-weighted evaluation, probability calibration.

Real-world echoes

• The Kaggle credit-card fraud dataset (ULB). The canonical public fraud dataset is 284,807 transactions with just 492 frauds — a 0.17% positive rate, so a model that predicts "never fraud" scores 99.8% accuracy while catching zero fraud. It is the textbook demonstration of the accuracy paradox, and the reason every serious tutorial on it reports precision, recall, and PR-AUC instead of accuracy. The same trap, scaled from a demo to a production authorization path, is exactly this drill.

Transfer

For your highest-stakes classifier on a rare event — fraud, default, churn, a rare disease flag — what does a "predict the majority class" model score on your headline metric? If you don't know, you can't tell whether your model is working or whether you're celebrating the number for doing nothing.

Make it a ticket: for one imbalanced classifier, replace the headline metric with recall-at-threshold plus cost-weighted expected loss, add the confusion matrix at the live threshold and the naive-baseline score to the standard review slide, and gate promotion on beating that baseline on the decision metric.

Related

• Drill 93 — The Metric That Lied — the sibling failure: an offline metric (AUC) that was real but didn't mean what the room assumed about production.
• Drill 1 — The Model That Scored Everyone the Same — another headline number hiding a model that had effectively stopped discriminating.
• Drill 4 — The Stale Score — a different gap between a green metric and the reality it was supposed to track.
• Pattern #132 — Decide on the metric that matters — pick the evaluation from the decision and its costs, not from the default.