Interactive Lab · Model Evaluation

Accuracy can be 95% and still useless.

A classifier isn't one number. It scores every case, and you pick where to cut. Slide that cutoff and precision, recall, and accuracy all move at once. Sweep it across every value and you trace the ROC curve, the whole truth about the model, before you've committed to any threshold. The area under it, AUC, is one honest score that accuracy can't fake. Drag the line and watch both worlds move together.

🎬 Watch the 90-second explainer, then slide the threshold and separation below ↓

▶ The playground, drag the threshold, tune the model

Two score distributions. Red = sick patients (positive) · blue = healthy (negative). Everything right of the line is predicted positive. Drag the line.

The ROC curve: every threshold is one point (FPR→, TPR↑). The purple area is AUC. The diagonal is a coin-flip. The dot is your current threshold.

Threshold 0.50

0.00 · predict everything positivepredict everything negative · 1.00

Class balance, % positive 50%

1% · heavy imbalance50% · balanced

Separability, model quality 1.6

0 · random3.4 · near-perfect

Predicted +Predicted −

Actually + / −

True Pos

sick, caught ✓

False Neg

sick, missed ✗

False Pos

healthy, alarmed ✗

True Neg

healthy, cleared ✓

TPR / recall

FPR

Precision

Accuracy

AUC

predicted + 0 false alarms 0 missed 0 AUC = 0.50

positive (sick) negative (healthy) threshold-FREE quality = AUC

🧭 Where this fits

ROC/AUC is the standard interview and industry answer to 'accuracy is 95%, is the model good?' on imbalanced data: fraud (0.1% positives), medical screening, ad conversion. The accuracy illusion you just reproduced is the reason model cards report AUC/PR curves, and threshold selection (see the Threshold lab) is a separate business decision.

The code that's actually running

🔮 Predict, then test

Set class balance to ~5% positive and push the threshold up near 1.0 so the model predicts almost everything negative. Before you look: what happens to accuracy, and to the ROC dot? Now read them. Accuracy stays sky-high (~95%) while the dot collapses toward the origin (TPR ≈ 0, it catches nobody). Why is accuracy the wrong metric here, and what is AUC telling you instead? Then drag separability from 0 to 3.4 and watch AUC climb from 0.5 (coin flip) toward 1.0 (perfect), without touching the threshold.

The takeaway

Your new mental model of ROC & AUC

A classifier is not one number. Lowering the threshold catches more positives (TPR↑) but rings more false alarms (FPR↑). Sweeping the threshold high→low traces every point of the ROC curve, the tradeoff, made visible.
AUC summarizes quality across ALL thresholds at once. It equals the chance a random positive outscores a random negative: 0.5 = coin flip, 1.0 = perfect. It doesn't move when you slide the threshold, only a better model (more separability) lifts it.
Accuracy lies under class imbalance. With 5% positives, "predict everything negative" scores 95% accuracy yet AUC ≈ 0.5 and catches nobody. Accuracy rewards the majority class; ROC/AUC don't.
You don't pick AUC as your operating point, you pick a threshold on the ROC curve based on the real cost of a false positive vs. a false negative. AUC picks the model; the ROC picks the cutoff.

📖 Read the lecture notes Try: Precision vs Recall Continue the journey →