ML-01-009-P · Primer — Evaluating Classifiers: Accuracy, Confusion Matrix, Precision and Recall

The idea in one sentence

A classifier's grade is not one number — accuracy tells you how often it's right, while the confusion matrix, precision, and recall tell you what kind of mistakes it makes and whether those mistakes are the ones you can live with.

Why it exists

Imagine a spam filter that simply labels everything as "not spam." If only 5 out of 100 emails are actually spam, this lazy filter is 95% accurate — and completely useless. Accuracy alone hid a disaster, because it treats every mistake as equal when in the real world they never are. Letting one spam email through is annoying; sending a job offer to the spam folder is a catastrophe. To catch this, we need to break "wrong" into its two flavors: false positives (crying wolf) and false negatives (missing the wolf). The confusion matrix lays out all four outcomes in a tidy 2×2 grid, and precision and recall distill that grid into the two numbers that actually match how you'll feel about each error. These tools exist so you can pick the metric that reflects the cost of being wrong, not just the count of being wrong.

How it works

accuracy_score counts the fraction of predictions that match the truth — simple, but blind to error type. The confusion_matrix fixes that by tabulating, for a binary problem, the four cells: true negatives (TN), false positives (FP), false negatives (FN), and true positives (TP) (sklearn lays them out as [[TN, FP], [FN, TP]]). From those cells, precision = TP / (TP + FP) answers "of everything I called spam, how much really was?" — it punishes false alarms. Recall = TP / (TP + FN) answers "of all the real spam, how much did I catch?" — it punishes misses. The two trade off: crank up how eagerly you flag spam and recall rises but precision falls, and vice versa. In the code below we fit a LogisticRegression on a tiny "spamminess score → is-it-spam" dataset, then read all four metrics — watch how one accuracy number splits into a richer story.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (accuracy_score, confusion_matrix,
                             precision_score, recall_score)

# Feature: a single "spamminess score" per email. Label: 1 = spam, 0 = not spam.
X = np.array([[1],[2],[2],[3],[3],[4],[4],[5],[5],[6],
              [6],[7],[8],[9],[10],[3],[4],[7],[2],[8]])
y = np.array([0,0,0,0,1,0,1,1,0,1,1,1,1,1,1,0,0,1,0,1])

clf  = LogisticRegression().fit(X, y)
pred = clf.predict(X)

print("accuracy :", accuracy_score(y, pred))          # 0.85
print("confusion:\n", confusion_matrix(y, pred))       # [[8 1]
                                                        #  [2 9]]  -> TN=8 FP=1 FN=2 TP=9
print("precision:", precision_score(y, pred))          # 0.9   (of flagged emails, 90% were spam)
print("recall   :", recall_score(y, pred))             # 0.818 (caught 9 of 11 real spam)

The one gotcha

On imbalanced data, accuracy is a liar. If 95% of your emails are "not spam," a model that blindly says "not spam" every time scores 95% accuracy while having a recall of zero — it caught none of the spam it was built to catch. Always look at the confusion matrix and precision/recall (or their harmonic mean, the F1 score) alongside accuracy, especially when one class is rare or one type of error is far more expensive than the other.

See it — from raw predictions to the two metrics that matter

flowchart TD
  A["Model predicts labels<br/>(spam vs not-spam)"] --> B["Compare to truth<br/>fill confusion matrix"]
  B --> C["TP: caught spam"]
  B --> D["FP: false alarm<br/>(good email flagged)"]
  B --> E["FN: missed spam<br/>(spam let through)"]
  B --> F["TN: correctly cleared"]
  C --> G["Precision = TP / (TP+FP)<br/>trust of a flag"]
  D --> G
  C --> H["Recall = TP / (TP+FN)<br/>coverage of real spam"]
  E --> H
  G --> I["Pick metric by<br/>cost of each error"]
  H --> I

The one-line takeaway

Accuracy says how often you're right; the confusion matrix, precision, and recall say what kind of wrong you are — and that's what decides if a model is safe to ship.

Where you'll meet it

Every classification project you touch — spam filters, fraud detection, medical screening, churn prediction, content moderation — is graded on precision and recall long before anyone trusts its accuracy. In the deep AIM track you'll go further: tuning the decision threshold to slide along the precision–recall curve, reading ROC-AUC, and choosing F1 or cost-weighted metrics to match real business stakes.

Named concepts: accuracy_score, confusion_matrix, precision and recall