AIM-01-012-P · Primer — Class Imbalance

The idea in one sentence

Class imbalance is when one outcome is far rarer than the other, so a model can post a high score by quietly ignoring the very class you built it to catch.

Why it exists

The problems worth solving are almost always lopsided. Think of a haystack where you are looking for a single needle: fraud, disease, manufacturing defects, and rare equipment failures are the needle; the ordinary, healthy, working cases are the entire haystack. When 1% of your rows are the event of interest, "get most rows right" and "find the event" become two completely different goals — and the default training objective chases the first one, because that is where the numbers are.

How it works

Consider 10,000 credit‑card transactions where exactly 100 — just 1% — are fraud. A naïve model that always predicts "not fraud" will be correct on 9,900 of them, yielding 99% accuracy, but it catches zero frauds.

Worked example: Suppose you evaluate the model on the same 10,000 rows. The confusion matrix looks like:

               Predicted
               Fraud   Not fraud
Actual Fraud      0          100
Actual Not fraud  0          9,900

Precision = 0/0 (undefined, treated as 0), Recall = 0/100 = 0%, and the loss function sees only the 9,900 correct majority predictions, so the cheapest rule it can learn is "always guess the majority." The math that is supposed to punish mistakes barely notices when every rare case is missed.

See it — "always guess majority" scores 99% yet catches zero fraud

flowchart TD
    A["10,000 rows<br/>100 fraud (1%)"] --> B{Model rule<br/>predict not fraud}
    B --> C["9,900 majority<br/>all correct"]
    B --> D["100 fraud<br/>all missed"]
    C --> E["Accuracy 99%<br/>looks great"]
    D --> F["Recall 0%<br/>caught nothing"]
    E --> G["Verdict:<br/>score lied"]
    F --> G

The one gotcha

99% accuracy on a 1%‑fraud problem means you caught nothing. A model that never fires beats a real detector on accuracy every time, so accuracy here is not merely unhelpful — it actively hides the failure. The real trap is treating accuracy as the sole metric; it masks the model's inability to detect the minority class. The fix is to stop trusting the average: measure recall (of the 100 frauds, how many did you catch?) and precision (of your alarms, how many were real?), and rebalance the training signal by oversampling the minority (e.g. SMOTE), undersampling the majority, or weighting the loss so a missed fraud costs many times more than a false alarm.

See it (visualization plan)

A BigNumberScene counting an accuracy meter up to 99 while a second figure — "frauds caught" — sits at 0, then a CompareScene/Tiles pair contrasting Accuracy 99% (green-looking but useless) against Recall 0% (red). Anchor on the worked numbers: 10,000 rows, 100 fraud, 9,900 "correct."

The one-line takeaway

On rare events, accuracy lies — measure the class you care about.

Where you'll meet it

scikit-learn's class_weight="balanced", the imbalanced-learn library that ships SMOTE and random under/oversamplers, and the pos_weight argument on PyTorch's BCEWithLogitsLoss. You will feel it the first time a fraud, churn, or medical model reports a dazzling accuracy and a recall near zero — and the fixes have their own videos: precision vs recall (AIM-01-030-X), F1 and the PR curve (AIM-01-031-P), and threshold selection (AIM-01-035-X).

Named concepts: class imbalance