DA-01-008-P · Primer — Spotting and Handling Outliers

The idea in one sentence

An outlier is a value that sits far from the rest of your data — and your real job is not to detect it, but to decide whether it is a mistake to fix, a rare truth to protect, or a distortion to tame.

Why it exists

One wrong number can quietly wreck an entire analysis. A single order logged as 900 instead of 45 drags your average up, inflates your totals, and can flip a chart's story from "steady" to "explosive." So every analyst learns two habits: a way to find the values that don't belong, and a way to reason about what to do with them. The trap is treating "outlier" as a synonym for "delete." Sometimes the extreme value is a data-entry typo you should correct or drop; sometimes it is your single most important customer, a fraud case, or the exact rare event you were hired to study. The point of this skill is to slow down at that fork in the road and choose on purpose instead of reflexively.

How it works

The most robust everyday rule is the IQR (interquartile range) rule: take Q1 (25th percentile) and Q3 (75th percentile), measure the spread IQR = Q3 - Q1, and flag anything below Q1 - 1.5·IQR or above Q3 + 1.5·IQR. It works because it describes the middle of your data and ignores the extremes while measuring — the outlier can't corrupt the fences it's being judged against. A z-score rule ((x - mean) / std, flag |z| > 3) is the alternative, but be careful: the mean and standard deviation are themselves pulled by the very outlier you're hunting, so IQR is safer on skewed or messy data. Once flagged, you pick a deliberate action: keep it (real signal), cap it to the fence (winsorize — tame its influence without inventing a deletion), or drop it (confirmed error). The code below finds the culprit with the IQR rule and shows capping in one line.

import pandas as pd

# Daily order counts for a small shop; one value is clearly wrong (a typo)
orders = pd.Series([42, 47, 39, 44, 51, 46, 40, 900, 43, 45],
                   name="daily_orders")

# IQR rule: fence off the usual middle 50%, flag anything far outside it
q1, q3 = orders.quantile(0.25), orders.quantile(0.75)
iqr = q3 - q1
low, high = q1 - 1.5 * iqr, q3 + 1.5 * iqr
flag = (orders < low) | (orders > high)
print(f"IQR fences: [{low:.1f}, {high:.1f}]")
print("Flagged outliers:", orders[flag].tolist())   # -> [900]

# CAP instead of delete: pull extremes back to the fence (winsorize)
capped = orders.clip(lower=low, upper=high)
print("900 capped to:", capped.max())                # -> 53.5 (the upper fence)

The one gotcha

A flag is a question, not a verdict. The math tells you a value is far away; it cannot tell you why. Before you delete anything, ask: is this a typo (900 vs 45), a real-but-rare event (Black Friday), or a sign your column mixes two different things (retail and wholesale orders in one series)? Deleting a true signal is how analysts accidentally erase their most important finding. And never judge outliers on a value you're about to model — decide, document why, and prefer capping over silent deletion so the row (and your reasoning) stays auditable.

See it — keep / cap / drop decision

flowchart TD
  A["Value flagged by<br/>IQR or z-score rule"] --> B{"Is it a data<br/>error? (typo, unit,<br/>bad join)"}
  B -->|"Yes, confirmed"| C["DROP or fix<br/>the value"]
  B -->|"No, it is real"| D{"Does it distort<br/>the analysis?"}
  D -->|"No"| E["KEEP it —<br/>it is signal"]
  D -->|"Yes, it dominates"| F["CAP to the fence<br/>(winsorize)"]
  C --> G["Document why<br/>you did it"]
  E --> G
  F --> G

The one-line takeaway

Find the far-away values with the IQR rule, then keep, cap, or drop each one on purpose — and write down why.

Where you'll meet it

Cleaning a sales or sensor dataset before you report an average; investigating fraud or churn where the outlier is the story; prepping features for a model that's sensitive to extremes; and defending your numbers when a stakeholder asks "why did you remove that row?"

Named concepts: outlier detection, IQR and z-score rules, keep, cap, or drop decisions