DA-01-009-P · Primer — Missing Data: Finding the Patterns

The idea in one sentence

Every real dataset has holes in it, and the analyst's job is not just to count the holes but to ask why they are there — because a value that is missing at random and a value that is missing for a reason demand completely different handling.

Why it exists

No dataset arrives complete. A survey respondent skips a question, a sensor drops offline for an hour, a signup form makes "age" optional, a join fails to find a match — and each of those leaves a blank cell that pandas records as NaN (Not a Number) or None. Missing data exists because the world that generates your data is messy and partial, and pretending otherwise is how good-looking analyses go quietly wrong: mean() silently skips the gaps, a chart drops rows without telling you, a model trained on the survivors learns a distorted world. So this topic exists to give you a discipline — locate the missingness, then interrogate it — before you reach for any fix. The blanks are not just noise to be swept away; they are evidence. Where a value is missing, and which other columns predict that it's missing, often tells you as much about your data as the values that are present.

How it works

The workhorse is df.isna(), which returns a same-shaped DataFrame of True/False — True wherever a value is missing. Because True counts as 1, df.isna().sum() gives you a null count per column: your first map of where the gaps are. But a count only answers "how much?" — the analytical question is "is this missing at random, or is there a pattern?" To test that, you compare the missingness itself against another column: pd.crosstab(df["signup"], df["age"].isna()) asks whether age goes missing more for one signup source than another. If the blanks pile up entirely in one group, you've found structured missingness — a reason, not luck — and that changes everything about how you should handle it. A bar chart of the null counts turns the whole picture into something you can read in one glance. Watch all three steps on tiny data:

import pandas as pd
import matplotlib.pyplot as plt

nan = float("nan")
df = pd.DataFrame({
    "user":   ["u1", "u2", "u3", "u4", "u5", "u6"],
    "age":    [34,   nan,  28,   nan,  41,   nan ],
    "income": [52,   61,   nan,  47,   nan,  58  ],
    "signup": ["web","app","web","app","web","app"],
})

# STEP 1 -- WHERE is data missing? Count nulls per column.
print(df.isna().sum())
# user      0
# age       3   <- worst
# income    2
# signup    0

# STEP 2 -- WHY is age missing? Is it random, or tied to signup source?
print(pd.crosstab(df["signup"], df["age"].isna()))
# age    False  True
# signup
# app        0     3   <- EVERY app row is missing age -> a PATTERN, not luck
# web        3     0

# STEP 3 -- see the shape of the gaps at a glance
df.isna().sum().plot.bar(title="Missing rows per column", ylabel="count")
plt.tight_layout()
# chart: 'age' is the tallest bar (3), 'income' next (2), the rest at zero
plt.show()

The crosstab is the punchline: age isn't missing at random — the app signup form never asked for it. That single fact rules out "just fill with the average" and points you toward flagging instead.

The one gotcha

Deleting or filling missing values before you understand the pattern is how you inject bias without ever seeing an error. The tempting reflex is df.dropna() (delete every row with a blank) or df.fillna(df.mean()) (paper over blanks with the average) — and both run instantly with no warning. But look at what they'd do above: dropna() would delete every app user, because every app row is missing age — so your "cleaned" data now describes web users only, and any conclusion you draw is secretly about the wrong population. And fillna(mean) would invent an age for people the form never asked, smuggling made-up numbers into your statistics as if they were real. The honest move depends on the why: drop only when the missing rows are few and truly random (losing them doesn't skew who's left); fill (impute) when a value is plausibly recoverable and missing at random, and even then prefer a group-aware fill over a blunt global mean; and flag — add a boolean age_missing column and keep the row — when the missingness is itself informative, so the pattern becomes a feature instead of a lie. The rule to tattoo on your wrist: count the nulls, cross-tabulate to find the pattern, and only then choose drop, fill, or flag.

See it — choosing drop vs fill vs flag from the missingness pattern

flowchart TD
    A["df.isna().sum()<br/>(count the gaps)"] --> B{"Is the column<br/>mostly missing?"}
    B -->|"Yes, e.g. 80%+ blank"| C["Consider dropping<br/>the whole column"]
    B -->|"No"| D["Cross-tabulate<br/>missingness vs<br/>other columns"]
    D --> E{"Blanks tied to<br/>one group?"}
    E -->|"No, random"| F{"Few rows<br/>affected?"}
    F -->|"Yes"| G["DROP the rows<br/>(low bias risk)"]
    F -->|"No"| H["FILL / impute<br/>(group-aware mean)"]
    E -->|"Yes, a pattern"| I["FLAG it: add<br/>is_missing column,<br/>keep the row"]

The one-line takeaway

Count nulls with df.isna().sum(), cross-tabulate to learn why they're missing, and let that reason — random or structured — decide whether you drop, fill, or flag.

Where you'll meet it

You'll meet missing data on literally every dataset you ever touch, and it's one of the fastest ways to tell a careful analyst from a careless one. You'll meet it when a mean() comes back suspiciously clean because pandas silently skipped the gaps, or when a chart has fewer points than rows and you didn't notice the drop. You'll meet the pattern question in interviews the moment someone hands you data and asks "how would you handle the nulls?" — the strong answer is never "I'd drop them" or "I'd fill with the mean," it's "first I'd count them per column, then cross-tabulate to see if they're random or structured, then pick a strategy." And you'll meet the stakes for real when a model or a report built on dropna()'d data quietly describes only the survivors — the web users, the people who answered — and everyone downstream mistakes that convenient subset for the whole truth. Learning to treat the blanks as evidence, not just as dirt to sweep away, is one of the core habits of exploratory data analysis.

Named concepts: isna and null counts, missingness patterns, drop vs fill vs flag