PD-01-011-P · Primer — Handling Missing Data

The idea in one sentence

Real data always has holes — a survey nobody filled in, a sensor that skipped a reading, a join that found no match — and pandas marks each hole with a special value called NaN, giving you three moves to deal with it: isna to find the gaps, fillna to patch them with a sensible substitute, and dropna to delete the rows or columns that have them.

Why it exists

You will almost never receive a perfect, fully-filled table. Someone leaves the "middle name" box blank, a thermometer glitches for one hour, a spreadsheet has an empty cell where a number should be. pandas needs a single, consistent way to say "there's supposed to be a value here, but we don't have one" — different from zero (which is a value), different from an empty string (also a value), different from "we haven't looked yet." That placeholder is NaN ("Not a Number"), and pandas spreads it automatically wherever data is absent.

The reason this matters is that missing values are contagious. Try to average a column with one hole in it, or sort it, or feed it to a model, and the gap either poisons the result or crashes the operation. So pandas gives you a tiny, three-verb toolkit built specifically around NaN: one verb to see where the holes are (isna), one to repair them by substituting a value you choose (fillna), and one to remove the incomplete rows entirely (dropna). Every real cleaning job is some combination of "detect, then repair or remove," and this is the vocabulary for it.

How it works

Start by detecting: df.isna() returns a same-shaped grid of True/False, one flag per cell, True exactly where the value is missing — and because True counts as 1, df.isna().sum() gives you a per-column tally of holes. Once you can see the damage, you choose per column: repair or remove. To repair, fillna swaps every NaN for a value you supply — and passing a dictionary lets you use a different fix per column, which is the usual case (a numeric column wants a number like its mean; a text column wants a label like "unknown"). To remove, dropna deletes every row that still contains any NaN, keeping only complete rows. Both fillna and dropna return a new DataFrame and leave your original untouched, so you can compare before and after. Here is the whole detect → repair → remove cycle on four rows:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name":  ["Ana", "Ben", "Cleo", "Dara"],
    "score": [90.0, np.nan, 75.0, np.nan],
    "city":  ["NYC", "LA", np.nan, "LA"],
})

# 1) DETECT — count the gaps in each column
print(df.isna().sum())
# name     0
# score    2
# city     1
# dtype: int64

# 2) REPAIR — fill score with its mean (82.5), city with a label
filled = df.fillna({"score": df["score"].mean(), "city": "unknown"})
print(filled)
#    name  score     city
# 0   Ana   90.0      NYC
# 1   Ben   82.5       LA      <- score patched with the mean
# 2  Cleo   75.0  unknown      <- city patched with a label
# 3  Dara   82.5       LA

# 3) REMOVE — drop any row that still has a gap (only Ana is complete)
print(df.dropna())
#   name  score city
# 0  Ana   90.0  NYC

Notice df["score"].mean() is 82.5 — the average of 90 and 75, because the mean simply skips the NaNs rather than treating them as zero. That's why repairing with the mean is a safe default: it leaves the column's average unchanged.

The one gotcha

NaN is never equal to anything — not even to itself — so you cannot find missing values with ==; you must use isna. This is the single most common beginner trap. It feels natural to write df[df["score"] == np.nan] to grab the empty scores, but that returns zero rows, always, no matter how many gaps exist — because by design np.nan == np.nan evaluates to False. pandas inherits this rule from the IEEE floating-point standard, where NaN means "undefined," and two undefined things can't be declared equal. The result is a bug that fails silently: no error, just an empty answer that looks like "great, no missing data!" when in fact your filter never matched anything. The fix is to reach for the purpose-built tools every time: df["score"].isna() gives you the mask of missing values (and df["score"].notna() gives the opposite). A second, quieter gotcha: dropna() is greedy — one NaN anywhere in a row deletes the whole row, so a table riddled with sparse gaps can shrink to almost nothing. Always run df.isna().sum() first to see how much you'd lose, and prefer fillna when the data is too precious to throw away.

See it — detect, then repair or remove

flowchart TB
    S["Raw table<br/>(has NaN holes)"] --> D["isna: flag every<br/>hole (True/False)"]
    D --> Q{"Keep the rows,<br/>or drop them?"}
    Q -->|"value is worth<br/>saving"| F["fillna: patch each<br/>NaN with a substitute"]
    Q -->|"row is unusable"| R["dropna: delete rows<br/>with any NaN"]
    F --> C["Clean, complete table<br/>(no NaN remains)"]
    R --> C

The one-line takeaway

Mark gaps as NaN, find them with isna (never ==), then either patch them with fillna or delete them with dropna — repair what's worth saving, remove what isn't.

Where you'll meet it

You'll meet missing data on the very first real dataset you load — a CSV with blank cells, a merged table where some keys didn't match, an API that returned null for optional fields. Cleaning it is step one of almost every analysis, so isna, fillna, and dropna become muscle memory fast. In interviews and take-home tasks, expect "how many missing values are in each column?" (df.isna().sum()), "fill the missing ages with the median age" (fillna with a computed value), and "drop rows where the target label is missing" (dropna(subset=["target"])). A great answer knows why you can't use == np.nan, can explain the trade-off — filling preserves rows but invents data, dropping keeps only real data but discards rows — and always checks df.isna().sum() before dropping so it knows exactly how much it's about to lose. Get fluent here and messy data stops being intimidating: it's just detect, then decide.

Named concepts: NaN, isna, fillna, dropna