DA-01-012-P · Primer — Correlation Is Not Causation

The idea in one sentence

Two things moving together tells you they share a pattern — never that one makes the other happen.

Why it exists

Every analyst eventually finds a beautiful, tight correlation and feels the urge to announce a cause. It is the single most expensive mistake in data work: you rename a coincidence as a lever, someone pulls the lever, and nothing moves. The reason the mistake is so easy is that our brains are causal-story machines — show us A and B rising together and we instantly narrate "A drives B." But three different worlds produce the exact same scatterplot: A really causes B, B really causes A, or a hidden third thing C quietly drives both while A and B never touch. From the numbers alone you cannot tell these apart. This idea exists to install a permanent pause between "I see a strong r" and "I found a cause," so you go looking for the lurking C before you write the headline.

How it works

The question to ask is never "how strong is this correlation?" but "what else could be moving both columns at once?" — that suspect is the confounder (a lurking third factor). A correlation with a hidden common cause is called spurious: it is statistically real but causally empty. The classic tell: ice-cream sales and drownings rise together across the year, correlation near a perfect 1.0 — yet ice cream drowns no one; summer heat independently pushes up both cones sold and people swimming. The way to see through the illusion is to hold the suspected confounder fixed: regress each variable on it, then correlate the leftover residuals (the wiggle temperature can't explain). If the eye-popping r melts toward zero once temperature is removed, the original link was the confounder's shadow, not a real bridge. Below, a raw r of 0.999 collapses to near-zero — proof the two never touched.

import pandas as pd
import matplotlib.pyplot as plt

# Beach town, 12 months. Heat is the TRUE driver of both columns.
temp = pd.Series([46, 50, 58, 67, 76, 85, 89, 86, 75, 63, 52, 47], name="avg_temp_f")
wig1 = [3, -4, 2, 5, -3, 1, -2, 4, -5, 2, -1, 3]              # tiny noise
wig2 = [.1, -.1, .05, -.05, .1, -.1, .05, -.05, .1, -.1, .05, -.05]
df = pd.DataFrame({
    "avg_temp_f": temp,
    "ice_cream_sales": (temp * 8 - 250 + wig1).round().astype(int),
    "drowning_deaths": ((temp - 45) * 0.22 + wig2).round(1),
})

# 1) The scary raw link: ice cream "predicts" drownings almost perfectly
raw = df["ice_cream_sales"].corr(df["drowning_deaths"])
print("raw r =", round(raw, 3))                    # raw r = 0.999  -> looks like a smoking gun

# 2) Remove the lurking factor: regress each on temp, correlate the leftovers
def resid(y, x):
    x1 = x - x.mean()
    beta = (x1 * (y - y.mean())).sum() / (x1 ** 2).sum()
    return y - (y.mean() + beta * x1)

partial = resid(df["ice_cream_sales"], df["avg_temp_f"]).corr(
          resid(df["drowning_deaths"], df["avg_temp_f"]))
print("r after removing temp =", round(partial, 3)) # ~ -0.3, basically noise: the link was fake

# 3) Color the SAME scatter by temperature -> the confounder becomes visible
plt.scatter(df["ice_cream_sales"], df["drowning_deaths"],
            c=df["avg_temp_f"], cmap="coolwarm", s=90)
plt.colorbar(label="avg_temp_f")
plt.xlabel("ice_cream_sales"); plt.ylabel("drowning_deaths")
# The dots march cold -> hot along the line: temperature drives BOTH axes.
plt.show()

The one gotcha

"Controlling for" the confounder only works if you actually have that column — you cannot subtract a lurking factor you never measured, so the deadliest confounders are the ones missing from your table entirely. A flat partial correlation means "no link beyond what this one variable explains," not "no link at all"; a different unmeasured factor could still be hiding. And never control for a variable that sits on the causal path between your two columns (a mediator) — removing it erases the very effect you were trying to measure. When in doubt, list your suspects out loud before you touch .corr().

See it — a strong correlation, three possible worlds

flowchart TD
    R["Strong correlation<br/>between A and B"] --> Q{"Which world<br/>produced this?"}
    Q -->|"A drives B"| C1["Real cause<br/>(pull A, B moves)"]
    Q -->|"B drives A"| C2["Reverse cause<br/>(you had it backwards)"]
    Q -->|"C drives both"| C3["Spurious link<br/>(lurking factor C)"]
    C3 --> T["Control for C:<br/>correlate residuals"]
    T --> Z["r collapses to ~0<br/>= fake link confirmed"]

The one-line takeaway

Before you call a correlation a cause, name the lurking third factor — then hold it fixed and watch whether the link survives.

Where you'll meet it

Every "X boosts Y" claim in dashboards, A/B readouts, marketing-attribution reports, and health headlines; any time a stakeholder points at a chart and says "so if we do more of this…" — that's your cue to hunt the confounder.

Named concepts: confounding variables, spurious correlation, lurking third factors