DA-01-011-P · Primer — Correlation and Scatter Plots

The idea in one sentence

A scatter plot draws one dot per row so you can see how two numeric columns move together, and the Pearson correlation r squeezes that whole picture into a single number between −1 and +1 that reports the direction and strength of the straight-line part of that relationship.

Why it exists

The first real question in any analysis of two numbers is "do they move together, and if so, how?" — does spending more on ads bring more sales, does a heavier car burn more fuel, does more study time raise a test score? You can't answer that by staring at two columns of figures side by side; the pattern is buried in the pairing of the rows. A scatter plot exists to make that pairing visible: put one variable on the x-axis, the other on the y, and drop a dot for every row. Now the relationship is a shape your eye reads in one glance — a rising cloud, a falling cloud, a formless blob, or a curve. But a picture is hard to compare, log, or put in a report, so Pearson correlation exists to compress the up-and-to-the-right-ness of that cloud into one comparable number. The plot shows you the shape; the number scores the strength and direction of the linear trend — you almost always want both, because each one lies to you when used alone.

How it works

Pearson's r measures how tightly the dots hug a single straight line: r = +1 is a perfect upward line, r = −1 a perfect downward line, and r = 0 means no linear trend at all. The sign is the direction (up together, or one up while the other goes down); the magnitude is the strength (closer to 1 means a tighter, more line-like cloud). In pandas you get the number from series_a.corr(series_b), or a whole correlation grid from df.corr(numeric_only=True), and you get the picture from ax.scatter(x, y) in matplotlib. The habit that makes you good at this is doing both at once — compute r, then look at the plot and ask "does the shape agree with the number?" Here we do exactly that on six weeks of ad spend and sales:

import matplotlib
matplotlib.use("Agg")            # headless backend; omit in a notebook
import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame({
    "ad_spend": [10, 20, 30, 40, 50, 60],   # $ thousands, one row per week
    "sales":    [12, 19, 33, 37, 52, 58],   # units sold that week
})

r = df["ad_spend"].corr(df["sales"])        # Pearson correlation
print(round(r, 3))                          # -> 0.992  (strong, positive, near +1)

fig, ax = plt.subplots()
ax.scatter(df["ad_spend"], df["sales"])     # each dot = one week
ax.set_xlabel("ad_spend"); ax.set_ylabel("sales")
ax.set_title(f"r = {r:.2f}")
# the chart shows dots marching tightly up-and-to-the-right:
# a clear rising line -> confirms the strong positive r of 0.99

The number (0.99) and the shape (a tight rising line) tell the same story, so you can trust the read.

The one gotcha

r only sees straight lines — a correlation near 0 does not mean "no relationship," it means "no linear relationship," and a curved pattern can be strong yet score zero. This is the trap that separates people who quote r from people who understand it. Take a perfect U-shape: as x rises, y falls, bottoms out, then rises again — a beautifully strong, perfectly deterministic relationship. Pearson looks at it and reports r = 0.0, because there is no single straight line that fits, and the down-half and the up-half cancel out:

import pandas as pd
u = pd.DataFrame({"x": [1, 2, 3, 4, 5, 6, 7],
                  "y": [9, 4, 1, 0, 1, 4, 9]})   # a clean U (y = (x-4)^2)
print(round(u["x"].corr(u["y"]), 3))              # -> 0.0  (!) yet y is fully determined by x

If you had trusted the number alone, you'd have concluded "x and y are unrelated" and thrown away a rock-solid pattern. The reverse trap is just as real: a single wild outlier can drag r up to a big impressive value when the bulk of the cloud is actually a shapeless blob. Both traps have the same cure — never report a correlation you haven't looked at. Draw the scatter plot first: it instantly reveals whether the relationship is linear (trust r), curved (reach for a different tool), or being faked by one stray point. The number summarizes; the plot keeps the number honest.

See it — is Pearson r the right summary for this cloud?

flowchart TD
    A["Two numeric columns"] --> B["Draw the scatter<br/>plot FIRST"]
    B --> C{"What shape<br/>do the dots make?"}
    C -->|"Tight rising/falling line"| D["Linear: trust r<br/>(sign = direction,<br/>size = strength)"]
    C -->|"Curve or U-shape"| E["Nonlinear: r near 0<br/>lies — r is wrong tool"]
    C -->|"Blob + 1 stray dot"| F["Outlier faking r:<br/>investigate the point"]
    C -->|"Formless cloud"| G["No relationship:<br/>r near 0 is honest"]

The one-line takeaway

Always pair the two: draw the scatter plot to see the shape, then read Pearson r to score the straight-line part — because r alone can call a strong curve "nothing" and call one outlier "everything."

Where you'll meet it

You'll meet this the first time someone asks "does X drive Y?" — ad spend vs sales, temperature vs ice-cream orders, hours studied vs grade — and your honest first move is a scatter plot plus one .corr() call. You'll meet the correlation grid, df.corr(), when you're hunting which of many features move with your target before building a model, or spotting two columns that are near-duplicates (r ≈ 1). You'll meet the linear-vs-nonlinear gotcha the moment a real relationship curves — dose vs effect, speed vs fuel economy — and a near-zero r tempts you to dismiss it. And you'll meet it in interviews constantly, where "what's the difference between correlation and a scatter plot, and why do you need both?" and "does r = 0 mean the variables are independent?" (no — it means no linear trend) are classic checks that you reason about the shape, not just the number. Remember the discipline: what question am I asking, what does the plot reveal, and what shape would fool the number I'm about to report?

Named concepts: scatter plots, Pearson correlation, linear vs nonlinear relationships