DA-01-013-P · Primer — Split-Apply-Combine Analysis with groupby

The idea in one sentence

groupby takes a table with many rows, sorts those rows into piles by some category, computes one summary number per pile, and hands you back a tiny table with a single tidy answer per category — turning "here's all the data" into "here's the answer, broken down by the thing you actually care about."

Why it exists

Almost every real analytical question has a hidden "…by what?" attached to it. Not "what's our average revenue?" but "average revenue by region"; not "how many signups?" but "signups by month." A single grand total flattens exactly the comparison you were hired to find — it tells you the company made money without telling you which region carried it or which one is quietly bleeding. You could answer this by hand: filter to West, take the mean, write it down; filter to East, take the mean, write it down; repeat for every category until you retire. groupby exists to make that loop disappear. It formalizes a pattern so common it has a name — split-apply-combine: split the rows into groups, apply the same calculation to each group, combine the per-group results into one compact table. That pattern is the workhorse of analysis because it matches how humans actually reason about data — we don't want the average of everything, we want to compare slices, and comparison is where insight lives.

How it works

groupby answers one shape of question: "what is [one summary statistic] of [this column], for each [category]?" You name the grouping column (the categories), pick the column you want summarized, and pick the aggregation — .mean(), .sum(), .count(), .max(), and so on. Pandas silently does the split (bucket the rows by category), the apply (run your aggregation inside each bucket), and the combine (stack the results into one row per category, indexed by the category name). What comes back is much smaller than what went in: nine messy order rows become three clean per-region numbers you can read in a glance or drop straight into a bar chart. Because the result is indexed by category, sorting it ranks your groups instantly — biggest to smallest, best to worst. Watch it collapse nine orders into a three-region answer and a chart:

import pandas as pd
import matplotlib.pyplot as plt

# 9 orders across 3 store regions — revenue per order (dollars)
df = pd.DataFrame({
    "region":  ["West", "West", "West", "East", "East", "East", "South", "South", "South"],
    "revenue": [   120,    140,     20,    300,    280,    320,     90,     80,    100],
})

# SPLIT by region -> APPLY mean to each group -> COMBINE into one row per region
by_region = df.groupby("region")["revenue"].mean()
print(by_region)
# region
# East     300.0   <- clear leader
# South     90.0
# West      93.3

# Turn the per-group answer into a bar chart:
by_region.sort_values().plot.bar(edgecolor="black")
plt.ylabel("mean revenue per order ($)")
plt.xlabel("region")
# -> 3 bars, one per region; East's bar towers over West & South,
#    so "which region sells best?" is answered at a glance.
plt.tight_layout()
plt.show()

The nine-row table became a three-number verdict — East averages 300, more than triple the others — and the bar chart turns that verdict into something a stakeholder reads in one second.

The one gotcha

Which per-group statistic you pick can flip the story, and the count that quietly hides missing values will fool you. A region can lead on mean revenue while trailing badly on sum — high average per order but very few orders — so "best region" depends entirely on whether you asked for average order size or total money brought in; those are different questions wearing the same word. Always say out loud which one you mean before you trust the ranking. The second trap is sneakier: .count() counts only the non-missing values in a group, so if some rows have a blank revenue, groupby("region")["revenue"].count() will report fewer rows than the group actually has, and a region can look small just because its data is patchy, not because it's quiet. When you want "how many rows are in this group, no matter what," use .size() (which counts every row, blanks included) — and eyeball .count() versus .size() side by side whenever a group's number looks surprisingly low, because the gap between them is exactly your missing data.

See it — the split-apply-combine flow

flowchart TD
    A["Table of rows<br/>(region, revenue)"] --> B["SPLIT<br/>bucket rows by region"]
    B --> C["West rows"]
    B --> D["East rows"]
    B --> E["South rows"]
    C --> F["APPLY mean<br/>to each bucket"]
    D --> F
    E --> F
    F --> G["COMBINE:<br/>one row per region"]
    G --> H{"Which stat<br/>did you ask for?"}
    H -->|"mean"| I["avg per order<br/>East wins big"]
    H -->|"sum"| J["total money<br/>can rank differently"]

The one-line takeaway

groupby runs split-apply-combine — pile the rows by category, compute one statistic per pile, combine into a tiny answer table — but name the statistic first, because mean and sum (and count vs size) can tell opposite stories.

Where you'll meet it

You'll meet groupby the very first time someone asks a "by" question — revenue by region, churn by plan, error rate by service — which is to say, roughly every day, because comparison-across-slices is the native language of business questions. You'll meet the split-apply-combine pattern again and again as you graduate from one statistic per group to many at once (.agg(["mean", "sum", "count"])), to grouping by two categories at a time, and eventually to pivot tables and time-series resampling — all of them the same three-step move underneath. You'll meet the per-group aggregation gotcha the day your "top region by count" mysteriously shrinks and you discover half its rows had blank values, or the day switching from mean to sum re-ranks your entire leaderboard and you have to explain to a stakeholder which question you actually answered. Learning to ask "grouped by what, summarized how?" before you touch the keyboard is one of the earliest habits that turns a pile of rows into a decision.

Named concepts: groupby, per-group aggregation, split-apply-combine pattern