PD-01-016-P · Primer — Concatenating Tables

The idea in one sentence

pd.concat glues DataFrames together — stacking them on top of each other to get more rows, or side by side to get more columns — and the direction you glue is chosen by one tiny argument, axis.

Why it exists

Real data almost never arrives as one tidy table. You get January's signups in one file and February's in another; the sales team hands you one spreadsheet and finance hands you another for the same customers. pd.concat exists for exactly this: taking a list of tables that belong together and combining them into a single DataFrame in one move. It's the pandas answer to "I have these pieces — now make them one." The reason it deserves its own tool (rather than looping and appending row by row) is that it does the combining in a single fast pass and it lines up your data intelligently, matching columns by name when you stack rows and matching rows by index label when you stack columns. That alignment is the quiet superpower — and, as we'll see, the quiet trap.

How it works

You call pd.concat([...]) with a list of DataFrames, and the axis argument decides the direction. The default, axis=0, stacks them vertically — piling rows to make a taller table — and pandas aligns them by column name, so columns with matching names line up even if they were in a different order. Switch to axis=1 and it stacks horizontally, gluing columns side by side and aligning by index label instead. When you stack rows, the original index labels come along for the ride, which usually leaves you with repeated labels like 0, 1, 0, 1; passing ignore_index=True throws those away and hands you a clean fresh 0, 1, 2, 3. Watch two monthly tables become one:

import pandas as pd

jan = pd.DataFrame({"name": ["Ana", "Ben"],  "signups": [120, 95]})
feb = pd.DataFrame({"name": ["Cleo", "Dan"], "signups": [47, 88]})

both = pd.concat([jan, feb], ignore_index=True)
print(both)
#    name  signups
# 0   Ana      120
# 1   Ben       95
# 2  Cleo       47
# 3   Dan       88

Two two-row tables went in; one clean four-row table came out, columns matched up by name and the index renumbered 0..3.

The one gotcha

When you stack rows, pandas aligns columns by their name — so a single misspelled or mismatched column name silently splits your data into two half-empty columns padded with NaN. It does exactly what you told it to, not what you meant. If January calls the column signups but February calls it sign_ups, concat sees two different columns, keeps both, and fills the gaps with missing values:

import pandas as pd

jan = pd.DataFrame({"name": ["Ana", "Ben"],  "signups":  [120, 95]})
feb = pd.DataFrame({"name": ["Cleo", "Dan"], "sign_ups": [47, 88]})  # typo!

print(pd.concat([jan, feb], ignore_index=True))
#    name  signups  sign_ups
# 0   Ana    120.0       NaN
# 1   Ben     95.0       NaN
# 2  Cleo      NaN      47.0
# 3   Dan      NaN      88.0

No error, no warning — just a wider table riddled with NaN and a column count that should have stayed the same. So before you concat, confirm the pieces share the exact same column names (df.columns is your friend), and remember that axis=0 matches on column names while axis=1 matches on the index — a mismatch on either one shows up as NaN, never as a crash.

See it — one list of tables, two directions

flowchart TD
    A["jan (2 rows)<br/>+ feb (2 rows)"] --> B{"which axis?"}
    B -->|"axis=0<br/>stack rows"| C["align by<br/>column name"]
    C --> D["4 rows x 2 cols<br/>(taller table)"]
    B -->|"axis=1<br/>stack columns"| E["align by<br/>index label"]
    E --> F["2 rows x 4 cols<br/>(wider table)"]
    D --> G["name mismatch<br/>-> NaN padding"]
    F --> G

The one-line takeaway

pd.concat([a, b]) stacks tables — axis=0 piles rows (aligned by column name), axis=1 glues columns (aligned by index) — and any name or label that doesn't match becomes NaN, not an error.

Where you'll meet it

You'll reach for pd.concat the first time you have twelve monthly CSVs to combine into one year: read each into a DataFrame, drop them in a list, and pd.concat(frames, ignore_index=True) in a single line. You'll meet it when you compute new columns in a separate DataFrame and want them bolted onto the original with axis=1. You'll meet it in interviews the moment someone asks "how would you combine these files?" — and the strong answer names concat, picks the right axis, and mentions checking that column names line up first. It's also the tool people mix up with merge: concat is for pieces that already have the same shape and just need gluing, while merge is for joining on shared keys. Knowing that difference — and remembering that mismatched names quietly become NaN — is what turns "the code ran" into "the data is right."

Named concepts: concat, stacking rows, aligning columns, axis