Drill 7 — The Column Shift

Theme: Data Disasters · Format: 20-min drill · Difficulty: intro

The story

A data vendor adds a helpful new field to their daily CSV feed — in the middle of the row. Your parser reads columns by position. From that morning, the income field contains ZIP codes and the zip field contains phone area codes. Everything downstream keeps running: ZIP codes are perfectly good numbers, and the model happily scores them.

⏸ Stop here and ask the room: what breaks, what doesn't, and when does a human find out?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Why didn't anything crash?

Listen for: The data stayed type-valid — a ZIP code is a legal number, so every layer that checks syntax (parser, type assertions, model input validation) passes cleanly; the failure is semantic, and nothing in the stack checks meaning. Weak signal: "The parser had a bug" — blaming the code, when the code did exactly what it was told; the real defect is an unwritten positional contract with a counterparty.

What's the cheapest check that catches this on day one?

Listen for: A value-level check at ingestion — median or percentile of income compared to yesterday's, or simply asserting on header names before loading a single row; one line of validation catches the whole class. Weak signal: "More tests on the parser" or a pipeline rewrite — solutions aimed at this specific cause, when a range check on the values catches every cause that produces nonsense numbers.

Whose job is it to know the vendor changed the file?

Listen for: Right now, nobody's — that's the actual bug; the fix is a data contract with the vendor plus an ingestion gate that makes your own border the tripwire, and a named owner for the feed with an escalation path. Weak signal: "The vendor should have told us" — true and useless; you don't control the vendor's change process, only what you accept at your boundary.

The reveal

Twelve days. Approval quality sagged subtly — caught when a risk analyst investigated why approved applicants' "incomes" were suspiciously five digits and non-round.

Why this happens — the mechanics

CSV (comma-separated values) is a schema-free format: the file carries no declaration of what each column means, so position becomes the contract by default. That contract lives in two places at once — the vendor's export code and your parser — and neither side can see the other's copy. When the vendor adds a field, their change is locally additive and perfectly reasonable; the breakage happens entirely on your side, in code they've never seen.

The failure stays invisible because every validation layer that does exist checks form, not meaning. A ZIP code passes the "is it a number" test, fits the column width, and lands inside the model's numeric range. The model doesn't know what income is — it knows there's a feature at index 14, and there still is one. From the infrastructure's point of view, nothing happened.

Competent teams hit this because the parser has usually worked flawlessly for years, which reads as evidence that it's solid — when it's actually evidence that the vendor hasn't changed anything yet. Nothing in code review flags positional parsing as a live dependency on another company's release schedule, and cross-company change notification is exactly the kind of contract nobody writes down until it burns them. The result is discovery by domain smell — a human noticing that incomes look like ZIP codes — which is why detection took twelve days instead of twelve seconds.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant V as Vendor
    participant P as Parser
    participant M as Model
    participant RA as Risk analyst
    V->>P: Daily CSV with a new<br/>column inserted mid-row
    Note over P: Reads columns by position -<br/>no header check (silent)
    P->>M: income holds ZIP codes -<br/>zip holds area codes (silent)
    Note over M: ZIP codes are valid<br/>numbers - scores them happily
    Note over M: Approval quality sags<br/>subtly for 12 days
    RA->>M: Why are approved incomes<br/>five digits and non-round?
    Note over RA,M: Day 12 - found by a human,<br/>not a monitor

Solution — where the guardrails go:

flowchart TD
    A["Vendor inserts new column<br/>mid-row"] --> B["Ingestion gate: columns<br/>parsed by name"]
    B --> C["Type and range validation:<br/>median income check"]
    C --> D["Mismatch detected at the<br/>border"]
    D --> E["Feed quarantined, pipeline<br/>stops"]
    D -.-> F["Alarm fires day one:<br/>vendor schema change"]
    E --> G["Model only ever sees<br/>validated rows"]

What good looks like

Parse by header name, never by position — and hard-fail on unknown, missing, or reordered columns rather than guessing.
Fingerprint every incoming file (column names, order, types) and diff it against the written contract before a single row loads; any mismatch quarantines the file.
Range and distribution checks on every decision-critical field: compare median and p5/p95 against a trailing 30-day window — warn at a 20% day-over-day shift in median income, hard-stop the pipeline at 50%.
Quarantine, don't drop: rejected files go to a holding area with an automated diff report, so the fix is a review, not an archaeology dig.
A written data contract per vendor feed — columns, types, allowed ranges, delivery cadence, and a change-notice SLA (Service Level Agreement) — revisited at every contract renewal.
A named owner per feed, with the vendor's escalation contact in the runbook.
Canary the content, not just the schema: score a 100-row sample from each new file and compare the score distribution to yesterday's before the full load proceeds.

The lesson

Parse defensively: columns by name, types asserted, and range/distribution validation at ingestion (median income suddenly $60,000 → $10,999 should slam the pipeline shut). A feed change should fail loudly at the border, not quietly at the decision.

Named concepts: data contracts, schema validation, ingestion gates, fail-fast principle.

Real-world echoes

Unity, 2022 — the company ingested bad data into its ad-targeting models and disclosed roughly a $110M revenue impact: the column-shift class of failure, at public-company scale.
NASA Mars Climate Orbiter, 1999 — one team produced thruster data in pound-seconds, the other consumed it as newton-seconds. Every number was valid; the meaning was wrong — a semantic data contract failure that cost the spacecraft.

Transfer

Which of our feeds are parsed by position, or trusted without range checks? What would happen today if a vendor inserted a column?

Make it a ticket: inventory every external feed we ingest; for each, record whether parsing is by name or position and whether any range check exists — then add an ingestion gate to the worst offender this sprint.

Drill 11 — The Label That Changed Meaning — the same contract break one level up: schema intact, meaning changed.
Drill 12 — The Join That Ate the New Customers — another silent boundary where valid-looking data hides a broken assumption.
Theme 1, Drill 1 — The Model That Scored Everyone the Same: type-valid garbage keeps every technical signal green.