Theme 2 — Data Disasters
The reflex this theme builds: the data is guilty until proven innocent.
Why this theme matters
In traditional software, the logic lives in code and the data is just what the logic runs on. In AI/ML (Artificial Intelligence / Machine Learning) systems the data layer is the logic layer: behavior is learned from data and steered by data, so a corrupted feed rewires what the system does — with zero code changes, zero failed deploys, and zero stack traces. Data bugs don't crash; they redirect. A ZIP code sitting in an income field is a perfectly valid number, so everything keeps running while every decision downstream quietly changes.
What makes these failures so durable is that pipelines are held together by unwritten contracts: the column order of a vendor file, the immutability of history, what an inner join asserts about the world, what a label means in the human process that produced it. No compiler checks these contracts, and when one breaks, nothing announces it. Every drill in this theme is one of those contracts breaking silently — and the guardrail that would have made it loud. Abbreviations are expanded on first use; the full reference is glossary.md.
The drills
| # | Drill | One-line hook | Problem diagram | ~Difficulty |
|---|---|---|---|---|
| 7 | The Column Shift | A vendor inserts a column mid-row — ZIP codes become incomes, and the model keeps scoring. | sequenceDiagram | intro |
| 8 | The Backfill That Rewrote History | An in-place cleanup edits the past — and every backtest starts leaking the future. | sequenceDiagram | core |
| 9 | The Merge That Couldn't Be Unmade | Entity resolution merges a father and son — and the split takes three months and a regulator. | sequenceDiagram | core |
| 10 | The Contaminated Year | A feature cache leaked eval rows into training for a year. You found it — now what? | timeline | advanced |
| 11 | The Label That Changed Meaning | Ops changes a workflow; labels change meaning with zero schema diff — and auto-retrain makes it worse. | sequenceDiagram | advanced |
| 12 | The Join That Ate the New Customers | An inner join silently drops a third of traffic — exactly the segment where fraud concentrates. | sequenceDiagram | core |
| 13 | The Day That Happened Twice | A rerun ingests Tuesday twice — fraud rules trip while the volume dashboards celebrate. | sequenceDiagram | core |
Best practices — the data-reliability checklist
What to actually do, distilled from drills 7–13:
- Write a data contract for every feed — expected columns, types, allowed ranges, update cadence — and enforce it at an ingestion gate. A vendor's schema change should fail loudly at the border, not quietly at the decision.
- Parse fail-loud: columns by name, never by position, with type assertions on every field and range/distribution checks on the values. Median income halving overnight is a pipeline bug, not an economy.
- Keep history append-only. Corrections are new events with their own timestamps — never in-place edits. Revoke routine UPDATE rights on historical tables, and alarm when anyone uses the exception.
- Preserve point-in-time correctness: every training set and backtest must be reconstructible as of the decision moment, using only what was knowable then. If the past can be edited, every backtest in the company is on borrowed time.
- Use reversible representations for destructive operations: merges as views with full lineage (soft-merge), deletions as flags (soft-delete), source records immutable. Route low-confidence automated actions to humans — the blast radius of "wrong" can include regulators.
- Run contamination checks and out-of-time validation in CI (Continuous Integration — the automated pipeline that tests every change): every training run automatically verifies that no evaluation-period rows touched training data, instead of waiting for an annual audit.
- Assert row counts and coverage across every join and filter, with alarms on unexplained attrition. Every join is a population decision — and evaluate the model per segment, so a silently dropped population can't hide inside aggregate metrics.
- Make ingestion idempotent — safe to run twice: natural keys with deduplication, or transactional exactly-once writes per partition. Then prove it by actually rerunning jobs in staging; reprocessing safety is a property you test, not assume.
- Put label-producing teams in your change-notification loop. Labels are manufactured by a business process; when the process changes, the data silently changes meaning with zero schema diff.
- Gate automated retraining on a diagnosis step whenever label statistics shift. A metrics crash can mean the world changed — or just the bureaucracy did — and retraining on the wrong answer makes the model genuinely worse.
- Treat correctness failures with full incident machinery: declared severity, honest disclosure to leadership and clients, re-stated metrics, blameless postmortem. The coverup, not the contamination, is what destroys trust.