Pattern — Data Lineage

The problem it solves

The same gap shows up in two costumes. A regulator asks a lender which model scored this exact application, on what inputs, trained how — and nobody in the building can answer, because the artifact was a bare current.pkl overwritten four times and no decision recorded its version (Drill 89). A vendor silently shifts a CSV column and twelve days later a human notices income holds ZIP codes — with no record of which file, schema, or field version fed each score (Drill 7). The common shape: an output is fully determined at the moment it's produced, and then the evidence evaporates — the artifact gets overwritten, the warehouse gets backfilled, the feed changes — because nothing in the code ever asked for the trail to be kept.

The pattern

Data lineage is the recorded chain that ties every output back to the exact inputs, code, and versions that produced it — provenance you write down synchronously at production time, never reconstruct later from mutable state. Two moves make it real: give every artifact an immutable, content-addressed version (a hash-pinned identity, never a mutable path), and emit a provenance record per output — one row, written in the same transaction as the decision, carrying the version, an input hash/snapshot, the code SHA, and the timestamp. Lineage turns "the model decided" and "the pipeline ran" into a query you can answer in minutes.

How to apply it

1. Version every artifact immutably — content-address it. Promotion registers a new version pinned to an artifact hash (credit-scorer:v37 → sha256:9f2a…); it never overwrites a path. The scoring service loads a specific registered version and can never load a bare file. Block any deploy whose artifact hash isn't in the registry. Target 100% of production outputs carrying a resolvable model_version — a filename is not an identity.
2. Emit one provenance record per output, synchronously. In the same transaction as the decision, write {output_id, model_version, input_hash, feature_snapshot (or its hash + rehydratable pointer), code_git_sha, dataset_snapshot_id, seed, upstream_source_id, ts}. Alert if any adverse-action decision lands without a matching lineage row — that count must hold at 0. Reconstruction later reads a row, not a warehouse.
3. Fingerprint the inputs at the border. Hash each incoming file's (column names, order, types) and record which fingerprint fed each downstream output, so a mid-row column shift is attributable to a specific file version, not discovered by domain smell twelve days later. Capture lineage at column level where fields are decision-critical, dataset level elsewhere to bound cost.
4. Set retention to the audit window, and make deletion lineage-aware. Keep artifacts and provenance records for at least the regulator's window — 25 months for ECOA / Reg B adverse-action, longer where model-risk policy (SR 11-7) demands. Deletion must refuse to remove any artifact still referenced by an in-window record. Prove it quarterly with a reconstruction game day: pick a random past output and rebuild it end to end — version, inputs, reproduced score — with a stopwatch; if it takes more than an hour, the lineage is already broken.

flowchart TD
    T["Training run"] --> P["Pin data snapshot + git SHA<br/>+ seed (reproducible)"]
    P --> R["Registry: immutable<br/>content-addressed version"]
    IN["Input file / feature vector"] --> FP["Fingerprint + hash<br/>at the border"]
    R --> D["Output produced"]
    FP --> D
    D --> L["Provenance row (synchronous):<br/>version + input hash + SHA + ts"]
    L -.-> Q["Audit query rebuilds any<br/>past output in minutes"]
    RET["Retention = audit window"] -.-> Q

When to use it — and when not

Reach for it on every output that a regulator, auditor, or incident review could ask you to reconstruct — credit and lending decisions, medical or safety scoring, anything moving money or adjudicating people. Skip the full apparatus where it earns nothing: throwaway exploratory analysis, dashboards, and low-stakes internal jobs don't need per-row provenance, and column-level lineage across every field is expensive — a synchronous record on high-cardinality, low-stakes traffic costs storage and write latency the audit will never repay, so drop to dataset-level or sampled lineage there. Know its limit too: lineage records what happened, it doesn't make the artifact reproducible. A version pointer to code that ran against a moving warehouse with no seed still can't be re-run; pair lineage with pinned, reproducible training (dataset snapshot + git SHA + fixed seed) so the recorded version actually rebuilds.

Signals it's working

Coverage is the headline: the share of production outputs carrying a resolvable version and a matching provenance row should sit at 100%, and decisions with no lineage row must be 0 — alarm the instant either slips. Track reconstruction time as a live SLO: a random past output rebuilt in under an hour (game-day p95). Watch for orphaned artifacts (a version referenced by a decision but missing from the store) and fingerprint mismatches (an input feed whose schema hash changed without a contract update) — both must be 0. In CI, fail any training run that produces a model with no recorded lineage.

Seen in the wild

• MLflow Model Registry, DVC, and Weights & Biases version models and datasets as immutable, hash-addressed entities rather than mutable files.
• OpenLineage + Marquez standardize column- and dataset-level lineage across pipelines; dbt and Databricks Unity Catalog render the provenance graph automatically.
• Delta Lake / Iceberg time travel pin an exact table snapshot ID so a decision can be replayed against the data it actually saw.
• ECOA / Reg B and SR 11-7 are the regulatory reason lending shops keep model + decision lineage for the 25-month adverse-action window.

See it

The ChainScene carries the mechanism — pin the version, fingerprint the input, write the synchronous provenance row, reconstruct on demand — and the CardsScene shows the load-bearing config (100% version coverage, 0 decisions without a row, 25-month retention, sub-hour reconstruction).

Named concepts: data lineage, decision logging, immutable versioning, provenance capture.