BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 89 — Which Model Decided?

Theme: Model Lifecycle & MLOps · Format: 20-min drill · Difficulty: core

The story

A digital lender approves and declines credit-card applications with a machine-learning model. It has grown up the way most production models do: a data scientist trains a candidate in a notebook, exports it, and the platform team copies the file to s3://models/credit/current.pkl. The scoring service loads whatever sits at that path. When someone ships a better model, they overwrite current.pkl. Deploys are a file copy; rollbacks are copying the old file back — if anyone still has it.

For eight months this is invisibly fine. Applications come in, scores come out, adverse-action letters go to the declined. Nothing errors. Then a declined applicant files a complaint, and under the Equal Credit Opportunity Act the regulator sends a routine request: for this specific application, on this date, tell us which model scored it, what inputs it saw, and why it produced a decline.

The team pulls the thread and finds nothing to hold. The scoring logs record the application ID, the timestamp, and the final decision — but not which model produced it. There is no model version, because the artifact was never versioned; it was current.pkl, and current.pkl has been overwritten four times since that date. The exact feature vector the model saw wasn't logged either — the service recomputes features from a warehouse that has itself been backfilled twice. And the notebook that trained the model in question ran against a data snapshot no one pinned, on a library version no one recorded, with no fixed seed. Retraining it today produces a different model. Nobody did anything reckless; every step looked like normal shipping.

⏸ Stop here and ask the room: the model works, the decision was made, the letter went out. So why can nobody in this building say which model made it, or reproduce the decision the regulator is asking about?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

The regulator wants the exact model that decided this application. What do you need to have logged — at decision time — to answer that?

The artifact was current.pkl, overwritten four times. Where did versioning need to live, and why isn't a filename enough?

Even with the version, retraining it produces a different model. What makes a training run reproducible?

The reveal

There was no way to answer the regulator, because the answer was never written down. The fix wasn't a smarter model — it was a lifecycle. They stood up a model registry: every promoted model becomes an immutable version pinned to an artifact hash, and the scoring service can only load a registered version, never a bare file. They added a decision log: one row per decision, written synchronously, carrying model_version, a snapshot of the exact feature vector, an input hash, and the timestamp. They made training reproducible — pinned dataset snapshot, git SHA, library versions, fixed seed, all stored as the version's lineage. And they set a retention policy matching the ECOA/Reg B audit window so none of it could age out before a regulator could ask. The next request took minutes: version, inputs, and the reproduced decision, on demand.

Why this happens — the mechanics

A machine-learning decision is a function of three things that all move independently: the model artifact, the exact inputs it scored, and the code/data that produced the artifact. Traditional software has one of these — the code — and version control already pins it. ML adds two more moving parts, and teams that carry over "just deploy the file" habits from app code silently drop the other two. The decision is fully determined at inference time and then the evidence evaporates: the artifact gets overwritten, the feature warehouse gets backfilled, the training data drifts. Nothing errors, because nothing is supposed to be preserved — no line of code asked for it.

The organizational root is that lineage is a cross-cutting concern nobody owns until an auditor creates the owner. The data scientist's job ends at "good metrics," the platform team's job is "the file is served," and no single seam is responsible for "we can reconstruct any past decision." In regulated lending this isn't optional hygiene — ECOA requires a specific reason for every adverse action, and model-risk guidance (SR 11-7) expects you to know exactly which model, trained how, made which call. The failure is invisible right up until someone outside the building asks a question that the inside of the building never wrote down the answer to.

Diagrams

Problem — how the decision becomes unreconstructable:

sequenceDiagram
    autonumber
    participant AP as Applicant
    participant SVC as Scoring service
    participant ART as Model file (S3)
    participant REG as Regulator
    participant ML as ML / platform team
    AP->>SVC: Apply for credit
    Note over SVC: Score with current.pkl<br/>no version recorded
    SVC-->>AP: Declined (adverse action)
    Note over ART: current.pkl overwritten<br/>4 times since (silent)
    REG->>ML: Which model declined this?
    ML--xART: Fetch the exact model<br/>that scored it
    Note over ML,REG: No version, no inputs,<br/>artifact gone — no answer

Solution — where the lineage guardrails go:

flowchart TD
    A["Application scored"] --> B["Decision log row<br/>(version + feature<br/>snapshot + input hash)"]
    T["Training run"] --> P["Pinned data + code<br/>+ seed (reproducible)"]
    P --> R["Model registry<br/>(immutable versions)"]
    B --> R
    R -.-> AL["Audit query resolves<br/>a past decision"]
    B -.-> AL
    RT["Retention policy<br/>(matches audit window)"] -.-> AL
    AL --> H["Reconstruct exact<br/>decision on demand"]

What good looks like

The lesson

An ML decision is only as auditable as what you wrote down at the moment it was made — the model version, the exact inputs, and the recipe that built the model. Version artifacts immutably, log every decision's lineage synchronously, and keep it as long as someone can ask; you cannot reconstruct after the fact what you never recorded.

Named concepts: model registry & versioning, decision logging & lineage, reproducible training, audit retention policy.

Real-world echoes

Transfer

For your highest-stakes model, pick one real decision it made last quarter and try to answer the regulator's three questions: which version, on what inputs, trained how? If you can't produce all three in an hour, your decisions aren't auditable — you just haven't been asked yet.

Make it a ticket: stand up a model registry so nothing reaches production except a registered immutable version, add a synchronous decision log capturing model_version + feature snapshot + input hash on every scored application, pin training (dataset snapshot + git SHA + seed) into each version's lineage, and set retention to the audit window — with a game day that reconstructs a random past decision end to end.

Related