Drill 99 — The Right to an Explanation

Theme: Compliance, Privacy & Model Governance · Format: 20-min drill · Difficulty: core

The story

The consumer-lending team ships a new underwriting model everyone is proud of: a gradient-boosted ensemble, nine hundred trees over roughly two hundred features, a couple of points of AUC over the old scorecard. It goes live behind the application form. An applicant fills in the form, the model returns 0.31, that falls under the approval cutoff, and the loan-origination system does what it has always done — it generates an adverse-action notice and mails it. The notice, populated by the field the new model actually exposes, says: "Your application was declined because your credit score did not meet our requirements." Every declined applicant for the past four months has received a variant of the same sentence.

Then a declined applicant writes back and asks the obvious question — which factors, specifically? — and forwards the letter to a regulator. The Equal Credit Opportunity Act's Regulation B (Reg B) requires that an adverse-action notice state the specific principal reasons the application was denied; "score too low" is not one of them. For any applicant in the EU the same demand arrives through GDPR Article 22: a decision made solely by automated processing carries a right to meaningful information about the logic involved and a right to human review. Compliance forwards the request to the data-science team with one line: give me the top reasons this person was denied.

The team opens the model. Global feature importance they have — averaged over the whole training set. Per-decision reasons they do not. The old scorecard was a sum of points; you could read off which factors cost this applicant the most. The ensemble is nine hundred interacting trees, and nobody in the room can point at applicant #48213 and say, with a straight face and a number behind it, these four things, in this order, caused your denial.

⏸ Stop here and ask the room: the notice has already gone out to thousands of people. What do you tell the regulator — and what should have shipped alongside the model?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Why isn't "the score was too low" a legal reason — the model really did produce a low score?

• Listen for: Reg B (and the FCRA behind it) demands the specific principal reasons — the actual factors, ranked — not the output of the scoring step; "score too low" restates the decision instead of explaining it. The right the applicant holds is to contest the decision, which is impossible without the factors. GDPR Article 22 adds the parallel European demand: meaningful information about the logic, plus a route to a human.
• Weak signal: "but the score is accurate" — accuracy is not the question; explainability of the individual decision is. A perfectly calibrated model with no per-decision reasons is still non-compliant.

Global feature importance says income and utilization matter most overall — can we just print those on every notice?

• Listen for: no — global importance is a property of the dataset, not of this applicant; the reasons must be the factors that pushed this decision below the line, which differ person to person. You need a local attribution — SHAP (Shapley Additive Explanations) values per prediction, or a monotonic model whose contributions are directly readable — mapped to a fixed dictionary of Reg B reason codes.
• Weak signal: "print the same top-five features for everyone." That is a template dressed up as an explanation; two applicants denied for opposite reasons would get identical letters, and the first spot-check exposes it.

Explainability was never in the model's acceptance criteria — where should it have entered, and who owns the human-review path?

• Listen for: explainability is a launch gate, not a retrofit: no per-decision reason codes, no production traffic — the same shape as the validation gate in this theme. And Article 22's human-review right is a real operational path — a named queue, an underwriter with authority to override, an SLA — not a promise nobody staffed.
• Weak signal: "we'll add a LIME/SHAP script later." Later is after the notices shipped; the point of the drill is that later is a regulator's inbox. Also weak: "a human rubber-stamps the model" — review without authority to overturn is not review.

The reveal

The honest answer to the regulator is that the notices were non-specific for four months — a systemic Reg B defect across every denial, not one bad letter. Remediation is not "write a script": the team has to bolt a per-decision explanation layer onto a model that was never designed to have one, re-derive reasons for every affected applicant, re-issue corrected notices, and stand up a human-review queue that did not exist. The model's two points of AUC bought nothing that survived contact with the first applicant who asked why. The lesson lands hard: a credit decision you cannot explain is a credit decision you are not allowed to make — the accuracy was never the binding constraint.

Why this happens — the mechanics

The failure is a mismatch between what the model optimizes and what the law requires. A gradient-boosted ensemble is an interaction machine: its lift comes precisely from nine hundred trees composing non-linear, feature-crossing rules, and that is exactly what destroys a simple per-decision reason. The old scorecard was additive — each factor contributed a fixed number of points, so the reasons were the biggest point deductions, readable by construction. Trading the scorecard for the ensemble traded inherent explainability for accuracy, and nobody priced the explanation back in. It is the classic accuracy-versus-explainability tension, made concrete: the two AUC points and the missing reason codes are the same trade seen from two sides.

It stays silent because nothing in the pipeline fails on the day the model ships. The scores are good, the notices generate, the letters mail, tickets close. There is no exception, no error rate, no dashboard that turns red — a notice saying "score too low" is a successful notice as far as every system can tell. The regulatory clock is not driven by any signal the team monitors; it is driven by the first applicant who reads the letter and asks a question the model can't answer. So the defect compounds across every denial for months, invisibly, and then arrives all at once as a systemic finding rather than a single mistake — the same inverted cost curve every drill in this theme rides.

Underneath both is a governance seam: explainability was treated as a model property to be checked if someone remembered, not a shipping requirement wired into the gate. The team that could have answered "which factors?" in an afternoon at design time — while they knew the model cold and the reason-code mapping was one table — is asked to answer it two quarters later, for thousands of past decisions, under a regulator's deadline, by reverse-engineering their own ensemble.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant AP as Applicant
    participant GBM as Ensemble model
    participant NT as Notice system
    participant REG as Regulator
    AP->>GBM: Submit credit<br/>application
    GBM->>NT: Score 0.31 below<br/>the cutoff
    NT->>AP: Notice says<br/>score too low
    Note over NT,AP: Months of generic<br/>notices ship (silent)
    AP->>REG: Which factors<br/>denied me
    REG->>GBM: State the specific<br/>principal reasons
    GBM--xREG: 900 trees give no<br/>per-decision reason
    Note over GBM,REG: Systemic Reg B defect<br/>across every denial

Solution — where the guardrails go:

flowchart TD
    A["Applicant denied by<br/>the ensemble"] --> RC["Reason-code layer<br/>(per-decision SHAP or<br/>monotonic model)"]
    RC --> MAP["Top 4 drivers mapped<br/>to Reg B codes"]
    MAP --> TMPL["Adverse-action template<br/>(specific reasons, no<br/>generic fallback)"]
    RC --> DOC["Model documentation<br/>(reason mapping, monotonic<br/>constraints)"]
    TMPL --> HR["Human-review path<br/>(named queue, override<br/>authority)"]
    MAP -.-> G["Launch gate + notice lint<br/>block any generic reason"]
    DOC -.-> G
    HR -.-> G
    G -.-> AL["Zero notices ship without<br/>4 specific reasons"]

What good looks like

• Per-decision reason codes, not global importance: compute SHAP (Shapley Additive Explanations) values per prediction, take the top four negative contributors (Reg B commentary treats up to four principal reasons as sufficient), and map each to a fixed dictionary of adverse-action reason codes. Constrain the ensemble with monotonic constraints on regulated-relevant features so a factor's direction is defensible and the SHAP-derived reasons never contradict the model's own logic. Guardrail metric: monotonicity violations = 0 in validation.
• A notice-generation lint that blocks the generic fallback: the notice service refuses to send any adverse-action letter whose reason field is empty or the generic "score too low" default. Target: 100% of adverse actions carry ≥1 specific reason code, alarm on the first exception — this is the monitor that would have caught the four-month defect in week one.
• Reason codes documented in the model registry: the SHAP-to-reason-code mapping, the monotonic constraints, and the validation that reasons are directionally consistent live as versioned artifacts on the registry entry — so an examiner's "how do you generate reasons?" is a lookup, not an archaeology project.
• A staffed human-review path (GDPR Article 22 / Reg B): any declined applicant can request human review through a named queue; a human underwriter with authority to override the model responds within the 30-day adverse-action window. Metric: reviews requested vs. completed within SLA, page if the queue ages past the deadline.
• Adverse-action templates driven by the reason codes: the letter is populated from the top-four codes in plain language — never free-typed, never the model's raw score — so every applicant's notice states the factors specific to their decision.
• A quarterly explanation rehearsal: pull a random production denial and produce its four specific reasons and its human-review record within one day; every gap found in the drill is a Reg B finding you don't receive in the real complaint.

The lesson

If you cannot state the specific reasons for an automated credit decision, you are not allowed to make it — explainability is a launch gate, not a retrofit. Buy accuracy with a black-box ensemble only if you also ship the per-decision reason codes, the documented mapping, and the human-review path that turn "the model said no" into a specific, contestable, lawful answer.

Named concepts: adverse-action reason codes, SHAP (Shapley Additive Explanations), ECOA Reg B / GDPR Article 22, meaningful human review.

Real-world echoes

• Apple Card, 2019: viral allegations that spouses sharing finances received very different credit limits triggered a New York State Department of Financial Services investigation. The regulator ultimately found no unlawful discrimination — but the episode's enduring lesson is how badly it went while the institutions struggled to explain individual limit decisions on demand. "The algorithm decided" satisfied no applicant, journalist, or examiner; the specific per-decision reasons have to exist before the question arrives, which is exactly the capability this drill is about building.

Transfer

Take our highest-volume automated denial and ask for one applicant's specific principal reasons — the actual top factors for that decision, plus the record of how they could get a human to review it. If the honest answer is the model's score or a generic sentence, we are already out of compliance on every notice that endpoint has ever sent; we just haven't received the complaint yet.

Make it a ticket: For each model that drives an adverse action, add a per-decision reason-code layer (SHAP or monotonic) mapped to Reg B codes, a notice lint that blocks any generic reason, and a staffed human-review queue with a 30-day SLA — and gate production traffic on all three, starting with the highest-volume denial path.

Related

• Drill 52 — The Proxy Variable — disparate-impact testing and this drill's reason codes are the two halves of an ECOA-compliant model: one proves the decision was fair, the other explains it to the person it was made about.
• Drill 50 — The Examiner's Question — the reason-code mapping and monotonic constraints are exactly the kind of artifact the validation report has to document; no explanation capability, no defensible validation.
• Drill 49 — The Zombie Model — without a registry entry there is nowhere for the reason-code mapping to live, and no way to prove every scoring model can explain itself.