Drill 39 — The Vendor's Silent Update

Theme: Feedback Loops & Adversaries · Format: 20-min drill · Difficulty: core

The story

A bureau attribute your underwriting model weighs heavily is quietly redefined by the vendor — the contract permits it, and notice was duly sent to an email alias nobody monitors. Scores shift for a subpopulation. Approval mix drifts for six weeks, showing up as a vague "the model feels off" before anyone connects it to an input. The vendor confirms: "documented in the March release notes."

⏸ Stop here and ask the room: whose failure is this, and what system makes it a non-event next time?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

The stop question has two halves; hold the room on "whose failure" until someone gets past blaming the vendor — the contract was honored, the notice was sent.

"How would you detect an input's meaning changing when its schema doesn't?"

Listen for: per-source distribution monitoring — population stability index (PSI) or a similar drift metric on each critical attribute, sliced by segment, because this shift hit one subpopulation and could hide inside the global distribution.

Weak signal: schema validation, type checks, and null-rate alarms as the whole answer. The field stayed a well-formed number throughout; semantic drift passes every structural test ever written.

"Where should vendor release notes land?"

Listen for: the same change timeline as internal deploys — a monitored intake with an owner and an acknowledgment step, so "attribute redefined in March" sits next to "model retrained in March" when someone investigates drift. Not an alias; a process.

Weak signal: "fix the email alias" or forward it to a bigger distribution list. A notice nobody must acknowledge is still a notice nobody reads — the alias failed precisely because it was fire-and-forget.

"What belongs in the contract that isn't there?"

Listen for: notification service level agreements (SLAs) with teeth — advance-notice windows for semantic changes, a machine-readable change feed, versioned attributes served in parallel during transitions, remedies for unannounced changes.

Weak signal: "we'll raise it at the next vendor quarterly business review." A conversation is not a control; the current contract already permitted everything that happened.

The reveal

The detection signal existed from day one: that attribute's distribution shifted sharply for one segment. Nobody monitored inputs per source — only model outputs.

Why this happens — the mechanics

The root structure is a reliability boundary drawn in the wrong place. The team monitored what it deployed — its own models, its own pipelines — while a vendor attribute change is functionally a deploy into your system by someone else, on their schedule, appearing nowhere on your change timeline. When drift showed up, investigators correlated it against internal events, found nothing, and stalled: "the model feels off" is what an unexplained shift sounds like when the explaining event lives on a timeline you can't see.

Second, the change was semantic, not structural — and most input validation is structural. The attribute kept the same type, range, and field name; every schema check passed. What changed was what the number meant for one subpopulation, which is visible only as a distribution shift within that segment. Output-level monitoring eventually caught the echo — approval mix drift — but outputs are far from causes: by the time a redefined input has propagated through a model into approval rates, the signal is diluted, delayed, and ambiguous among a dozen candidate explanations.

Finally, notice-by-email is a legal control masquerading as an operational one. The vendor's obligation was discharged the moment the message hit the alias; the team's actual need — a correlated event on the timeline an investigator consults — was never anyone's contractual deliverable. The gap between "duly notified" and "operationally aware" is exactly six weeks wide in this story.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant VD as Vendor
    participant EA as Email alias
    participant MI as Model inputs
    participant RT as Risk team
    VD->>EA: Attribute redefinition<br/>notice - March release notes
    Note over EA: Alias unmonitored - notice<br/>never read (silent)
    VD->>MI: Redefined attribute flows<br/>into scoring (silent)
    Note over MI: Distribution shifts for<br/>one subpopulation (silent)
    MI->>RT: Approval mix drifts<br/>for six weeks
    RT->>RT: Vague sense - the model<br/>feels off
    RT->>VD: Finally traced to the input
    VD-->>RT: Documented in the March<br/>release notes

Solution — where the guardrails go:

flowchart TD
    A["Vendor redefines the<br/>bureau attribute"] --> B["Vendor comms routed into<br/>the change timeline"]
    B -.-> T["Change event lands next to<br/>internal deploys"]
    A --> C["Attribute distribution<br/>shifts for one segment"]
    C --> M["Per-source input<br/>distribution monitoring"]
    M -.-> AL["Alarm fires on day one"]
    AL --> R["Shift correlated with the<br/>vendor event; model<br/>retrained"]
    T -.-> R

What good looks like

Per-source, per-attribute distribution monitoring on every model-critical external input — PSI or equivalent, computed per segment, alarmed to the risk team.
Segment-level baselines, because a shift confined to one subpopulation can be invisible in the global distribution.
Vendor communications routed into the change timeline: a monitored intake (parsed inbox or API feed) that creates a change event with an owner and an acknowledgment SLA.
A dependency register: each external input mapped to its vendor, contract, notification channel, monitoring status, and the models that consume it.
Contractual notification SLAs for semantic changes — advance notice measured in weeks, a machine-readable change feed, and versioned attributes served in parallel during transitions.
A drift-investigation runbook whose first step is overlaying the change timeline — internal and vendor — on the drift window, before anyone hypothesizes.
Quarterly reconciliation of vendor release notes against observed input distributions: every documented change matched to a visible shift, or an explicitly confirmed non-event.

The lesson

Your reliability boundary extends into every vendor. Monitor inputs like outputs (per-source distribution alarms), route vendor communications into the same change-timeline as your own deploys (Drill 11's lesson, external edition), and negotiate notification SLAs with teeth. A dependency you can't observe is a dependency you've decided to trust blindly.

Named concepts: dependency governance, input monitoring, change-event correlation, vendor SLAs.

Real-world echoes

Google Flu Trends — the celebrated search-query flu tracker degraded quietly as search behavior and Google's own search features shifted underneath it, famously overestimating flu prevalence badly in the 2012–13 season before being shut down. The inputs' meaning drifted while the pipeline kept running — this drill's mechanics, with the "vendor" being the searching public and the platform itself.

Transfer

List our five most model-critical external inputs. Which have distribution monitoring today? Where do their vendors' release notes actually go?

Make it a ticket: Build the dependency register for the top five model-critical external inputs — monitoring status, vendor notification channel, contract clause — and file one gap ticket per unmonitored input.

Drill 41 — The Coached Application — model inputs shaped outside your walls, deliberately rather than incidentally.
Drill 35 — The Model That Ate Its Own Tail — another way the world under a model changes while its metrics keep reading well.
Drill 11 — The Label That Changed Meaning (Theme 2) — the internal change-timeline discipline this drill extends across the vendor boundary.