BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 104 — The Sole-Source Model Provider

Theme: Vendors & Dependencies · Format: 20-min drill · Difficulty: core

The story

A wealth manager runs an advisor copilot with one job on the hot path: before every client call, an advisor asks it to summarize the client's portfolio and flag any suitability issues — concentration risk, an unsuitable product for the stated risk profile, a missing disclosure. The copilot calls a large language model (LLM) behind a single vendor's application programming interface (API), pinned to a specific model snapshot, and returns a tidy summary with a short list of flags. It has worked, unremarkably, for a year. Advisors trust it enough that the flags list is the thing they read first.

There is no abstraction between the copilot and the vendor: the model identifier is hard-coded in roughly forty call sites, the request and response shapes are the vendor's own, and the vendor's software development kit (SDK) is imported directly wherever a call is made. There are no evals against any other provider — nobody has ever measured whether a second model would do this job as well — and there is no fallback path. The vendor is not a dependency in anyone's mental model; it is simply how the copilot works.

Then the vendor sends a routine notice: the pinned model snapshot retires in thirty days, and after that, traffic auto-routes to its successor model. The team reads it as a non-event — same API, same prompt, "it'll just keep working." On migration morning the vendor also has a capacity incident and starts returning 503s. With no fallback provider, the copilot goes dark during market hours; advisors get spinners on the calls that matter most. By midday the errors clear and the successor model is answering — and the summaries still look completely plausible.

⏸ Stop here and ask the room: the API never changed and the endpoint is green again by lunch. So what, exactly, is still broken?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Read the story aloud and pause at the ⏸. The room will want to call the outage the whole incident. Hold them there — the outage is the half you can see.

The endpoint is green again — why isn't the incident over?

One vendor decision took down a production system. Where should the guarantee have lived?

The successor model degraded quality silently. How would you ever have caught that before a client did?

The reveal

The outage was the loud half. The quiet half runs for two weeks: the successor model is chattier and misses a specific disclosure-pattern flag the old snapshot always caught. Every dashboard stays green because availability and latency are fine and the outputs parse. It surfaces only when a suitability recommendation that should have been flagged reaches a client, compliance opens a review, and QA finally traces it back to the silent model swap on migration morning. One vendor's release-calendar line item produced two incidents — a market-hours outage and a fortnight of degraded advice — and the firm had no abstraction, no eval, no fallback, and no exit to blunt either one.

Why this happens — the mechanics

A sole-source dependency on the hot path is a single point of failure by construction: whatever the vendor decides — deprecate a snapshot, throttle capacity, re-price per call, or simply have a bad day — arrives in your production path with no seam to absorb it. This is the theme's asymmetric-attention trap in its sharpest form. Retiring a model snapshot is a line item in the vendor's release plan, discharged with one notice; for you it is every advisor call. The vendor is optimizing their roadmap, not your uptime, and nothing about "same API" guarantees "same behavior" — a successor model is a different model, and auto-migration swaps it under you while the interface stays byte-for-byte identical.

The organizational root is that the dependency was never modeled as a dependency. With the model identifier hard-coded across forty call sites and the vendor SDK imported directly, there is no single place to change providers and — more damning — no place to even run an alternative. That is why nobody has evals across two providers: without an abstraction, "try the other vendor" is a multi-week refactor, so it never happens, so there is no baseline to detect that the successor got worse. The missing abstraction doesn't just prevent the fallback; it prevents the measurement that would have told you the fallback was needed. Availability monitoring completes the trap: it watches whether the endpoint answers, never whether the answer is any good, so a silent quality regression sails straight through a wall of green.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant A as Advisor
    participant S as Copilot service
    participant V as Sole LLM vendor
    A->>S: summarize holdings<br/>flag suitability
    S->>V: call pinned<br/>model snapshot
    V-->>S: good summary<br/>with flags
    Note over S,V: works fine for a year
    Note over V: snapshot retires<br/>auto-routes to successor
    S->>V: same hardcoded<br/>model id
    V--xS: 503 capacity incident
    Note over S: no fallback provider<br/>copilot dark mid-market
    V-->>S: successor answers<br/>flags subtly worse
    Note over S,A: dashboards green<br/>silent for two weeks

Solution — where the guardrails go:

flowchart TD
    A["Advisor asks the<br/>copilot for a summary"] --> B["Provider-abstraction<br/>gateway (one call site)"]
    B --> C["Primary provider<br/>(pinned snapshot)"]
    C -->|"5xx over 2 percent"| D["Fallback provider<br/>(pre-qualified)"]
    E["Nightly evals across<br/>two providers"] -.-> G["Guardrail — page on<br/>regression or no fallback"]
    F["Quarterly game day<br/>plus exit runbook"] -.-> G
    C -.-> G
    D -.-> G

What good looks like

The lesson

A dependency you cannot swap, cannot measure against an alternative, and cannot fail away from is not a vendor — it is a single point of failure wearing an API. Put every external model behind an abstraction, eval it against a live second provider, pin it, give it a fallback, and rehearse the exit — so no one vendor's decision can ever become your outage.

Named concepts: provider abstraction layer, multi-provider evals, fallback and version pinning, vendor exit plan.

Real-world echoes

Transfer

Which of your systems calls exactly one LLM vendor by a hard-coded model identifier on the hot path right now — and if that snapshot were auto-migrated tomorrow, what would tell you the new model got worse?

Make it a ticket: pick your most business-critical LLM call path this sprint; put it behind a gateway with a single provider-agnostic interface, pin the snapshot, stand up a nightly eval against one alternate provider with a numeric regression gate, and schedule a game day that black-holes the primary. Record the measured time-to-switch on the dependency registry.

Related