Drill 62 — The 99.9% That Didn't Matter

Theme: Vendors & Dependencies · Format: 20-min drill · Difficulty: core

The story

The primary fraud-data vendor goes down — fully — for an entire business day. No fallback was ever designed: no cached last-known values, no secondary vendor, no degraded-mode policy. Applications queue for nine hours while the ops floor refreshes the vendor's status page. The contract's SLA (Service Level Agreement — the contracted reliability promise, usually with penalties) performs exactly as written: a service credit of $400 lands on next month's invoice. The day's business impact is six figures.

⏸ Stop here and ask the room: the contract worked precisely as agreed. Who failed?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Read the story and pause at the ⏸. The instinct in the room will be to blame the vendor — let that happen, then point back at the sentence "the contract worked precisely as agreed" and hold the question open. The drill turns when someone says "we did."

"What does that SLA actually promise — reliability, or a refund?"

Listen for: someone who has read the remedy clause and can say it out loud — credits capped at a fraction of monthly fees, consequential damages excluded. The strong answer states the reframe unprompted: an SLA prices their failure at their rates; it does nothing to prevent yours.

Weak signal: quoting the nines as if they were a physical property of the service ("they're three nines, so this can't really happen") — or nobody in the room ever having seen the actual remedy language on a contract we've signed.

"What is our policy during their outage: approve cautiously, hold, degrade — and who chose it?"

Listen for: a written, business-approved degraded-mode policy — which application classes proceed with a flag, which hold, for how long, and the name of the person who chose that trade-off in daylight, with the risk appetite on the table.

Weak signal: "we'd figure it out in the incident channel" — which means the policy is queue-and-wait, chosen by nobody, at 9 a.m., under pressure. A default decided by absence is still a decision; it's just an unowned one.

"Has anyone ever watched what our system does when this vendor simply isn't there?"

Listen for: a game day — the vendor black-holed in staging, actual behavior observed, findings turned into tickets. Strong answers know the difference between a fallback that exists in an architecture diagram and one that has actually engaged.

Weak signal: "the retry logic should handle it" — should is the tell. A fallback that has never engaged is a hypothesis, and nine hours of queueing is what hypothesis-testing in production looks like.

Facilitator close: answer the ⏸ question explicitly — the vendor kept their contract; we never designed our side of the outage. Both things are true, and only one of them is fixable by us.

The reveal

The SLA was read at procurement as a reliability guarantee; it is a refund policy. Nobody had ever asked "what do WE do during their outage?" — so the answer got decided by default, at 9 a.m., by nobody: queue everything and wait.

Why this happens — the mechanics

An SLA (Service Level Agreement) credit is a refund, not resilience. Read the remedy clause on almost any vendor contract: credits are capped at a percentage of the monthly fee, consequential damages are excluded, and the payout prices the outage at the vendor's rates — what they charged you for the service — not at yours, which is what the service protects. $400 against six figures isn't a broken contract; it's the contract working exactly as its authors intended. The SLA transfers a sliver of cost and none of the risk, and the risk was always the point.

The gap opens at procurement, in translation. "99.9% uptime" reads like an engineering guarantee, but the nines describe intent and history, not a bound — and even a perfectly honored 99.9% permits nearly nine hours of downtime a year, with nothing said about whether those hours arrive one minute at a time or all on the day your queue is deepest. The number that mattered was never in the contract: what does an hour of this vendor being gone cost us? Nobody computes it, because the SLA's confident-looking nines feel like the answer to a question they don't actually address.

And the real decision — what do we do while they're down — has no natural owner, so it never gets made. It's a business-risk trade-off (approve cautiously and eat some fraud risk, hold and eat the queue, degrade to a cached or secondary signal) that engineering assumes the business will call, while the business doesn't know the dependency exists. Decisions with no owner get made by defaults, and the default in every system is the same: block on the dead dependency and wait. That's how "who chose queue-and-hold?" comes to have the worst possible answer — nobody, at 9 a.m., in real time.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant VN as Vendor
    participant AP as Applications
    participant SLA as SLA contract
    participant BZ as Business
    Note over VN: Fraud-data vendor goes<br/>fully down for a day
    AP--xVN: Every call fails - no cache,<br/>no secondary, no policy
    Note over AP: No degraded mode was<br/>ever designed (silent)
    Note over AP: Applications queue for<br/>nine hours
    Note over BZ: Ops floor refreshes the<br/>vendor status page
    VN-->>SLA: SLA performs exactly<br/>as written
    SLA->>BZ: Service credit of $400 on<br/>next month's invoice
    Note over BZ: Day's impact is six figures -<br/>the contract worked perfectly

Solution — where the guardrails go:

flowchart TD
    A["Primary fraud-data vendor<br/>goes down"] --> B["Fallback cache engages:<br/>last-known values with age<br/>labels"]
    B --> C["Degraded-mode policy<br/>chosen by the business in<br/>daylight"]
    S["Secondary vendor covers<br/>the critical subset"] --> C
    C --> D["Low-risk applications<br/>proceed flagged, high-risk<br/>hold"]
    A -.-> M["Dependency health probe<br/>detects the outage in<br/>minutes"]
    M -.-> AL["Ops paged with a runbook,<br/>not a status page"]
    G["Quarterly game day: black-<br/>hole the vendor in staging"] -.-> B

What good looks like

• A fallback-mode spec per critical vendor, written down: trigger condition, chosen mode (cache-and-degrade, secondary source, or queue-and-hold), maximum duration, who can invoke it, and exit criteria for returning to normal.
• Business sign-off on the degraded mode, recorded: the risk trade-off (approve flagged vs. hold) is chosen by a named business owner in daylight, reviewed annually — not improvised in an incident channel.
• Cached last-known values with age labels and a max-age policy: decisions made on stale data carry the staleness visibly, and there's a written answer to "how old is too old to use?"
• Dependency health probes with runbook-linked paging: your own synthetic checks detect the vendor outage in minutes and page with the fallback runbook attached — nobody's job is refreshing a status page.
• A quarterly vendor game day: black-hole the vendor in staging, watch what actually happens, ticket every gap between the spec and the observed behavior. A fallback that has never engaged is a hypothesis.
• An SLA-review step in procurement: for every critical vendor, compute worst-case credit vs. worst-case hourly business impact and put both numbers in the risk register — the gap between them is the size of the fallback you owe yourself.
• Degraded-mode decisions flagged downstream: anything approved during a vendor outage is marked for later review, so the fallback's cost is measurable, not folded invisibly into the book.

The lesson

SLA credits are not business continuity. Every critical dependency needs a designed, tested fallback mode — cache-and-degrade, a secondary source, or an explicit queue-and-hold policy chosen by the business in daylight, not improvised during the incident. Then run a game day: black-hole the vendor in staging and watch what actually happens, because a fallback that has never engaged is a hypothesis.

Named concepts: SLA vs. business continuity, fallback modes, vendor game days, single-source risk.

Real-world echoes

• Fastly (June 2021). A single customer's configuration change triggered a latent bug in Fastly's CDN (Content Delivery Network) and took major sites — news outlets, e-commerce, government portals — offline worldwide for about an hour. Sites with multi-CDN failover came back sooner; single-vendor ones simply waited. One vendor, one config, global blast radius.
• CrowdStrike (July 2024). A faulty content update in CrowdStrike's security sensor crashed roughly 8.5 million Windows machines worldwide — grounding flights and disrupting hospitals and banks. The vendor's update pipeline was, in effect, a deploy into every customer's fleet, with no customer-side canary or cooling-off. Vendor updates are deploys, and no SLA credit resembles the cost of a day like that.

Transfer

Take our most critical vendor. What literally happens in the first hour of their total outage — and who decided that? If the answer is "nobody decided," the game day schedules itself.

Make it a ticket: schedule the game day — black-hole that vendor in staging for four hours, record what actually happens, and bring a proposed degraded-mode policy to the business owner for a signature. The signature is the deliverable.

Related

• Drill 61 — The Price That 10×'d — the other contract term read optimistically at procurement; there it was pricing, here it's the remedy clause.
• Drill 63 — The Acquisition — single-source risk on a slower clock: the vendor doesn't go down, it goes away.
• Drill 22 — quotas: the same rhyme when the dependency isn't gone but rationed — someone else's ceiling becomes your outage.