Drill 103 — The SLA With No Teeth

Theme: Vendors & Dependencies · Format: 20-min drill · Difficulty: core

The story

A payments platform runs every card authorization through a third-party fraud-scoring API. It's a hard dependency on the hot path: no score, no decision. When the vendor was selected two years ago, procurement checked the box that mattered most — a 99.9% uptime SLA — and everyone moved on. The number felt like insurance. Because it felt like insurance, nobody funded a fallback: no cache-and-degrade mode, no second provider, no queue-and-hold. The SLA was the resilience plan.

One Thursday evening at peak, the vendor's scoring endpoint starts returning HTTP 500s on roughly a third of requests and 30-second timeouts on the rest. It never goes fully dark — the login page loads, the status dashboard is reachable, the docs are up. But the one endpoint the platform actually calls is effectively unusable. With no fallback, the platform's timeout path declines transactions it can't score, and for four hours good customers are told their cards were refused. The incident channel lights up; the vendor's status page stays a cheerful green.

The next morning, calmer, the team does the obvious thing: they file for the SLA service credit. That is the moment they read the contract for the first time since signing — and discover that the guarantee they'd been treating as a safety net was never load-bearing at all.

⏸ Stop here and ask the room: the vendor had a 99.9% SLA, and it was down for four hours at peak. Why is the credit going to be almost nothing — and why is that the smaller problem?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Read the story aloud and pause at the ⏸. Let the room reach for "we'll get a big refund" before you show them why they won't — the disappointment is the teaching moment.

The vendor's dashboard says 99.94% this month. Your users had four broken hours. Who's measuring wrong?

• Listen for: the recognition that uptime is defined by whoever wrote the SLA. The vendor measures a control-plane probe (is the service reachable?) on a five-minute poll from their own network, averaged across a whole calendar month — so a four-hour data-plane degradation barely dents the monthly figure. The strong answer: "we have no idea what our own availability was, because we never measured the endpoint we actually call."
• Weak signal: "the vendor is lying" or "we'll show them our logs." The vendor isn't lying; they're measuring a different thing on purpose, and the contract says their measurement governs.

The endpoint returned 500s and timeouts but never went fully offline. Does the SLA even cover that?

• Listen for: the distinction between total unavailability and partial degradation / elevated latency, and the fact that most SLAs only promise the former. "Down" in the contract means a continuous total outage measured by the provider — a request that returns a 500 in 200 ms is arguably a response. The covered failure mode has to be enumerated, or the failure mode that actually hits you falls in the gap.
• Weak signal: assuming any bad experience counts as downtime. The contract is a list of exactly which failures earn a credit, and everything not on the list is free for the vendor.

Say the credit is real — 5% of one month's fee. Is the SLA doing its job?

• Listen for: an SLA is a refund policy, not a reliability guarantee, and a credit capped at a few percent of one month's spend creates ~zero incentive for the vendor to be reliable and doesn't come close to covering your loss. The real job of an SLA is the leverage it gives you: meaningful credits, a chronic-breach exit, and covered failure modes you can measure. The strong answer pivots from "get our money back" to "we should never have been single-sourced on a hot-path dependency with no fallback."
• Weak signal: treating the credit amount as the whole point. The credit was never going to save you; the resilience you didn't build is the finding.

The reveal

The credit is trivial, and it's trivial in three independent ways, each fatal on its own. First, the SLA's "uptime" is measured by the vendor's own reachability probe on a monthly average, so a four-hour peak degradation shows as 99.94% — above the 99.9% floor, so no breach occurred by the contract's math. Second, even if it had dipped below, the failure mode that hit you — elevated errors and latency, not a total outage — is explicitly excluded; the SLA only covers "total unavailability of the Service for a continuous period exceeding 30 minutes as measured by Provider." Third, the remedy is a service credit capped at 5% of one month's fee, claimable only within 30 days with your own evidence, and named as your "sole and exclusive remedy." The guarantee was theatre: a number that reads like protection, measured by the party it protects against, excluding the failures that actually happen, and paying out a rounding error if it pays at all. The real finding in the postmortem isn't the credit — it's that a hot-path dependency had no fallback because a line in a contract was standing in for engineering.

Why this happens — the mechanics

An SLA is a commercial instrument, not an engineering control, and it's written by the vendor to bound their liability, not to guarantee your uptime. Three levers do the work, and all three point the same way. The definition of "up" is the vendor's: they choose the probe (usually control-plane reachability, not the data-plane call you depend on), the granularity (a monthly average washes out even a multi-hour incident), and the vantage point (their network, not yours). The list of covered failure modes is short by design — "total unavailability" is rare, while the degradations that actually hurt you (partial errors, latency cliffs, poisoned responses) live in the uncovered gap. And the remedy is capped and exclusive — a credit worth a few percent of one month's fee, which is cheaper for the vendor to pay than to prevent, and which the "sole remedy" clause bars you from exceeding no matter what you lost.

Underneath sits the asymmetry this whole theme is about: the dependency matters enormously to you and almost nothing to the vendor. Reliability is your production incident and their line item. So the SLA the vendor offers is calibrated to their indifference, and treating it as your resilience plan imports that indifference straight into your architecture. The trap is specifically that a credible-looking SLA suppresses the work that would actually make you resilient — you don't build the fallback, don't measure your own availability, don't dual-source, because a number in a contract told you not to bother. The guarantee's real damage is the engineering it convinced you to skip.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant A as Auth Service
    participant V as Vendor API
    participant P as Vendor Probe
    participant S as Vendor Support
    A->>V: score txn (peak load)
    V--xA: 500s and 30s timeouts
    Note over A: fallback declines<br/>good customers 4h
    P->>P: control-plane reachable<br/>logs 99.94 percent
    Note over V,P: partial degradation<br/>not a total outage
    A->>S: file SLA credit claim
    S--xA: excluded failure mode
    Note over S: remedy 5 percent<br/>of one month fee

Solution — where the guardrails go:

flowchart TD
    A["Vendor outage or<br/>partial degradation"] --> B["Your probe scores every<br/>call at the integration<br/>point (2xx under budget)"]
    B -.-> AL["Alarm: 5-min error rate<br/>over 1 percent OR p99<br/>over 800 ms"]
    A --> C["Contract enumerates<br/>covered failure modes<br/>(latency, partial errors)"]
    C -.-> AL
    AL --> D["Auto-failover to second<br/>provider or defined<br/>fallback policy"]
    D --> E["Tiered credits to 100 pct<br/>+ exit-for-cause right<br/>(sole-remedy clause struck)"]

What good looks like

• Measure your own SLO at the integration point: instrument every vendor call and count success as an HTTP 200 that returns useful data within your latency budget (e.g., ≤ 800 ms). Compute availability and p99 per-endpoint from your side, on a rolling 5-minute window — never trust the vendor's dashboard as your system of record. Page when the 5-minute error rate exceeds 1% or p99 exceeds budget.
• Enumerate covered failure modes in the contract: the SLA must name elevated error rates (e.g., > 1% for > 15 min), latency breaches (p99 over budget), and partial degradation as covered — not just "total unavailability." Make your measurement the system of record for a claim, not the vendor's probe.
• Negotiate credits that actually bite: tiered service credits that scale with severity up to 100% of the monthly fee, and strike the "sole and exclusive remedy" language so a catastrophic outage isn't capped at a rounding error. A credit smaller than your incident cost is not a deterrent.
• Win an exit-for-cause right: a termination clause that triggers on chronic breach — e.g., three SLA breaches in a rolling quarter, or any single breach over a defined duration — with no early-termination penalty and contracted data-return terms. Leverage exists only at signing and renewal; spend it there.
• Dual-source or define a fallback for every critical path: a second scoring provider, or a deterministic degrade policy (cache-and-score, rules-only, or queue-and-hold) chosen by the business in daylight, with automatic failover when the primary's 5-minute error rate crosses 1%.
• Run a vendor game day quarterly: black-hole the dependency in staging and watch what the system actually does. A fallback that has never engaged is a hypothesis; the game day is how you learn it declines good customers before a Thursday at peak does.

The lesson

Read every SLA as a refund policy, not a reliability guarantee — the vendor defines "up," lists which failures count, and caps what it pays. Your resilience is the SLO you measure yourself, the fallback you build, and the exit you negotiated while you still had leverage; the number in the contract is theatre until you can measure it and enforce it.

Named concepts: SLA vs SLO, independent SLO measurement, covered failure modes, service credits and exit rights, multi-vendor redundancy.

Real-world echoes

• AWS US-EAST-1, December 2021. A multi-hour outage disrupted a large slice of the internet, yet cloud SLAs make service credits the "sole and exclusive remedy," cap them at a fraction of the monthly bill, measure availability per-region on the provider's own definition, and exclude much of what actually degrades. Businesses that lost far more than a month's spend were entitled only to those capped credits — same anatomy as this drill: the guarantee is calibrated to the vendor's indifference, and the credit was never going to be your resilience plan.

Transfer

Pick the single vendor whose outage would hurt you most. When did you last measure your own availability against that vendor's endpoint — not their dashboard — and do you know which failure mode their SLA excludes?

Make it a ticket: instrument your top critical dependency to compute a per-endpoint SLO this sprint (success = 2xx under an 800 ms budget, alarm at 1% error over 5 minutes), and read that vendor's SLA to record, in one line, the failure mode it excludes and the credit cap. If either the measurement or the fallback is missing, that's the next ticket.

Related

• Drill 62 — The 99.9% That Didn't Matter: the sibling in this theme where a full-day vendor outage nets a $400 credit; this drill dissects why the number is theatre and how to build the guarantee it isn't.
• Drill 59 — The Deprecation Email: the same asymmetric-attention trap, where a vendor's obligation is discharged by an announcement that matters only to you.
• Drill 23 — Multi-AZ: the cross-theme rhyme on the cloud side, where redundancy you never exercised isn't redundancy — the fallback that has never engaged is a hypothesis.