Drill 22 — The Quota Wall

Theme: Scale & Cloud · Format: 20-min drill · Difficulty: core

The story

Promo weekend, volume 4× — planned, load-tested, capacity-modeled. At peak, scoring calls start failing with 429s. Not CPU, not memory, not the database: a cloud provider API quota nobody knew existed (per-second auth-token issuance) is throttling. The emergency quota-increase request enters the provider's support queue, where it waits its turn like everyone else's.

⏸ Stop here and ask the room: the capacity plan was wrong. What dimension did it miss, and what do you do right now?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

What limits exist in our stack that we didn't set?

Listen for: a genuine inventory instinct — provider API (Application Programming Interface) rate limits, auth-token issuance, account-level caps, per-service defaults, connection-pool sizes, third-party vendor rate limits, license seats. The strongest answers note that a limit you didn't set is still your limit, and that defaults are sized for average customers, not for your promo weekend. Someone should say: the capacity plan modeled resources we own and skipped limits we rent.

Weak signal: the list stops at compute — "CPU, memory, disk, maybe network." If nobody names a single quota that lives on the provider's side of the line, the team has never gone looking, which is exactly how this incident happens.

Why didn't the load test catch it?

Listen for: the load test ran in an environment that didn't share production's limits — a different account, a different tier, mocked or short-circuited auth, cached credentials. A load test proves the path it exercises; if the token-issuance call isn't on that path at production shape, the quota is invisible. Strong answers land on "load-test in a production-shaped account — same quotas, same auth path — because the point of the test is to find the limits, not just to pass."

Weak signal: "we should have load-tested harder / longer / with more virtual users." Volume wasn't the gap — fidelity was. More load through the same unrepresentative path finds nothing new.

What's the immediate mitigation — and what should have been negotiated before the weekend?

Listen for: right now — cut the call rate, not the traffic: cache and reuse auth tokens for their full lifetime, batch or coalesce calls, shed non-critical consumers of the same quota, and invoke a business-approved degraded mode if scoring still can't keep up. Before — a quota inventory checked against the 4× projection, raises pre-negotiated weeks ahead, and the event on the provider's radar via the TAM (Technical Account Manager) so an emergency isn't your first conversation.

Weak signal: "file the quota increase and escalate the ticket." That's the story's failure mode restated — at peak, the support queue is the outage. Anyone whose entire plan routes through the provider's ticketing system hasn't found a mitigation.

The reveal

Token caching cuts the call rate enough to limp through. The quota was documented — in a page nobody had read, at a default nobody had questioned.

Why this happens — the mechanics

Cloud quotas are invisible by design until you hit them. They don't consume a resource you can watch fill up — no graph climbs toward a ceiling, because the ceiling belongs to the provider and most dashboards don't render it. A quota at 30% and a quota at 99% look identical: green. The first observable symptom is the 429 ("Too Many Requests" — the provider's throttling response), and by then you're at peak, in public, with the one mitigation path (a quota raise) gated on another company's queue. This is the general shape of invisible provider limits: they surface only under stress, which is precisely when the discovery costs the most.

The capacity plan missed the dimension because capacity planning inherits its checklist from the datacenter era: CPU, memory, storage, bandwidth — things you own and can graph. A modern stack adds a second category: permissions to proceed, granted per-second or per-account by someone else. Auth-token issuance is the purest example — it scales with call count, not with data volume, so a 4× traffic event can produce 4× token requests even though every machine is idle. Any plan that inventories resources but not permissions will pass review and still hit the wall.

Staging never shows it because quota limits attach to accounts and tiers, not to code. The test account has different quotas (often lower, sometimes unmetered), the test harness authenticates once and reuses the session, and the auth provider in the loop may be a stub. The failure needs production's exact account, production's auth path, and production's per-second call pattern — which is why the checklist item is "load-test in a production-shaped account," and why the cheapest fix of all is reading the quota documentation before the weekend instead of during it.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant TR as Traffic
    participant AQ as Auth quota
    participant SC as Scoring
    participant PS as Provider support
    Note over TR: Promo weekend - volume 4x,<br/>planned and load-tested
    TR->>SC: Peak requests arrive
    SC->>AQ: Every call requests<br/>an auth token
    Note over AQ: Hidden per-second<br/>issuance limit (silent)
    AQ--xSC: 429 - throttled
    SC--xTR: Scoring calls fail at peak
    SC->>PS: Emergency quota-increase<br/>request
    Note over PS: Waits its turn in the<br/>support queue
    Note over TR,PS: Outage persists through<br/>the peak

Solution — where the guardrails go:

flowchart TD
    A["Quota inventory lists<br/>every limit that can say<br/>no"] --> B["Limits checked against 4x<br/>peak projection"]
    B --> C["Load test in production-<br/>shaped account finds token<br/>quota"]
    C --> D["Quota raise pre-negotiated<br/>before the weekend"]
    C --> E["Token caching cuts<br/>issuance rate"]
    D --> F["Promo peak served within<br/>limits"]
    E --> F
    B -.-> G["Alarm fires at 80 percent<br/>of any quota"]

What good looks like

A living quota inventory, pulled automatically from the provider's quota APIs (e.g., AWS Service Quotas, GCP quota metrics) plus a hand-maintained section for third-party and license limits — reviewed before every known peak, owned by a named role.
Quota utilization on a dashboard with alarms at 80% — a quota you can't see is a quota you'll discover at peak; a quota alarmed at 80% is a ticket, not an incident.
Auth tokens cached and reused for their full lifetime, refreshed proactively at ~80% of expiry — token issuance rate should scale with token lifetime, never with request volume.
Load tests run in a production-shaped account: same account tier, same quotas, same auth path, no mocked credentials — with "which limits did we hit?" as an explicit deliverable of every test.
Pre-negotiated raises for planned events, filed weeks ahead, plus the event on the provider's calendar through the TAM — emergency tickets during a public peak wait like everyone else's.
A business-approved degraded mode for throttling: what scoring serves when a dependency says 429 (cached decision, queue-and-callback, conservative default) is decided in daylight and rehearsed, not improvised at peak.
429s tracked as a first-class SLI (Service Level Indicator), sliced by dependency — a rising throttle rate is the earliest warning that a quota wall is approaching.

The lesson

Capacity planning must inventory every limit that can say no: provider quotas (API, account, service), connection pools, license seats, third-party rate limits. Load-test in a production-shaped account to find them, raise them ahead of known peaks, and put big events on your provider's radar (that's what TAMs are for).

Named concepts: quota management, capacity planning beyond compute, load testing fidelity, graceful degradation under throttling.

Real-world echoes

Slack, January 4, 2021 — on the first workday of the year, Slack went down as traffic ramped back up: network scaling limits in their cloud provider's infrastructure (AWS Transit Gateways that couldn't scale up fast enough) caused packet loss, degraded their own monitoring, and complicated recovery. The constraint that mattered lived on the provider's side of the line, and it surfaced exactly at the predictable peak.

Transfer

List five limits in our stack that a cloud provider set and we never changed. Now check them against our next peak-traffic projection.

Make it a ticket: Build the quota inventory — export every provider quota via the quota APIs, add third-party and license limits by hand, compare each against the next peak projection, and add an 80%-utilization alarm for the ten closest to the line.

Drill 21 — The Thundering Herd — the other way peak traffic finds a constraint nobody modeled; retries against a 429 wall make this drill's incident worse.
Drill 25 — The Key That Expired on Saturday — quotas and credentials belong in the same inventory-or-incident category.
Drill 2 — Green Dashboards, Zero Traffic (Theme 1 — Silent Failures): a limit with no dashboard is the same species as a service with no traffic alarm — invisible until it's an outage.