Drill 25 — The Key That Expired on Saturday

Theme: Scale & Cloud · Format: 20-min drill · Difficulty: intro

The story

Saturday, 3:04 a.m.: vendor-data calls start failing. The service-account credential — created 366 days ago by an engineer who left in the spring — has expired. On-call can't fix it: nobody knows where the key is provisioned, the vendor portal login lives in the departed engineer's password manager, and the runbook says "contact Priya" (also departed). Resolution arrives nine hours later via a former teammate's memory, reached through a personal phone number.

⏸ Stop here and ask the room: count the separate failures in this story.

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

How many of these failures are about the key?

Listen for: roughly none — the expiry itself is routine and inevitable; everything that made it a nine-hour incident is organizational. Strong answers enumerate: no inventory of what expires (nobody saw it coming), no pre-expiry alert (nobody was warned), no rotation automation (a human had to act), no ownership registry (nobody knew whose job it was), no break-glass documentation (nobody knew the manual path), and knowledge trapped in one departed person. The count matters less than the categorization: one technical trigger, five-plus organizational absences.

Weak signal: "the key should have had a longer lifetime" or "the engineer should have documented before leaving." Longer lifetimes just reschedule the incident with more accumulated forgetting; blaming the departed engineer locates a systemic failure in one person — which is the bug the drill is about.

What inventory would have prevented it?

Listen for: a queryable inventory of everything that expires — service-account keys, API keys, TLS (Transport Layer Security) certificates, licenses, domain registrations, reserved capacity, vendor contracts — each with expiry date, provisioning location, renewal procedure, and an owning role. The killer feature is the standing query: "what expires in the next 120 days, and who owns it?" run automatically, with alerts landing in a team channel long before the date. Strong answers add: the inventory must be populated at creation time (a key that isn't registered can't be issued), or it decays into another stale document.

Weak signal: "a spreadsheet" with no owner, no automation, and no creation-time hook — that's a diagram-lies problem (Drill 23) waiting to happen in table form. Also weak: inventorying only cloud credentials and forgetting vendor portals, certs, and licenses.

What does "bus factor" mean for credentials, vendor logins, and quota tickets?

Listen for: bus factor is the number of people whose departure breaks a capability — and this story's bus factor was one, then zero. For credentials it means: secrets live in a shared team vault, never a personal password manager; vendor portal access is held by at least two named humans and one role account; runbooks name roles ("data-platform on-call"), never people ("contact Priya"); and offboarding includes a transfer checklist for everything the departing person owned. Strong answers notice the quiet detail: the fix arrived through a personal phone number — the org's real continuity plan was social memory, which is not a plan.

Weak signal: treating it as a documentation problem only. Documentation decays; the durable fixes are structural — shared vaults, role-based ownership, automation that doesn't need to remember.

The reveal

At least five distinct failures: no expiry inventory, no pre-expiry alert, no rotation automation, no ownership registry, no break-glass documentation. The expired key was just the messenger.

Why this happens — the mechanics

An expiring credential is an incident with a countdown timer attached at creation — the rare failure whose exact date is knowable in advance. The reason it still lands as a surprise is that the countdown is invisible: expiry metadata lives wherever the key was created (a cloud console, a vendor portal, a laptop), not in anything that pages. Green dashboards measure the present; an expiry is a fact about the future, and almost no monitoring stack has a place to put those. So a fully deterministic event arrives with the surprise profile of a random one — at 3:04 a.m. on a Saturday, because expiry times are uniformly distributed and weekends are two-sevenths of the week.

The second mechanism is knowledge decay. The key worked for 366 days without anyone touching it — and every quiet month made the system look safer while making the organization less capable of fixing it. The creator left; the runbook's named human left; the vendor login stayed in a personal password manager. Manual, rare procedures rot in exactly this way: the knowledge to perform them lives in the people who performed them last, and tenure is shorter than credential lifetimes. That's why toil like manual rotation isn't merely inefficiency — it's fragility with a due date, and the due date is written down somewhere nobody looks.

Nothing about this failure is testable in staging, because staging shares the same organizational gaps — same missing inventory, same departed owner. The only rehearsal that works is the organizational kind: run the query ("what expires next quarter, who owns it — by role?"), and fire-drill the break-glass path with someone who didn't write it. If on-call can't renew the key using only the runbook, the runbook has failed its test in daylight instead of at 3 a.m.

Diagrams

Problem — how the failure unfolds:

timeline
    title A credential incident scheduled 366 days in advance
    Day 0 : Service-account key created : Expiry set 366 days out
    Spring : Owning engineer departs : No inventory, no handover (silent)
    Months pass : No expiry alert exists (silent) : Runbook still says contact Priya
    Saturday 3am : Key expires : Vendor-data calls start failing
    On-call scramble : Runbook points to ex-employee : Vendor login in ex-employee vault
    Nine hours later : Fix via a former teammate memory : Reached on a personal phone

Solution — where the guardrails go:

flowchart TD
    A["Expiry inventory covers<br/>every credential and cert"] -.-> B["Alarm fires 30 days before<br/>expiry"]
    B --> C["Rotation automation renews<br/>the key ahead of time"]
    A --> D["Ownership registry maps<br/>each secret to a role"]
    D --> E["On-call always has a live<br/>owner to page"]
    A --> F["Break-glass runbook<br/>documents the manual path"]
    C --> G["Saturday 3 a.m. passes<br/>without incident"]
    F --> G

What good looks like

A central expiry inventory populated at creation time: secrets are issued through the vault or a provisioning workflow that records expiry date, provisioning location, renewal procedure, and owning role — unregistered keys simply can't be created.
A nightly job that queries everything expiring within 120 days, posting to a team channel with escalating alerts at 30, 14, and 7 days — addressed to a role's channel, never to an individual's inbox.
Rotation automated wherever the provider supports it (managed rotation, short-lived tokens, workload identity instead of long-lived keys) — the best expiry alarm is one that automation already handled last week.
Ownership by role in a registry: every secret, vendor login, and quota ticket maps to a team role; runbooks say "page the data-platform on-call," and the offboarding checklist reassigns everything the departing person owned before their last day.
Shared team vault as policy: vendor portal logins and break-glass credentials live in the team vault with at least two humans holding access — a credential in a personal password manager is treated as an incident, not a convenience.
A break-glass runbook that's been fire-drilled: once or twice a year, someone who didn't write it renews a key in staging using only the document — if they can't, that's the finding.
Expiry dashboard reviewed quarterly alongside the anomaly registry: certificates, keys, licenses, domains, reserved capacity — anything with a date is on it, and the gaps in the query are themselves tracked as tickets.

The lesson

Every credential, certificate, and quota is an incident scheduled in advance unless it's inventoried, alerted ahead of expiry, rotation-automated, and owned by a role (not a person). Toil like manual rotation isn't just inefficiency — it's fragility with a due date.

Named concepts: secret management, toil elimination, ownership registries, bus factor, break-glass procedures.

Transfer

Query every credential and cert we hold: what expires in the next 120 days, and who — by role — owns each one? The gaps in that query are the finding.

Make it a ticket: Run the 120-day expiry query this week — inventory every credential, certificate, and license with its expiry date and owning role; file one ticket per gap (no date known, no role owner, no renewal doc) and add the 30-day pre-expiry alert to the team channel.

Drill 22 — The Quota Wall — quotas and credentials belong in the same inventory: limits and dates that can say no, discovered under stress unless found in daylight.
Drill 23 — "Multi-AZ" — "known to exactly one person" is this drill's bus factor wearing an infrastructure costume.
Drill 5 — The Twice-a-Year Bug (Theme 1 — Silent Failures): calendar periodicity as a fingerprint — expiry cycles belong on the same calendar review.