BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 34 — Locked Out at the Worst Moment

Theme: People & Process · Format: 20-min drill or 60-min tabletop · Difficulty: advanced

The story

Sev-1: the decisioning service is misbehaving and the one engineer who understands the subsystem is on the bridge, ready — and lacking production access. She transferred teams last month; least-privilege automation dutifully revoked her roles. The access-request workflow needs her new manager's approval; the manager is on a plane. The incident runs ninety extra minutes while people screen-share so she can dictate commands to someone with permissions.

⏸ Stop here and ask the room: least privilege worked as designed. Is the design wrong?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer. As a 60-min tabletop: after the reveal, walk your actual access stack against this scenario — pick a real subsystem, name the real expert who'd be locked out tomorrow, and time the real request path on a whiteboard.

The stop question is a genuine dilemma, not a gotcha — let the security-minded half of the room defend the revocation (they're right: standing access for ex-team-members is how breaches happen) and the reliability-minded half attack the missing exception path (they're right too). The drill lands when both sides notice they're describing the same requirement: fast, temporary, audited.

"How do you keep least privilege and emergency speed?"

Listen for: the reframe from trade-off to design problem — least privilege governs standing access; emergencies need a separate, pre-built temporary path. The knobs that make it safe are time (auto-expiry), audit (every action logged under the real identity), and authority (the incident commander grants it, not a org-chart approval chain that assumes business hours).

Weak signal: "keep access longer after transfers" or "give senior people permanent broad access" — both answers quietly abolish least privilege instead of designing its exception. The residue of that thinking is the standing god-mode account.

"What should break-glass access require, and what should it leave behind?"

Listen for: require — an active incident, an incident commander's explicit grant, a stated scope and duration, ideally two-person visibility (the grant announced on the bridge). Leave behind — a complete audit trail under the user's own identity, an auto-revocation event, and a mandatory postmortem review of every grant. The test of a good answer: it's fast because it's heavily audited, not despite it.

Weak signal: listing approvals — "security sign-off, then the manager, then…" — an emergency path with an approval chain is the normal path wearing a red sticker. If it's slower than the screen-share, the screen-share wins again.

"Is dictating commands through a screen-share actually safer than granting access?"

Listen for: a clear no, with the mechanics — the commands are hers, but executed by hands that can't sanity-check them, with transcription errors added, at incident speed; and the audit log now attributes every action to the wrong person. All of the risk of her having access, plus new risk, minus the truthful record. The workaround defeats the purpose of the control while honoring its letter.

Weak signal: "at least someone with permissions was in the loop" — the typist had permissions but not understanding; that's not a control, it's a relay with packet loss.

The reveal

The screen-share workaround was the worst of both worlds: her commands, executed by hands that couldn't sanity-check them, with the access system's audit log showing someone else did everything.

Why this happens — the mechanics

Least privilege was designed against one threat model — compromised or excessive standing access — and it performs beautifully there. But an access system is also a component of incident response, and nobody had designed for the intersection: the moment when the person with the most context has the fewest permissions. Each piece behaved correctly in isolation — the revocation automation, the approval workflow, the manager's flight — and the composition produced a ninety-minute failure. Security controls that ignore the emergency case aren't neutral about it; they silently delegate its design to whoever is on the bridge at the worst moment.

Under incident pressure, people don't stop when the sanctioned path is blocked — they route around it, using whatever the tools permit. Screen-sharing and dictation weren't chosen because anyone thought they were safe; they were chosen because they were available. That's the general law this drill teaches: if the sanctioned emergency path doesn't exist or is slow, the unsanctioned one will be invented live, under maximum pressure, with zero review — and it will be worse on every axis, including the one the original control was protecting.

The subtlest casualty is audit integrity. The point of access logging is a truthful answer to "who did what?" — and the workaround corrupted exactly that, crediting every action to the typist while the actual operator never appears. A post-incident review, a security investigation, or a regulator now reads a lie told by a compliant system. A break-glass path — pre-built emergency access, granted in seconds, time-boxed, auto-revoked, logged under the real identity — isn't a hole punched in least privilege; it's the mechanism that keeps the control honest under pressure, because the alternative to a designed exception is an undesigned one.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant BR as Sev-1 bridge
    participant EX as Expert
    participant AW as Access workflow
    participant MG as Manager on plane
    participant SR as Screen-share relay
    Note over BR: Decisioning service<br/>misbehaving
    BR->>EX: The one engineer who knows<br/>the subsystem joins, ready
    Note over EX: Transferred teams last<br/>month - roles auto-revoked<br/>(silent)
    EX->>AW: Requests production access
    AW--xMG: Needs the new manager's<br/>approval
    Note over MG: On a plane - request sits<br/>unanswered (silent)
    EX->>SR: Dictates commands to hands<br/>with permissions
    Note over SR: Hands that cannot<br/>sanity-check what they<br/>type (silent)
    Note over SR: Audit log credits someone<br/>else with every action<br/>(silent)
    Note over BR,SR: Incident runs 90 extra<br/>minutes

Solution — where the guardrails go:

flowchart TD
    A["Sev-1: the expert joins<br/>the bridge without access"] --> B["Incident commander grants<br/>break-glass access in<br/>minutes"]
    B --> C["Grant is time-boxed and<br/>auto-revoked"]
    B --> D["Every action logged under<br/>the expert's own identity"]
    D -.-> E["Audit trail tells the<br/>truth about who did what"]
    C --> F["Expert works directly — no<br/>dictation relay"]
    F --> G["Incident resolved 90<br/>minutes sooner"]
    B -.-> H["Postmortem reviews every<br/>break-glass grant"]

What good looks like

The lesson

Least privilege needs a designed exception path: break-glass access that any incident commander can grant in seconds — time-boxed, heavily audited, auto-revoked, reviewed in the postmortem. If the emergency path doesn't exist, people invent worse ones under pressure, and the audit trail lies about who did what.

Named concepts: break-glass procedures, least privilege with escape hatches, incident-commander authority, audit integrity.

Real-world echoes

Transfer

In a Sev-1 tonight, can our incident commander grant temporary prod access in under five minutes, with an audit trail? Walk the actual clicks tomorrow morning.

Make it a ticket: define the break-glass role for our most critical system — IC-grantable, time-boxed, auto-revoked, logged under the user's own identity — then run a timed drill of the grant-use-revoke-review loop and record the minutes.

Related