Drill 30 — The Hero's Hotfix

Theme: People & Process · Format: 20-min drill · Difficulty: core

The story

2 a.m., data incident: corrupted rows are blocking the decision pipeline. A senior engineer — the one who knows everything — hand-edits the production rows, unblocks the pipeline, and closes the incident. Everyone's grateful. No record of the edit exists outside their terminal history. Three weeks later, a downstream job breaks on "impossible" state; two engineers burn two days on it before someone stumbles across the edit in database audit logs.

⏸ Stop here and ask the room: the fix worked. What's the problem?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

The trap in this drill is moralizing. The room will want to either canonize the engineer ("they saved the night") or condemn them ("cowboy") — steer to the third reading: the engineer did the only fast thing the system offered. Keep the two-day downstream burn visible on the whiteboard; it's the price tag that makes the discussion concrete.

"What made the 2 a.m. edit feel reasonable?"

Listen for: an honest reconstruction — the pipeline was down, money was burning, the engineer had the knowledge and the access, and every sanctioned path was slower or didn't exist. Strong answers notice the incentive gradient: the fix earned gratitude immediately and the cost landed three weeks later, on someone else.

Weak signal: "they should have known better" — hindsight without mechanism. If the answer doesn't explain why a competent person chose this, it won't prevent the next one.

"What should 'emergency manual change' look like instead?"

Listen for: a designed fast path — a break-glass script that wraps the edit, records who/what/when/why automatically, and posts the artifact into the incident channel. Also good: the norm that every command run during an incident gets pasted into the channel as it runs, so the record is a by-product of working, not extra work.

Weak signal: "manual edits should be banned" — a ban with no fast alternative just guarantees the next edit is undocumented too.

"Is the problem the engineer — or the fact that manual editing was the only fast path?"

Listen for: the systems answer — heroics are a symptom of missing infrastructure. If the sanctioned path takes an hour of ceremony while the outage costs money by the minute, the system has effectively ordered the shortcut. Bonus insight: the success of the hotfix is what makes it dangerous, because it trains the team that this is how incidents end.

Weak signal: blaming or excusing the individual and stopping there — either way the next 2 a.m. looks identical.

The reveal

The hero's fix traded a visible incident for an invisible one. And because it worked, it taught the team that heroics are how incidents end.

Why this happens — the mechanics

At 2 a.m. under production pressure, humans optimize for the shortest path to "the pipeline is moving again." Fatigue narrows attention to the immediate goal — cognitive tunneling — and everything outside the tunnel, like writing down what you just did, feels like a luxury for tomorrow. The engineer wasn't careless; they were operating exactly as human attention operates at that hour, which is why the fix has to be the record rather than relying on a tired person to document afterward.

The gratitude is the trap. The team's response — relief, thanks, incident closed — is a reward signal, and it lands on the behavior that ended the pain: a fast, invisible, out-of-band edit. Reinforcement doesn't care what the postmortem says; it cares what got applauded at 2 a.m. Each celebrated rescue makes the next shortcut more likely and more normal — normalization of deviance with a hero's badge on it. Meanwhile the cost is displaced: three weeks away, in someone else's sprint, disconnected from its cause. The feedback loop that would teach the real lesson never closes.

And the invisible edit poisons something subtle: the team's shared model of production. Every engineer debugging downstream assumes the data got there by known processes — that's what makes the state "impossible." An undocumented change converts the system from something reasoned about to something archaeologically excavated. The two burned days weren't spent fixing anything; they were spent rediscovering a fact one person already knew.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant EN as Senior engineer
    participant PD as Production data
    participant INC as Incident
    participant DS as Downstream pipeline
    Note over INC: 2 a.m. - corrupted rows<br/>block the decision pipeline
    EN->>PD: Hand-edits the corrupted<br/>rows from a terminal
    PD-->>INC: Pipeline unblocked
    EN->>INC: Closes the incident -<br/>everyone grateful
    Note over EN: No record of the edit outside<br/>terminal history (silent)
    Note over INC: Team learns heroics are how<br/>incidents end (silent)
    PD--xDS: Three weeks later - job breaks<br/>on impossible state
    Note over EN,DS: Two engineers burn two days<br/>before finding the edit in<br/>audit logs

Solution — where the guardrails go:

flowchart TD
    A["2 a.m. incident: corrupted<br/>rows block the decision<br/>pipeline"] --> B["Incident channel: every<br/>command pasted as it runs"]
    B --> C["Manual repair goes through<br/>a break-glass script"]
    C --> D["Script audits itself: who<br/>changed what, when, why"]
    D -.-> E["Edit visible in the<br/>incident timeline"]
    C --> F["Pipeline unblocked —<br/>incident closed"]
    F --> G["Postmortem celebrates the<br/>systemic fix, not the<br/>midnight save"]
    E -.-> H["Weeks later: downstream<br/>oddity traced to the edit<br/>in minutes"]

What good looks like

A dedicated incident channel per incident, with the norm that every command is pasted as it runs — the channel is the log, and the log is a by-product of working, not a documentation task afterward.
An incident-scribe role in anything above the smallest severity: one person whose only job is the timeline — actions, times, decisions — so the responders don't have to choose between fixing and recording.
Break-glass repair scripts for the known classes of manual fix (row repair, queue drain, reprocess): each one requires a ticket reference and a one-line reason, supports dry-run, and writes who/what/when/why to an audit stream automatically.
An incident-close checklist that asks "were any out-of-band changes made?" — the incident cannot close until every manual action is captured as an artifact linked from the timeline.
Data changes leave markers: rows touched by manual repair carry a repaired_by/repaired_at annotation or an entry in a repairs table, so downstream "impossible state" debugging starts with one query.
Postmortems celebrate the systemic contribution — the guard added, the break-glass script written — by name, and pointedly not the midnight save; what leadership applauds is what the team learns to repeat.
Terminal sessions on production data hosts are recorded by default, so even the fix that bypasses everything still leaves a findable trace measured in minutes, not days.

The lesson

Incident discipline is the log: every action during an incident is recorded (a dedicated channel where commands are pasted works fine), out-of-band changes are captured as artifacts, and manual data repairs go through break-glass scripts that audit themselves. Celebrate the postmortem contribution, not the midnight cowboy — heroism that bypasses process is a debt instrument.

Named concepts: incident command discipline, break-glass with audit, action logging, heroism as anti-pattern.

Real-world echoes

GitLab, January 2017 — a fatigued engineer, working late on database replication lag, ran a deletion against the wrong production database directory. The incident is remembered for the backups (see drill 32), but the setup is this drill's shape: high-stakes manual operations on production data, by the person who knew the most, at the worst hour — and the recovery honesty that followed (a public, blameless postmortem, even a livestream of the recovery) is what turned a disaster into a lesson.

Transfer

During our last incident — could you reconstruct, today, every command that was run? If not, what's the lightest-weight logging ritual we'd actually follow at 2 a.m.?

Make it a ticket: create the incident-channel template with the paste-as-you-run rule and a close-checklist item for out-of-band changes; write the first self-auditing break-glass script for our most common manual data repair.

Drill 31 — The Postmortem That Named Names — what the postmortem does with this hotfix decides whether it happens again.
Drill 34 — Locked Out at the Worst Moment — the break-glass path this engineer needed, designed instead of improvised.
Other themes — Drill 8 (The Backfill That Rewrote History) and Drill 9 (The Merge That Couldn't Be Unmade) show what un-audited data changes do at scale.