BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 32 — The Backup That Wouldn't Restore

Theme: People & Process · Format: 20-min drill · Difficulty: core

The story

Nightly backups have run green for three years — the job is monitored, the dashboards confirm it. Then a bad migration drops a production table, and the team reaches for the backups. The restore fails: backups older than 90 days were encrypted with a key that was rotated away, and the recent ones restore to a schema three migrations old. Also, the restore takes 26 hours; the DR plan assumed four.

⏸ Stop here and ask the room: three years of green checkmarks. What were they actually measuring?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Let the room answer the stop question precisely — "the backup job exited zero" is the exact wording to land on. Note that this story contains three independent failures (unreadable old backups, stale-schema recent ones, 26-hour duration), and each would have been caught by the same single practice. Ask the room to spot that convergence; it's the drill's spine.

"What's the difference between 'backup succeeded' and 'we can restore'?"

Listen for: the process-vs-outcome distinction — "backup succeeded" measures that a job ran; "we can restore" measures that data comes back, readable, current-schema, within the time budget. Strong answers list what a restore proves that a backup can't: key validity, schema compatibility, storage integrity, duration, and that the runbook for doing it actually works.

Weak signal: "add better monitoring to the backup job" — more precise measurement of the wrong thing. The gap isn't observability of the backup; it's that restorability was never exercised.

"What would a restore drill look like, and how often?"

Listen for: a concrete drill: restore the most precious dataset to a scratch environment, validate row counts and schema against production, time the whole thing against the target, record results, file tickets for gaps. Quarterly as a floor, plus triggered runs after key rotations and major migrations — the two events that silently broke this story's backups.

Weak signal: "restore a sample once a year" — the two silent breakages here happened between annual checks, and a sample restore wouldn't have surfaced the 26-hour duration.

"Who owns the RTO number the business is quietly assuming?"

Listen for: the recognition that "four hours" lived in a DR plan nobody had validated — an assumption, not a measurement. Someone must own the Recovery Time Objective (RTO) as a tested number: measured in drills, reported to the business, renegotiated when reality (26 hours) diverges. If no name comes up, that's the finding.

Weak signal: "the infra team, probably" — an RTO without a named owner and a measurement date is a wish, and the business is making promises on top of it.

The reveal

The green checkmark measured "the backup job exited zero." Restorability — the only thing backups are for — had never been exercised once.

Why this happens — the mechanics

This is monitoring the process instead of the outcome. "Backup job exited zero" is cheap to measure and green every morning; "we can restore within four hours" is expensive to measure and nobody's job. Organizations drift toward the cheap metric, and the green checkmark does something actively harmful: it manufactures confidence. Three years of green taught everyone that backups were the solved problem — the checkmark wasn't just uninformative, it was disinformation with a dashboard.

Meanwhile, latent failures accumulated silently. Key rotation is a security best practice; schema migration is routine engineering — each executed correctly by a team that had no reason to think about backups, because nothing in either workflow declared backups a stakeholder. Restorability is a property that decays under everyone else's good hygiene. Nothing in this story broke loudly; three independent failure modes (unreadable ciphertext, incompatible schema, blown time budget) grew quietly under a green light, and only an actual restore exercises the full chain — key, schema, storage, duration, runbook — that the checkmark never touched.

The 26-hour surprise is the same disease at the planning layer. The Disaster Recovery (DR) plan's four-hour RTO — the Recovery Time Objective, how long recovery may take — was written, not measured, and an unrehearsed number rots exactly like an unrehearsed procedure: data grows, the restore path changes, and the paper figure stays frozen at the moment of writing. The Recovery Point Objective (RPO — how much recent data you may lose) usually suffers the same fate. Numbers the business plans around deserve the same treatment as code: executed, timed, and re-validated on a schedule.

Diagrams

Problem — how the failure unfolds:

timeline
    title Three years of green checkmarks
    section Years 1 to 3
        Every night : Backup job exits zero : Dashboards stay green
        Never : Nobody restores anything
    section Quiet decay
        One day : Encryption key rotated away : Backups over 90 days unreadable
        Later : Schema drifts three migrations : Recent backups restore stale
    section The day it matters
        Hour 0 : Bad migration drops a table : Team reaches for the backups
        Hour 1 : Restore fails
        Hour 26 : Working restore finally lands : DR plan assumed four hours

Solution — where the guardrails go:

flowchart TD
    A["Nightly backups run as before"] --> B["Scheduled restore drill to<br/>a scratch environment"]
    B -.-> C["Validate row counts and<br/>schema against production"]
    B -.-> D["Time the full restore<br/>against the RTO"]
    C -.-> E["Rotated key caught while<br/>the old key is still<br/>recoverable"]
    C -.-> F["Schema drift caught at the<br/>next migration"]
    D -.-> G["26-hour reality surfaces<br/>in a drill, not an outage"]
    E --> H["Key rotation and<br/>migrations treated as<br/>backup-breaking changes"]
    F --> H
    G --> H

What good looks like

The lesson

You don't have backups; you have restores you haven't tested. Schedule restore drills like you schedule backups: restore to a scratch environment, validate row counts and schema, time it against the RTO, and treat key rotation and schema migration as backup-breaking changes.

Named concepts: DR testing, RTO/RPO validation, restore drills, monitoring the outcome not the process.

Real-world echoes

Transfer

Pick our most precious dataset. When did we last restore it, end to end, and how long did it take? Put the drill on the calendar before this meeting ends.

Make it a ticket: schedule the first restore drill for our most precious dataset — scratch environment, row-count and schema validation, timed against the RTO — and add "restore one pre-rotation backup" to the key-rotation runbook and "verify restore compatibility" to the migration checklist.

Related