Echo — GitLab Database Deletion (2017)
What happened
Late on the night of 31 January 2017, a GitLab engineer was fighting a database problem on GitLab.com — replication had fallen behind, aggravated by a spike of spam load. Tired and working the live system directly, he ran a destructive removal command against what he believed was the empty data directory of the secondary database. It was the primary. Roughly three hundred gigabytes of live production data were deleted in seconds, and GitLab.com went down. The team then reached for their backups — and discovered, one recovery path at a time, that none of them worked. GitLab handled the aftermath with unusual transparency, live-streaming the recovery and publishing a detailed public post-mortem.
The mechanism
The deletion was human error under fatigue, but the incident that people remember is the second failure: recovery had never been exercised. GitLab had five separate backup and replication mechanisms, and in the moment of need not one of them was usable. The regular database dumps had been failing silently — a version mismatch between the dump tool and the server produced near-empty files, and the emails that were supposed to announce the failures were being rejected and never seen. Disk snapshots were not enabled where they were assumed to be. Other paths were empty or misconfigured. Every mechanism looked fine from the outside because nothing measured the only thing that mattered: whether a real restore would actually come back. The famous line from the post-mortem — that out of five techniques "none are working reliably or set up in the first place" — is the whole lesson in one sentence.
Blast radius
GitLab.com was down for hours while the team worked the recovery in public. They ultimately restored from a snapshot that happened, by luck, to have been taken about six hours earlier by an engineer setting up a staging environment — not from any of the designated backup systems. Because that snapshot was the freshest usable copy, roughly six hours of production data was permanently lost: issues, merge requests, comments, and new-user signups from that window — thousands of projects and comments, and hundreds of users, gone for good. The data itself was a modest fraction of GitLab; the reputational damage was the message that a well-run engineering org had five backups and zero tested restores.
The same-class lesson
A backup you have never restored is not a backup — it is a hope with a green checkmark. The transferable reflex: monitor the outcome, not the process. "The backup job exited zero" is cheap to measure and green every morning; "we can restore the real data, current-schema, within the time budget" is expensive to measure and is the only thing backups are for. Schedule restore drills the way you schedule backups — restore to a scratch environment, validate row counts and schema, time it against the RTO — and wire backup-failure alerts so a broken restore pages like the production incident it secretly already is. GitLab's silent, unread failure emails are the exact hazard: a decaying restore path under a dashboard that only ever measured the process.
Rhymes with
- Drill 32 — The Backup That Wouldn't Restore
Named concepts: restore drills, DR testing, backup verification, monitoring the outcome not the process.