Echo — Meta BGP Outage (2021)

What happened

On October 4, 2021, a routine maintenance job on Meta's global backbone network issued a command meant to assess backbone capacity. Instead, the command took the entire backbone offline, disconnecting Meta's data centers from each other and from the internet all at once. With the data centers unreachable, Meta's authoritative DNS servers withdrew the BGP routes that advertised them to the rest of the world. From the internet's point of view, Facebook, Instagram, WhatsApp, and Messenger simply vanished — for roughly six hours.

The mechanism

Two designs, each sensible in isolation, composed into a catastrophe. First, a safeguard: an audit tool was supposed to block commands that could damage the backbone, but a bug let the fatal command through. Second, a health check: Meta's DNS servers were built to withdraw their BGP route advertisements whenever they lost their connection back to the data centers — a protection meant to steer users away from an unhealthy region. When the backbone dropped, that health check did exactly what it was designed to do and pulled the routes globally, so nothing could even resolve Meta's domain names. The deeper failure was structural: the tools, dashboards, and access systems engineers needed to diagnose and repair the outage ran on the very network that was down. The recovery path shared fate with the thing it was supposed to recover.

Blast radius

A family of apps used by billions of people went dark for roughly six hours — one of the largest internet outages ever recorded. Recovery could not be done remotely: with internal tooling unreachable, engineers had to travel to the data centers and work on hardware in person, and even physical access systems were reportedly affected, slowing every step of getting people to the machines that needed resetting. The damage was not just the outage itself but the multiplier — the fix took far longer because the ordinary way in was gone.

The same-class lesson

The transferable reflex is simple to state and easy to skip: your recovery path must not depend on the system it recovers. An escape hatch that runs on the primary network, authenticates through the primary identity system, or lives behind the primary tooling is not an escape hatch — it is a component of the failure. Build recovery access out of band and pre-built: a break-glass path that stays reachable when everything else is down, tested before you need it, so that when the main network vanishes you can still get in. This is drill 34 at planetary scale — the moment the people who can fix it are locked out by the very failure they are there to fix.

Rhymes with

• Drill 34 — Locked Out at the Worst Moment: the recovery depended on access the incident had already removed, and no pre-built exception path existed that didn't loop through the failure.
• Drill 25 — The Key That Expired on Saturday: automation and safeguards firing at the worst possible moment, with no owner and no out-of-band way around them.

Named concepts: out-of-band recovery access, break-glass procedures, recovery-path independence, blast-radius containment.