Echo — Fastly (2021)

What happened

On June 8, 2021, a large slice of the internet went dark at the same instant. Amazon, Reddit, Twitch, PayPal, Spotify, the Guardian, the New York Times, Stack Overflow, and the UK government's own portal all failed to load, along with dozens of other major sites. The cause was not a cyberattack or a data-center fire — it was Fastly, a content delivery network (CDN) that sits in front of all of those sites. Weeks earlier, a Fastly software update had shipped a bug that did nothing at all until, on June 8, a single customer pushed a routine, perfectly valid configuration change that happened to meet the exact conditions the dormant bug needed to fire.

The mechanism

The failure had two ingredients separated in time. First, a software deployment in mid-May introduced a latent bug — code that was shipped, passed review, and then sat inert because nothing exercised the specific path it lived on. Second, weeks later, a customer made an ordinary configuration change that was valid on its face. That change was the trigger: it drove traffic down the exact path where the dormant bug waited, and the bug activated across Fastly's fleet at once. Neither event looked dangerous in isolation. The deploy caused no incident for weeks; the config change was the kind Fastly's customers make every day. The outage lived in the join between a forgotten code path and a later, innocent change that suddenly made it live.

Blast radius

The impact was broad and immediate. By Fastly's own account, a large majority of its network — roughly 85% — began returning errors the moment the bug fired, and every site fronted by those points of presence stopped serving traffic worldwide. The disruption was global and lasted on the order of an hour. What kept it from being far worse was the speed of the response: Fastly detected the disruption within about a minute, identified and disabled the offending configuration, and had the large majority of its network healthy again within roughly the hour. The blast radius was enormous, but the time-to-recover was short.

The same-class lesson

The transferable reflex: a change that is individually correct can still trigger latent behavior no one remembers is there — so a configuration change deserves the same guardrails as a code deploy. Teams instinctively gate code with staged rollout, canaries, and fast rollback, then wave config changes through as "just a setting." But config is what selects which code paths run, which means a config change can wake dormant code that has been dead for weeks. Treat it the same way: roll config out progressively rather than fleet-wide, watch the blast radius as it widens, and keep the ability to disable it in minutes rather than hours. Fastly's saving grace was not that it avoided the bug — it was that it could detect and pull the trigger back fast. That is the drill's whole point: dormant paths do not stay dormant, and the thing that reactivates them is usually a change that looked routine.

Rhymes with

• Drill 58 — The Flag That Forked Reality: a flag left live long enough to fall out of the team's mental model, so a later change reactivates behavior nobody remembers is loaded.
• Drill 28 — The Friday Flag: a valid-looking config change rides a release into production with no post-deploy verification and no progressive rollout to contain it.

Named concepts: latent-bug activation, config-change blast radius, staged rollout, fast detection and rollback.