Echo — AWS S3 us-east-1 (2017)

What happened

On the morning of February 28, 2017, an authorized engineer was debugging the Amazon S3 billing system in the us-east-1 region, following an established operational playbook. One input to a command was entered incorrectly, and it removed a far larger set of servers than intended. Those servers backed two of S3's core subsystems, and taking that much capacity offline at once meant both subsystems had to be fully restarted before S3 could serve requests again. For roughly four hours, object storage in us-east-1 — and the long list of AWS services and public websites that quietly depend on it — was effectively unavailable.

The mechanism

The removed capacity supported the index subsystem, which tracks the location and metadata of every object in the region, and the placement subsystem, which allocates storage for new writes and itself depends on the index. Below a threshold of available capacity, there was no graceful degradation — only a cold restart of both subsystems. And the restart was the real trap: S3 in us-east-1 had grown enormously and had not been fully restarted in years, so the safety checks that validate metadata integrity took far longer than anyone expected, and until they finished S3 stayed dark. The failure compounded because the recovery path had never been exercised at the system's current scale. It gets sharper still: the AWS Service Health Dashboard was itself hosted on S3, so the status page could not be updated to show the outage — it stayed green while S3 was down. The tooling meant to report the failure had an undocumented dependency on the thing that had failed.

Blast radius

S3 in us-east-1 is a foundational dependency for a huge slice of the internet, so the impact was correspondingly wide. Many well-known websites and applications degraded or went dark, and other AWS services in the region that lean on S3 were impaired for the duration. Public estimates of the economic damage ran into the hundreds of millions of dollars. The precise reach is hard to pin to a single number precisely because so many of the affected systems depended on S3 without their operators fully realizing it — the outage was, among other things, a live census of hidden dependencies.

The same-class lesson

Two transferable reflexes. First: a recovery path you have never exercised at real scale is not a capability — it's a rumor. The restart procedure existed on paper, but it had not been run against a system this large in years, so its true duration was unknown until the worst possible moment revealed it. Rehearse the restart, at scale, on purpose — the same instinct as killing a zone in a game day rather than waiting for the zone to die on its own. Second: a status page or recovery tool that rides on the very system it is meant to report cannot tell you when that system is down. Map what your incident tooling secretly depends on, and give every "temporary" single point of failure an owner and an expiry date before an external event audits them all at once.

Rhymes with

• Drill 23 — "Multi-AZ": an untested resilience claim is a rumor; only induced failure — a game day that actually exercises the failure and recovery path — turns the claim into a property.
• Drill 26 — The Failback That Split the Brain: the return trip (recovery, failback) is the unrehearsed half of the procedure, and hidden dependencies surface exactly when the recovery mechanism runs.

Named concepts: blast-radius controls, hidden dependencies in recovery tooling, recovery-path rehearsal at scale, single-point-of-failure analysis.