Echo — Slack Outage (2021)

What happened

On Monday, January 4, 2021 — the first workday back after the winter holidays — much of the world signed in to Slack within the same few hours, and the service buckled. Messages failed to send, channels wouldn't load, and for a large part of that morning Slack was down or badly degraded for users across the globe. It was not a bad deploy and not a single broken server: it was a sharp, synchronized surge in demand hitting infrastructure that could not scale into it fast enough, then a set of automated systems that made the hole deeper before they made it better.

The mechanism

Slack runs on AWS, with its network spread across many private networks (VPCs) stitched together by managed AWS Transit Gateways. Transit Gateways scale their capacity up in response to traffic — but gradually, over minutes, not instantly. That morning the load did not ramp; it stepped. The returning-to-work surge jumped past what the gateways were currently provisioned for, and they began dropping packets between Slack's own servers.

From there the failure cascaded. Packet loss turned into latency; latency stalled request threads; slow servers failed their health checks and were pulled out as if unhealthy, concentrating load onto the survivors. Meanwhile the automation turned on the incident: the fleet had scaled down over the quiet holiday week, so a large scale-up was needed exactly when the provisioning path was itself strained, and the dashboards the responders relied on were degraded by the same network trouble — so the people fighting the fire were partly flying blind. A component that scales gradually, a step-change in demand, and control loops reacting to distorted signals combined into an outage far larger than the original network blip.

Blast radius

Slack is used by millions of people and hundreds of thousands of organizations every day, and this landed on the single worst morning of the year for it to land — the first standup, the first message, the first shared file of the new work year, all stuck. The disruption ran for hours before capacity was restored and the network stabilized. Because Slack sits in the critical path of so many companies' daily coordination, the outage rippled well beyond Slack itself into every team that had come to depend on it.

The same-class lesson

A component that scales gradually cannot absorb demand that arrives as a step. The reflex this teaches is the one from the drills it rhymes with: model the ramp, not the average. Ask what happens the instant everyone arrives at once — the first-Monday login wall, the cache-flush-at-peak, the retry storm — and put the answer where a slow-scaling dependency (a gateway, a connection pool, a provisioning service) can be pre-warmed or shielded by backpressure and load shedding. Just as important: automation that reacts to demand is a control loop, and a control loop fed distorted signals (degraded monitoring, health checks that fire under load, scale-downs that just happened) can amplify a fault instead of damping it. Keep monitoring on an independent path, make scaling and health decisions robust to bad-data mornings, and rehearse the step-change, because staging's thin traffic never shows it.

Rhymes with

• Drill 21 — The Thundering Herd
• Drill 24 — The Autoscaler That Fed the Fire

Named concepts: thundering herd, cascading failure, control-loop design, backpressure, monitoring blind spots.