Echo — HealthCare.gov (2013)

What happened

On October 1, 2013, the U.S. federal health-insurance marketplace, HealthCare.gov, opened for business — the online front door to coverage under the Affordable Care Act for the roughly three dozen states that relied on the federal exchange. It launched to the entire country at once. Within minutes of going live it was effectively overwhelmed: users could not create accounts, pages hung or errored, and almost no one could complete an enrollment. By later accounts drawn from internal notes, only a handful of people — by some reports as few as six — managed to enroll on the first day. What was meant to be a landmark launch became a months-long public rescue effort.

The mechanism

The site was a big-bang go-live with no realistic end-to-end test at production scale. It was built by a sprawl of contractors integrating many moving parts — an identity/account system, a data hub talking to other federal agencies, and the plan-shopping experience — and those parts were stitched together late, with little time to exercise the whole thing under real load. The architecture made it worse: users were required to create an account before they could browse plans, which funneled the entire arriving nation through a single registration bottleneck at the very first step. When the account system stalled, frustrated users refreshed and retried, and those retries amplified the load — a thundering herd that turned a slow system into a stuck one and cascaded across the integrated components. The first genuine load test was launch day itself.

Blast radius

The impact was national and highly visible. For weeks the site was barely usable; the vast majority of people who tried in the opening days could not get through, and the failure became a lasting symbol of botched government technology. The response was a "tech surge" — an outside team of engineers brought in to triage and rebuild the worst bottlenecks — and it took roughly two months of intense work before the site was reliably functional around December 2013. The program's costs ran into the hundreds of millions of dollars. The rescue did eventually work: by the close of the first open-enrollment period in spring 2014, on the order of eight million people had signed up through the marketplaces. The damage was to trust, time, and money — not, in the end, to the policy's reach.

The same-class lesson

The first real test cannot be production. A launch is an experiment, and you do not learn whether a system survives real traffic by pointing real traffic at it on day one and hoping. The transferable reflex: rehearse at production scale before go-live — synthetic load at the volumes you actually expect, exercising the whole path end to end, not each component in isolation. Do not open the doors to everyone at once; roll out in phases or a canary, so the first slice of real users reveals the bottleneck while it is still small enough to fix. And design for the herd: put backoff with jitter on the client, load shedding and queues on the server, and remove single-file chokepoints like "you must register before you can even look." A big-bang launch with no scale rehearsal isn't ambition; it's an untested bet placed with the whole country's traffic.

Rhymes with

Drill 54 — The Big Bang: the entire cutover concentrated into one all-at-once go-live, with the first real test arriving in production. Drill 21 — The Thundering Herd: a synchronized surge of demand plus retries with no backoff, saturating a shared dependency and cascading into a full outage.

Named concepts: big-bang launch, production-scale load testing, thundering herd, phased rollout / canary, cascading failure.

Timeline

timeline
    title HealthCare.gov launch, 2013
    Before : Federal marketplace built by many contractors : Parts integrated late, never load-tested end to end
    Oct 1, 2013 : Site opens to the whole country at once : Overwhelmed within minutes : Almost no one can enroll
    Opening days : Registration bottleneck stalls : Retries amplify the load : Failures cascade across systems
    After : A tech-surge team is brought in : Largely functional by December : About eight million enroll by spring 2014