CASE-22 · Case Study — Cloudflare 2019: The Regex That Ate the CPU

What happened

On July 2, 2019, a large slice of the web went dark for about 27 minutes. Sites behind Cloudflare returned errors worldwide — not because of an attack, but because of a single regular expression.

Cloudflare had pushed a new managed rule to its Web Application Firewall (WAF), meant to catch a class of malicious JavaScript. The rule contained a regex vulnerable to catastrophic backtracking: on certain inputs, the matching engine explores an exponentially growing number of possibilities, and CPU usage explodes. When the rule deployed globally, it drove CPU to 100% across Cloudflare's servers, so the proxies that handle a huge fraction of internet traffic could no longer serve requests. The fix was to disable the rule; service recovered quickly once they did.

See it — one rule, global CPU exhaustion

flowchart TD
  A["New WAF rule<br/>with a bad regex"] --> B["Deployed globally<br/>at once"]
  B --> C["Catastrophic<br/>backtracking on input"]
  C --> D["CPU to 100%<br/>on every server"]
  D --> E["Proxies can't serve<br/>-> global errors"]

The root cause

Two root causes combined. First, a regex with exponential worst-case behavior slipped through into production — a classic ReDoS pattern (nested quantifiers). Second, and more importantly, the rule was deployed everywhere, simultaneously, with no CPU budget or kill-switch to contain a misbehaving rule. A dangerous rule plus an all-at-once rollout equals a global outage.

The control that would have stopped it

Staged rollout is the decisive control: deploy the rule to a small fraction of traffic first and watch CPU before going global — a spike on 1% is an alert, not an outage. Pair it with resource limits (cap the CPU any single rule may consume, and abort matching that exceeds a time budget) and static analysis that rejects regexes with catastrophic-backtracking patterns before they ship. Cloudflare's postmortem committed to exactly these.

The blast radius

For ~27 minutes, a large portion of websites using Cloudflare were unreachable — the same property that makes a global edge network powerful (one change reaches everyone instantly) is what made the blast radius planetary. No data was lost or stolen; the cost was availability and trust.

How it unfolded

timeline
  title 27 minutes
  Jul 2 2019 13.42 UTC : WAF rule deployed globally
  Moments later : CPU spikes to 100% worldwide; errors surge
  14.00 UTC : Bad rule identified
  14.09 UTC : Rule disabled; traffic recovers
  After : Postmortem; staged rollout + limits adopted

The lesson

Availability is a security property, and your deploy pipeline is a blast-radius decision. Never ship a change to everything at once; earn each expansion with a healthy canary. Treat expensive computations (regex, parsers) as potential denial-of-service against yourself, and bound them with time and CPU limits so one bad input can't take down the fleet.

Where it rhymes

It rhymes with CrowdStrike's all-at-once update and every thundering-herd deploy story — and with the reliability discipline of canaries, error budgets, and buying each rollout step with evidence the last was safe.