Echo — Cloudflare WAF Regex (2019)

What happened

On July 2, 2019, Cloudflare — which sits in front of a large share of the internet's web traffic — pushed a routine update to its Web Application Firewall (WAF). The change added new managed rules meant to detect and block malicious inline JavaScript. Cloudflare's WAF rules deploy through a fast path built for responding to emerging threats, so the update went out to the entire global network at once. Within seconds, CPU on the machines that proxy HTTP and HTTPS traffic climbed to 100% worldwide, and visitors to sites behind Cloudflare began receiving 502 errors. Engineers traced the spike to the new deployment and used a global WAF "kill switch" to disable the rules; traffic recovered a short time later.

The mechanism

The root cause was a single regular expression in one of the new rules. It contained a sub-pattern of the shape .*(?:.*=.*) — nested, unbounded quantifiers with no anchoring. On certain inputs a regex engine that backtracks (like PCRE) explores an exponentially growing number of ways to match, a failure mode known as catastrophic backtracking (a form of regular-expression denial of service, ReDoS). A match that should take microseconds instead consumed a whole CPU core, and because every request through the proxy was evaluated against the rule, the fleet saturated almost instantly.

The organizational root is what turned a bad regex into a global outage: the rule was rolled out everywhere simultaneously, with no staged canary to catch it on a small slice first, and no CPU/time budget on regex execution to cut off a runaway match before it starved the box. The very deploy path designed for speed — so security rules could ship fast — was the path with the least safety around it.

Blast radius

For roughly 27 minutes, a large fraction of the websites that rely on Cloudflare returned 502 errors to their visitors — a broad, simultaneous, global failure rather than a localized one. Cloudflare's own dashboard and API were affected as well. Cloudflare later described this as one of its worst outages in years. The company published a detailed public postmortem the same week and followed with engineering changes: staged, progressive rollouts for WAF rules; CPU and time limits on regex evaluation; and a move toward regex engines that match in linear time and cannot backtrack catastrophically.

The same-class lesson

The transferable reflex is blast-radius control: no change is too urgent to earn a canary and a reachable kill switch. A rule, a flag, or a config value that ships to 100% of production in a single step has no seam where a problem can be caught before it is everyone's problem. The fix is progressive delivery — roll the change to a small slice, watch what production actually does, then widen — plus a hard resource budget so one pathological input can't take down the process, and a kill switch an operator can hit in seconds. This is exactly the shape of Drill 28 (The Friday Flag): a change that rode a fast, low-friction path straight into global production with no post-deploy behavioral check standing between it and every user. Speed of deploy is never a license to skip the blast radius.

Rhymes with

• Drill 28 — The Friday Flag: a config change that reaches production through a fast path with no canary and no post-deploy behavioral verification.
• Drill 58 — The Flag That Forked Reality: flag and rule hygiene at release scale, where the deployment mechanism itself is the hazard.

Named concepts: progressive delivery, blast-radius control, kill switch, change management.