Drill 4 — The Stale Score

Theme: Silent Failures · Format: 20-min drill · Difficulty: core

The story

A config refactor changes a cache TTL from 21m to 21d — a typo that passes review because the config diff was one line among forty. Velocity features (applications-per-device-per-day, card attempts per hour) freeze at their cached values. A fraud ring notices that repeat attempts aren't being counted anymore and cycles the same identities through, dozens of times.

⏸ Stop here and ask the room: losses are growing daily. What monitor catches this class of bug?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

Who tests config changes?

• Listen for: the honest answer — usually nobody, because config travels a lighter path than code — followed by the fix: config is a deploy. Review, canary to a small slice, post-change verification, plus schema validation with sanity bounds (a velocity-cache TTL (Time To Live — an expiry duration) of 21 days should fail validation before any human sees the diff).
• Weak signal: "the reviewer should have caught it" — blaming attention instead of building the system that catches one-line typos. A 40-line config diff will beat a tired reviewer every time; that's a property of reviewers, not of this reviewer.

What does "freshness" mean for each of our data products?

• Listen for: freshness as an SLI (Service Level Indicator) with a written number per product: velocity features are worthless beyond minutes, monthly aggregates tolerate days — and each carries a staleness alarm on the age of what's actually being served.
• Weak signal: "data should be fresh" with no numbers — if "how old is this value allowed to be?" has no written answer per product, there is nothing to alarm on, and freshness remains a vibe rather than an SLI.

How would you even discover the fraud ring is exploiting it?

• Listen for: effectiveness metrics on the control itself — the hit rate of repeat-identity checks, the distribution of velocity-feature values, repeat-attempt patterns — with alarms on sudden drops; plus a loss-rate backstop reviewed on days, not quarters. The strongest answers add the adversarial framing: fraud rings probe controls continuously, so their detection latency is shorter than yours.
• Weak signal: "we'd see it in chargebacks / the quarterly loss review" — that's the three-week failure mode from the story, restated as if it were a plan.

The reveal

Three weeks of losses. The tell was available the whole time: the hit rate of repeat-identity checks dropped to near zero — nobody had a freshness or effectiveness metric on the velocity features.

Why this happens — the mechanics

The technical root is that caches fail late and fail plausibly. The moment the TTL typo ships, nothing breaks: every read returns a valid-looking value — a stale count is still an integer, a frozen velocity is still a plausible velocity. The damage is the absence of updates, and absence emits no event to log, count, or alert on. Unlike a crash, which announces itself at the moment of the mistake, a staleness bug ships silently and expresses gradually, as reality drifts away from the cached snapshot one uncounted application at a time.

Organizationally, config changes carry the blast radius of code while traveling the paperwork of a comment fix. The pipeline that would have canaried, tested, and verified a code change often waves config through with a single review — and inside a 40-line diff, review attention diffuses. Every reviewer knows the experience of approving a mechanical-looking refactor; the one-character difference between 21m and 21d is exactly the kind of change human review is worst at catching and machine validation is best at.

What makes this drill sharper than a generic staleness story is the adversary. In fraud, a silent failure is not neutral decay — it is a vulnerability with a market. Fraud rings A/B-test controls constantly: retry the same device, the same card, the same identity, and watch what gets through. Their probing found the frozen counters within days; the provider's monitoring never found them at all. Against an active adversary, your MTTD (Mean Time To Detect — how long a failure goes unnoticed) is competing directly with theirs, and every day of the gap converts to losses on the book.

Diagrams

Problem — how the failure unfolds:

sequenceDiagram
    autonumber
    participant EN as Engineer
    participant RV as Config review
    participant CA as Feature cache
    participant FR as Fraud ring
    EN->>RV: TTL edit - 21m becomes<br/>21d (typo)
    RV->>CA: One line in a 40-line diff -<br/>approved (silent)
    Note over CA: Velocity features freeze<br/>at cached values (silent)
    FR->>CA: Probes - repeat attempts<br/>no longer counted
    FR->>FR: Cycles same identities<br/>dozens of times
    Note over FR: Losses grow for three weeks
    Note over EN,FR: Found by loss review,<br/>not by a monitor

Solution — where the guardrails go:

flowchart TD
    A["Config change: cache TTL edit"] --> B["Config treated as a<br/>deploy: review plus canary"]
    B --> C["Canary slice serves with<br/>the new TTL"]
    C --> D["Velocity features stop<br/>updating on the canary"]
    D --> E["Freshness SLI: maximum<br/>cache-entry age"]
    E -.-> AL["Staleness alarm fires in<br/>minutes"]
    D --> F["Repeat-identity hit rate<br/>drops toward zero"]
    F --> G["Effectiveness SLI on<br/>velocity checks"]
    G -.-> AL
    AL --> R["TTL typo reverted before<br/>full rollout"]

What good looks like

• Config schema validation with sanity bounds in CI (Continuous Integration): velocity-cache TTLs must fall within [1 minute, 48 hours]; anything outside fails the pipeline before a human reviews it. Typos become build failures, not incidents.
• Config deploys canaried like code: roll the change to a small slice first, with an automated before/after diff of the affected behavior metrics (cache-entry age, feature update rate) gating full rollout.
• A freshness SLI per cached or derived data product: alarm when the p95 age of served entries exceeds 2× the intended TTL — this fires within the hour on a 21m→21d typo, not in week three.
• An update-rate monitor on the velocity store: writes/refreshes per minute against a trailing baseline, with a zero-updates alarm inside 30 minutes — the direct detector for "features froze."
• Effectiveness SLIs on every fraud control: repeat-identity check hit rate and velocity-feature nonzero rate, alarming on a drop of more than 50% against the 7-day baseline. A control whose hit rate falls off a cliff is either fixed fraud or a broken control — both are worth a page.
• A daily loss and repeat-attempt anomaly report as the business backstop — quarterly loss review is an audit cadence, not a detection cadence.
• A monthly adversary-eyes exercise with the fraud team: "given our current monitors, which control could break silently — and how would we exploit it?" The fraud ring runs this exercise against you continuously; run it yourself first.

The lesson

Freshness is an SLI: every cached or derived data product carries a staleness alarm. And config changes are deploys — same review, same canary, same post-change verification as code.

Named concepts: freshness SLIs, configuration as code, canary releases for config.

Real-world echoes

Rhymes with Cloudflare, July 2019: a WAF (Web Application Firewall) rule change — treated as data rather than code — went global without the staged rollout that code deploys got, and a pathological regex spiked CPU worldwide. Different symptom, same root: a "harmless" config-class change that skipped deploy discipline.

Transfer

List our three most decision-critical cached values. Does anything alarm if they stop updating?

Make it a ticket: For each of those three values, write its freshness SLI (maximum allowed age, as a number) and wire a staleness alarm — and add TTL sanity bounds to the config validation pipeline while you're in there.

Related

• Drill 5 — The Twice-a-Year Bug — sibling in silent data corruption: there it's time-window misalignment, here it's frozen time.
• Drill 1 — The Model That Scored Everyone the Same — both serve valid-looking garbage with a 200; the missing monitor is on meaning, not machinery.
• Drill 36 — The $999 Pattern, theme 6 — the adversary's side of the table: fraud rings mapping exactly where your controls stop looking.