Drill 5 — The Twice-a-Year Bug

Theme: Silent Failures · Format: 20-min drill · Difficulty: core

The story

A scoring model's performance dips noticeably for a few days every March and November. The first time, it's attributed to seasonality. The second time, "marketing campaign, probably." The third time, it has a nickname — "the March dip" — and a shrug.

⏸ Stop here and ask the room: what would you do about a recurring anomaly everyone has stopped investigating?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

What's suspicious about the timing?

Listen for: periodicity as a fingerprint — twice a year, specifically March and November, is a calendar signature, and strong answers start enumerating what changes on those dates: daylight-saving transitions, fiscal boundaries, certificate cycles. "Seasonality" that lands on the same two weekends every year is not weather; it's a mechanism.
Weak signal: treating each dip as an independent event — "March is just a weird month for applications" — accepting the correlation with the calendar without ever asking what, mechanically, changes on those dates.

What does a nickname for an anomaly tell you about the team's culture?

Listen for: the nickname marks the exact moment investigation stopped — normalization of deviance in one word. A named-but-unexplained anomaly is an unfiled incident report, and every shrug teaches the team that unexplained model behavior is acceptable furniture.
Weak signal: "it's just shorthand, every team has these" — missing that each tolerated mystery lowers the bar for the next one, and erodes the signal-to-noise of every future anomaly (is this dip the March dip, or something new hiding under its name?).

How do you investigate something that resolved itself weeks ago?

Listen for: forensics needs a paper trail — an anomaly registry with timestamps, retained raw data and window boundaries from the affected days, incidents plotted on a calendar to expose periodicity, and reproduction by simulating the clock change against the pipeline in staging.
Weak signal: "wait for it to happen again and look closely then" — without a registry, retention, or a hypothesis, the next occurrence will be exactly as unexplainable, and it's a six-month wait between experiments.

The reveal

Daylight-saving transitions. A feature pipeline mixed UTC and local timestamps; twice a year, "last 24 hours" windows were off by an hour, corrupting aggregates for a couple of days. Two years of dips — because a recurring anomaly with a nickname had become furniture.

Why this happens — the mechanics

The technical root is a bug that exists year-round but expresses twice a year. Mixing UTC (Coordinated Universal Time) and local timestamps is harmless for as long as the offset between them is constant — every "last 24 hours" window is shifted consistently, and consistent shifts cancel out. Only when the offset itself changes, at the daylight-saving boundary, do window edges land an hour off, double-counting one hour and dropping another. Test suites run at a fixed, mocked time; they never cross a DST (Daylight Saving Time) boundary, so the bug is structurally invisible to CI and only ever fires in production, on a schedule.

The organizational root is how cheaply plausible explanations close investigations. "Seasonality" and "marketing, probably" are unfalsifiable — they fit any dip, require no work, and carry no owner. And a self-resolving anomaly deprioritizes itself: by the time anyone has bandwidth to dig, the metric has recovered, the urgency is gone, and the raw data may already be past retention. Each occurrence individually loses the priority battle; nobody is assigned to the pattern, because the pattern only exists if someone has been writing the occurrences down.

This is normalization of deviance — the process by which an unexplained signal degrades into background noise through repetition. The cruel part is that the diagnostic was nearly free the whole time: incidents plotted on a calendar make twice-a-year periodicity leap off the page, and periodicity narrows the hypothesis space to a handful of calendar mechanisms. The calendar is a stack trace — but only for teams that log the crashes.

Diagrams

Problem — how the failure unfolds:

timeline
    title Two years of the March dip
    section Year 1
        March : DST shifts time windows : Model performance dips : Blamed on seasonality
        November : Dip returns : Blamed on marketing
    section Year 2
        March : Dip gets a nickname : Nobody investigates
        November : Engineer plots dips on a calendar : Periodicity matches DST : Root cause found - 4 dips late

Solution — where the guardrails go:

flowchart TD
    A["Performance dip observed"] --> B["Anomaly registry entry<br/>with timestamp — no shrug<br/>close"]
    B --> C["Quarterly review plots<br/>incidents on a calendar"]
    C --> D["Periodicity fingerprint<br/>matches DST weekends"]
    D --> E["Investigation finds mixed<br/>UTC and local timestamps"]
    E --> F["Fix: UTC everywhere,<br/>convert at the edges"]
    F --> G["Window-alignment check on<br/>time aggregates"]
    G -.-> AL["Alarm fires on the next<br/>shifted window"]

What good looks like

An anomaly registry with teeth: every unexplained metric wobble gets an entry — timestamp, metric, magnitude, owner — and closing an entry requires a mechanism. "Seasonality" is only an acceptable close with evidence attached (a matching prior-year pattern and a named driver).
A quarterly registry review that plots entries on a calendar, explicitly checking the periodic suspects: DST weekends, month-end, quarter-close, certificate and key expiries, billing cycles. Twenty minutes, once a quarter, catches two years of March dips at dip number two.
UTC everywhere in storage and computation, conversion only at presentation edges — enforced by a lint/CI check that flags local-time construction (datetime.now() without timezone, and friends) in pipeline code.
A window-alignment assertion on time-based aggregates: row counts per window compared against expectation, alarming when effective window boundaries shift by ±1 hour — this fires on the very next DST transition instead of corrupting silently.
A scheduled clock-change test in staging: simulate the spring and fall DST transitions (and leap days) against the feature pipelines before the real calendar does it in production.
MTTD (Mean Time To Detect) tracked for chronic issues: date of first occurrence versus date of root cause, reviewed as a team metric. "Four dips late" should be a number somebody has to say out loud.
Retention aligned to forensics: raw inputs and computed window boundaries for anomaly-flagged periods kept long enough to investigate after the fact — an anomaly registry is useless if the evidence expires before the review.

The lesson

A recurring anomaly deserves a root cause, not a name. Keep an anomaly registry with timestamps and plot incidents on a calendar — periodicity is a fingerprint (DST, month-end, quarter-close, cert expiry cycles). And: UTC everywhere, convert at the edges.

Named concepts: postmortem culture (no shallow "seasonality" closes), anomaly registries, MTTD for chronic issues.

Real-world echoes

Rhymes with the Microsoft Azure leap-day outage of February 29, 2012: certificate-generation logic mishandled the leap day and triggered a widespread, hours-long outage — a calendar-boundary bug that fired precisely on schedule. Rhymes too with the Zune bug of December 31, 2008: a leap-year date-handling loop froze the devices worldwide on the same day — the calendar as a synchronized trigger, again.

Transfer

What anomalies has our team nicknamed instead of solved? Pull the list — it's the backlog nobody wrote down.

Make it a ticket: Collect every nicknamed anomaly into a registry with dates, plot them on a calendar at the next team review, and assign each entry an investigation owner and a close-with-mechanism deadline.

Drill 4 — The Stale Score — sibling in silent data corruption: there time freezes, here the window slides — neither throws an error.
Drill 13 — The Day That Happened Twice, theme 2 — timestamp semantics corrupting data at the storage layer instead of the aggregation layer.
Drill 31 — The Postmortem That Named Names, theme 5 — the culture side: what teams do after incidents determines whether anomalies get mechanisms or nicknames.