Drill 97 — The Backtest That Cheated

Theme: Evaluation & Validation · Format: 20-min drill · Difficulty: advanced

The story

A systematic credit desk builds a new signal: a gradient-boosted model that scores every corporate issuer for the probability it defaults inside the next twelve months, and the desk shorts the names that score worst. The quant who owns it runs the backtest over nine years of history, and it is gorgeous — AUC of 0.89 on the held-out rows, a simulated Sharpe of 2.1, a smooth equity curve that barely draws down through 2015 and 2020. The readout deck writes itself. Risk signs off, the model clears model-validation review, and the strategy goes live with real capital in the book.

For the first few weeks the P&L is noisy but fine, and nobody worries. Then the flat weeks stop being noise. Three months in, the desk's realized information coefficient is hovering near zero; the live AUC on the names it actually traded is 0.61, barely better than a coin flip. The strategy that printed a 2.1 Sharpe in simulation is running a live Sharpe around 0.2, and the drawdown the backtest never showed is happening in real dollars. Nobody shipped a bug. The model in production is byte-for-byte the model that was backtested, scoring the same features with the same weights.

The room's first instinct is that the market "regime changed," or that the edge was arbitraged away the moment it went live. Neither is true. The 0.89 was never real. The backtest was measuring a model that could see the future, and production is the first time it has been asked to work with only the past.

⏸ Stop here and ask the room: the live model is identical to the backtested one, on the same features — so how did the backtest score 0.89 while production scores 0.61?

Run it — facilitator script

Timing for a 20-minute session: 2 min story · 10 min staged questions · 3 min reveal · 5 min transfer.

The backtest and production run the same code on the same features. Where could the numbers diverge?

• Listen for: the backtest and production differ in what information each row had access to. In the backtest, a feature or label can be built from data that did not exist at that timestamp — normalization statistics computed over the whole window, a label horizon that reaches past the split, restated fundamentals the vendor overwrote after the fact. That is look-ahead bias: the historical row "knows" something no live row can know. Production is the control experiment the backtest never ran.
• Weak signal: "the market regime changed" or "alpha decays after publication" — reaching for a story about the world when the gap is inside the evaluation.

Name the concrete ways the future can leak into a nine-year panel.

• Listen for: (1) whole-window normalization — z-scoring a feature using the mean and standard deviation of the entire history, so every row is scaled by statistics that include its own future; (2) a random shuffle split instead of a time split, so rows from 2021 train a model that is then scored on rows from 2016; (3) restated / as-reported data — using the fundamentals or ratings as they look now rather than as they were known at time t, when a filing lands weeks late and gets restated later still; (4) label overlap — a twelve-month default label straddles the train/test boundary, so the test period leaks backward through the label window.
• Weak signal: "we used cross-validation, it's fine" — random K-fold on time-series data is exactly the shuffle that lets the future train the past.

How should this have been evaluated so the backtest number meant something?

• Listen for: reconstruct the data point-in-time (as-of joins; only information stamped as known at t), evaluate walk-forward (train on the past, test on the next slice, roll the origin), keep a final out-of-time holdout that is the most recent contiguous period and is touched exactly once, and purge and embargo around the label horizon so no training sample overlaps the test window. The offline number is only trustworthy if every row was built with the information a live row would actually have.
• Weak signal: "add more data" or "get a bigger test set" — more rows built the same leaky way just makes the wrong number more precise.

The reveal

Three leaks, stacked. The feature pipeline z-scored every input using the mean and standard deviation of the full nine years, so each row was silently normalized against its own future. The split was a random 80/20 shuffle, so the model trained on 2022 and was scored on 2017. And the fundamentals came from the vendor's current table — restated earnings, back-dated ratings — not the as-reported values known on the day. Individually each was a small sin; together they handed the backtest a model that could see forward, and the 0.89 was the score of that impossible model. The team rebuilt the evaluation point-in-time, switched to walk-forward with a purged, embargoed split and a single out-of-time holdout, and the "edge" mostly vanished on paper — which is exactly what production had already told them. The next candidate signal cleared a backtest of 0.66 that held at 0.64 live, because for the first time the backtest and reality were measuring the same thing.

Why this happens — the mechanics

A backtest is a claim about the past that stands in for a prediction about the future, and the claim is only honest if every row is reconstructed with exactly the information a live system would have had at that instant — no more. Look-ahead bias is any way that "no more" gets violated: the future sneaks into a row through a statistic computed over the whole window, through a split that lets later data train the model, through a data vendor that overwrites history with restatements, or through a label whose horizon reaches across the boundary. None of these throw an error. They all make the number better, which is why they survive review — a leaky backtest looks like a triumph, not a bug, and the incentive gradient points toward shipping it.

The reason time-series finance is a minefield here is that the samples are not independent and the world is revised after the fact. A twelve-month default label for an issuer observed in March overlaps the label for the same issuer observed in April, so a naive shuffle puts near-copies of a test row into training. Fundamentals arrive weeks after quarter-end and are restated quarters later, so the "value" of a feature at time t depends on when you look. Standard machine-learning validation assumes i.i.d. rows and immutable data; both assumptions are false here, and every one of the guardrails below exists to restore a single invariant — a row may only be built from data whose knowledge-date is on or before t.

Diagrams

Problem — how the future leaks into the backtest:

sequenceDiagram
    autonumber
    participant Q as Quant
    participant FS as Feature Store
    participant BT as Backtester
    participant PM as Live Desk
    Q->>FS: build z-score features
    Note over FS: mean and std from<br/>FULL history — future leaks
    FS->>BT: hand over labeled panel
    Note over BT: random shuffle split<br/>future rows train the past
    BT-->>Q: backtest AUC 0.89
    Q->>PM: ship signal to the desk
    PM--xPM: live AUC 0.61 — edge gone
    Note over Q,PM: caught in production,<br/>never in the backtest

Solution — an evaluation that can only see the past:

flowchart TD
    A["New signal / model"] --> B["Point-in-time panel<br/>(as-of joins only)"]
    B --> G1["Assert knowledge_date<br/>≤ as_of date"]
    G1 -.-> AL["Leakage check fails<br/>the build"]
    B --> C["Walk-forward eval<br/>(rolling origin)"]
    C --> G2["24 folds · report<br/>per-fold OOS spread"]
    G2 -.-> AL
    C --> D["Purge + embargo<br/>around label horizon"]
    D --> G3["Drop overlaps · embargo<br/>a 12-month gap"]
    G3 -.-> AL
    C --> E["Out-of-time holdout<br/>(last 20%, touched once)"]
    E --> G4["Promote only if OOT<br/>Sharpe holds"]
    G4 -.-> AL

What good looks like

• Point-in-time correctness with as-of joins: every feature row carries a knowledge_date, and the build asserts knowledge_date ≤ as_of for every join; a CI check fails the build if any feature references a value stamped after t. Lag fundamentals by their real reporting delay (e.g. hold each quarterly filing invisible until its actual availability date), and read as-reported vendor snapshots, never the restated current table.
• Walk-forward / rolling-origin evaluation, never a single number: train on [t0, t], test on (t, t+1 month], roll the origin forward — at least 24 folds across the history. Report the distribution of the out-of-sample metric across folds (median and spread), not one pooled AUC. A signal whose per-fold OOS AUC swings from 0.75 to 0.51 is not a 0.63 signal; it is an unstable one.
• Out-of-time holdout, never a random shuffle: quarantine the most recent contiguous slice (e.g. the last 20% of the timeline by date) and touch it exactly once, at promotion. Random K-fold and train_test_split(shuffle=True) are banned on time-series panels — a lint rule in the repo flags them.
• Purge and embargo around the label horizon: when a fold's test window opens, purge every training sample whose label window [event_time, event_time + 12 months] overlaps that test window, and embargo a gap after the test window (≈ the label horizon, so ~12 months here, or ~1% of samples) before training resumes — so serially-correlated, overlapping labels cannot leak across the boundary.
• A backtest-vs-live gap SLI: after go-live, track live AUC (or IC) against the backtested value on the same names; if the live metric underperforms the backtest by more than 0.05 AUC for two consecutive months, the signal is auto-flagged for a leakage post-mortem, not left to bleed.
• A promotion rule in writing: a signal is promoted only on its out-of-time, point-in-time, walk-forward number. The pooled shuffle-split backtest is not admissible evidence.

The lesson

A backtest is only worth the information discipline behind it: a row may use nothing that wasn't knowable at its own timestamp. Reconstruct point-in-time, evaluate walk-forward on an out-of-time holdout, and purge the label horizon — or you are not backtesting a strategy, you are measuring how well a model can cheat.

Named concepts: look-ahead bias, point-in-time correctness, walk-forward validation, out-of-time holdout, purge and embargo.

Real-world echoes

• The quant-industry "backtest overfitting" literature. Look-ahead bias is the best-documented failure mode in systematic finance: signals that show extraordinary simulated Sharpe ratios and then die in production because the backtest used restated data, whole-window normalization, or shuffled splits. Marcos López de Prado's work formalized the fixes this drill teaches — purged K-fold and the embargo period around the label horizon — precisely because so many "profitable" backtests were leaking the future through overlapping labels. It is a whole class of self-inflicted wound rather than one named blow-up, which is what makes it worth drilling.

Transfer

For your most important model, can you prove that every feature row was built only from data knowable at that row's timestamp — and that your test set is a contiguous future period, not a random shuffle? "We used cross-validation" is the weak answer; name the as-of join and the walk-forward split that make the backtest honest.

Make it a ticket: for the highest-stakes time-series model, add a point-in-time leakage assertion (knowledge_date ≤ as_of) to the feature build, replace any random split with a walk-forward, out-of-time holdout, and purge + embargo the label horizon — then re-run and record how much of the "edge" survives.

Related

• Drill 93 — The Metric That Lied — the sibling failure: the offline number was real but didn't mean what the room assumed about production.
• Drill 87 — Train-Serve Skew — the lifecycle cousin: the model was fine, the served features diverged from the trained ones.
• Drill 4 — The Stale Score — another offline-vs-reality gap, where the score stopped matching the world it was scoring.
• Pattern #132 — SLO / Error-Budget Policy — the analog of "decide on the metric that matters, measured the way production actually runs."