BitBootcampBitBootcamp ← all explainers 🔄 Review ⚙ Settings

Drill 38 — The A/B Test of Two Cities

Theme: Feedback Loops & Adversaries · Format: 20-min drill · Difficulty: core

The story

A new underwriting flow is "A/B tested": region A gets the new flow, region B keeps the old one. After a month, A outperforms B by 18% — decisive. National rollout. Six months later the gains haven't materialized; performance is worse than baseline in several regions.

⏸ Stop here and ask the room: the test data was real and the math was right. What was wrong?

Run it — facilitator script

Timing: 2 min story · 10 min questions · 3 min reveal · 5 min transfer.

Emphasize the trap in the stop question — nothing was fabricated, no calculation was wrong. The room has to find a failure that lives entirely in the design.

"What exactly did this experiment randomize?"

Listen for: the sharp answer — nothing. Assignment was deterministic by geography, and the unit of randomization defines the experiment. With regions assigned wholesale, the effective number of independent units is two, not a hundred thousand applications.

Weak signal: "the sample was big enough" — confusing the number of applications with the number of independent assignment units. A million applications across two regions is still a comparison of two things.

"What differed between regions besides the flow?"

Listen for: a concrete confounder list — population mix, channel mix, fraud base rate, local economy, seasonality interacting with geography — and the recognition that every one of them rides along with treatment assignment, indistinguishable from the flow's effect.

Weak signal: trying to adjust it away after the fact ("control for region in the regression"). With assignment perfectly correlated with geography there is no leverage — adjustment cannot separate the flow from everything else that is region A.

"How should the rollout have been structured to produce a true read?"

Listen for: individual-level randomization within every region and segment, a pre-registered primary metric, balance checks before unblinding, staged rollout with guardrail metrics. Bonus: someone names when geo-experiments are legitimate — many geo units, switchback or matched-pair designs, cluster-aware inference.

Weak signal: "run it longer" or "add more regions" without changing the assignment unit. More duration just collects more confounded data with more confidence.

The reveal

Regions A and B differ in population, channel mix, and fraud base rate. The experiment measured region A vs. region B, with the flow as a confounded passenger. The 18% was demographics wearing a treatment's clothes.

Why this happens — the mechanics

Randomization has one job: make treatment assignment statistically independent of everything else about the unit. Assign by geography and you've done the opposite — treatment is now perfectly correlated with every attribute of region A: its population, its channel mix, its fraud base rate, its local labor market. The measured 18% is the flow's effect plus the sum of all those differences, in unknown proportions, and no after-the-fact adjustment can separate terms that were never allowed to vary independently. This is confounding-by-assignment: the design, not the data, destroyed the answer.

The trap is compounded by a precision illusion. Each region contributes thousands of applications, so the per-region estimates carry tiny standard errors — the dashboard shows a confidently measured number. But the effective count of independent assignment units is two: region A and region B. Statistically, the team ran an n=2 study and read it with n=100,000 confidence. The math was right; it was answering a different question — "is region A different from region B?" — to which 18% was a perfectly good answer.

The six-month collapse is then mechanical. National rollout breaks the accidental correlation that manufactured the effect: the whole book isn't region A's population, so the demographic component of the 18% evaporates — and any segments where the new flow is genuinely worse now show through. The gains didn't "fail to materialize"; they never existed as a treatment effect in the first place.

Diagrams

Problem — how the failure unfolds:

flowchart TD
    A["Region A gets the new flow"] --> C["Compare after a month"]
    B["Region B keeps the old<br/>flow as control"] --> C
    D["Regions differ:<br/>population, channel mix,<br/>fraud base rate<br/>(silent)"] --> C
    C --> E["A wins by 18% —<br/>demographics wearing a<br/>treatment's clothes"]
    E --> F["National rollout"]
    F --> G["Gains never materialize;<br/>worse than baseline in<br/>several regions"]

Solution — where the guardrails go:

flowchart TD
    A["Proposed experiment"] --> R["Experiment design review<br/>before launch"]
    R --> B["Individual-level<br/>randomization within every<br/>segment"]
    B --> C["Treatment group: new flow"]
    B --> D["Control group: old flow,<br/>same population"]
    C --> E["Compare on the pre-<br/>registered metric"]
    D --> E
    E -.-> M["Per-segment balance checks<br/>confirm the read"]
    E --> F["True effect measured;<br/>rollout matches the test"]

What good looks like

The lesson

An experiment is defined by its unit of randomization, not by its dashboard. If assignment correlates with anything real — geography, time, channel, device — you measured that thing too. Randomize at the individual level within every segment, pre-register the metric, and treat geography-split "tests" as the anecdotes they are.

Named concepts: randomization units, confounding, experiment design review, guardrail metrics.

Transfer

Pull our last three "experiments." For each: what was actually randomized? Would the conclusion survive a skeptical re-read?

Make it a ticket: Audit the last three experiments against a one-page checklist — randomization unit, confounder list, pre-registered metric — and re-flag any live decision that rests on a geography-split read.

Related