Echo — Zillow Offers (2021)

What happened

Zillow Offers was Zillow's iBuying business: an algorithm forecast what a home would be worth, Zillow bought the home at a price derived from that forecast, did light work, and tried to resell it for a margin. In early November 2021 Zillow announced it was shutting the unit down entirely. The pricing model had been systematically overpaying — buying homes at prices the company could not resell for a profit — and by the time leadership pulled the plug, Zillow was sitting on thousands of houses worth less than it had paid. Winding down meant taking a write-down of more than half a billion dollars and laying off roughly a quarter of the company's staff. A model that looked accurate enough to bet the business on did not survive contact with the market it was making bets in.

The mechanism

The model's job was to forecast future home prices, and forecasts that look good against historical data are not the same thing as forecasts that hold up when you are spending real money at scale on them. Two things compounded. First, an offline-to-online gap: the pricing model was validated against past sales, but in production it drove actual purchase decisions in a fast-moving housing market whose conditions had shifted, so the error that was tolerable in backtest became dollars of loss on every house. Second, Zillow was acting on its own predictions at volume — the model didn't just observe the market, it bought into it, and the faster the business scaled buying, the faster small, correlated pricing errors turned into a large inventory of overpriced homes. The company itself said it could not forecast prices accurately enough, and that the swings were larger than the model had anticipated. Nothing "broke." The model did exactly what its offline evaluation implied it would; the offline evaluation just didn't measure the thing that mattered — realized profit on homes it had actually bought.

Blast radius

The impact was large and concrete. Zillow shut down the entire Zillow Offers segment, took write-downs reported at more than $500 million, and cut roughly 25% of its workforce — on the order of two thousand jobs. It was left holding thousands of homes it now had to sell at a loss or offload in bulk. Beyond the direct financial hit, a public, well-capitalized company exited a flagship line of business it had staked significant strategy on. The damage was not a brief outage measured in minutes; it was a strategic bet that a validated-looking model quietly turned into hundreds of millions of dollars of loss over months.

The same-class lesson

The transferable reflex: a model that wins on historical, offline data has proven a hypothesis, not a result — the verdict only comes from the business metric, measured live, on the population the model actually acts on. Zillow's model wasn't scored on the number that mattered (realized profit on homes it bought) in the world it operated in (a shifting market it was also buying into). That is exactly Drill 93's trap: the offline metric was precise, cheap, and available now, while the true objective — dollars, months later, on houses you have already purchased — was none of those, so the proxy quietly became the target. And it is Drill 35's trap too: when a model acts on its own predictions at scale, its errors don't stay small and independent — they compound through the very decisions the model is driving. The discipline is to gate scale-up on a live business-metric guardrail: roll out on limited volume, measure realized profit and loss against the model's forecast, and let reality — not the backtest — decide whether to pour more capital in. When offline and online disagree, the evaluation is wrong, not the world.

Rhymes with

• Drill 93 — The Metric That Lied: a challenger wins decisively on the offline metric, then loses in dollars in production, because the offline number measured the wrong thing on the wrong population.
• Drill 35 — The Model That Ate Its Own Tail: a model that acts on its own outputs amplifies its own bias, so small errors compound through a feedback loop instead of averaging out.

Named concepts: offline-online metric mismatch, acting on your own predictions, forecast error compounded at scale, business-metric guardrail before full rollout.