Interactive Lab · The two enemies of every model

Why can't you just make the error small?

Every model's error splits into two enemies, bias and variance, and they fight each other. Train a dozen models on a dozen fresh noisy datasets: if they all overlap, your model is stable but might all miss the truth the same way (bias). If they scatter wildly, each one chased its own sample's noise (variance). Push complexity to kill one and the other pops right back up. Drag the knob and watch the fits agree or thrash.

▶ The playground, drag complexity, watch the fits

true function (what we're chasing) 12 fits, one per fresh dataset average fit (bold) one sample's noisy dots

Expected test error, decomposed0.000

bias² 0.000 · systematically off variance 0.000 · unstable across data noise 0.000 · irreducible

Model complexity, polynomial degree 1

1 · a straight linea wild squiggle · 12

🧭 Where this fits

The bias-variance tradeoff is the daily bread of every ML practitioner: it's the reason behind regularization, early stopping, ensembling, and the size of every model you deploy. When a team debates “bigger model or more data?”, they are standing exactly on the curve you just traced.

The code that's actually running

🔮 Predict, then test

Crank complexity to max (12) and hit 🎲 Resample a few times, watch the faint fits thrash around the truth, a different tangle every time (that's variance). Now drop to degree 1: the fits barely move from resample to resample, but they all sit wrong in the same place (that's bias). Before you slide back: predict where total error bottoms out, and why it isn't at either extreme.

The takeaway

Your new mental model of the bias–variance tradeoff

Total error = bias² + variance + irreducible noise. The noise floor is baked into the data, no model, however clever, can beat it.
Simple models → high bias. Too stiff to bend into the truth, so every fit misses it the same way. Stable, but systematically wrong (underfit).
Complex models → high variance. Flexible enough to chase each sample's noise, so the fits swing wildly from one dataset to the next. It memorized the noise, not the pattern (overfit).
The best model minimizes the sum, not either term alone, the sweet spot sits between the extremes. And more data shrinks variance, which is exactly what lets you afford more complexity.

📖 Read the lecture notes Try: Overfitting Continue the journey →