AIM-01-005-P · Primer — Model Capacity

The idea in one sentence

A model's capacity is the richness of the patterns it can represent — the size of the set of functions it is even able to fit, before you ask whether fitting them is a good idea.

Why it exists

Picture drawing a curve through a scatter of dots. A stiff straight ruler can only ever make a line; a bendy wire can snake through every single point. Capacity is how "bendy" your model is allowed to be — its flexibility. We need the word because the same problem can be attacked with a stiff model or a flexible one, and the choice of flexibility, not just the data, decides what the model can and cannot express. Formally, that flexibility is the model's hypothesis space: the whole menu of shapes it is allowed to choose from. Low capacity is a short menu; high capacity is a vast one.

How it works

You set capacity with a few concrete knobs: the degree of a polynomial, the depth of a decision tree, the number of layers and parameters in a neural network, or the k in k-nearest-neighbors (small k = high capacity). For a polynomial, the whole hypothesis space is captured by one central formula — a sum of powers of \(x\) up to degree \(d\):

\[ f_d(x) = \sum_{j=0}^{d} \beta_j x^j = \beta_0 + \beta_1 x + \beta_2 x^2 + \cdots + \beta_d x^d, \]

where \(d\) is the capacity dial: each increase in \(d\) adds another free coefficient \(\beta_j\) and another degree of bend. (For k-nearest-neighbors the same dial runs the other way — the prediction is an average over the \(k\) nearest neighbors, \(\hat{y}(x) = \frac{1}{k}\sum_{i \in N_k(x)} y_i\), so a small \(k\) means each prediction is swayed by very few points, i.e. high capacity.) Work a real example: take ten points that roughly follow a gentle curve with a little noise. A degree-1 line, \(f_1(x) = \beta_0 + \beta_1 x\), is too stiff to follow the curve and misses badly — that's underfitting, too little capacity. A degree-3 curve, \(f_3(x) = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3\), bends just enough to track the real trend and ignore the noise — capacity matched to the problem. A degree-9 polynomial has enough freedom to pass exactly through all ten points: training error hits zero, and the curve wiggles wildly between them. With ten points and ten free coefficients \(\beta_0, \dots, \beta_9\), the degree-9 fit \(f_9(x) = \sum_{j=0}^{9} \beta_j x^j\) has exactly enough parameters to solve the interpolation problem exactly, so the training error collapses to zero:

\[ \begin{aligned} \text{Error}_{\text{train}} &= \frac{1}{n}\sum_{i=1}^{n}\bigl(y_i - f_9(x_i)\bigr)^2 \\ &= \frac{1}{10}\sum_{i=1}^{10}\bigl(y_i - y_i\bigr)^2 \\ &= 0. \end{aligned} \]

As you turn the capacity dial up, training error \(\text{Error}_{\text{train}}(d)\) keeps falling toward zero, but error on unseen data \(\text{Error}_{\text{test}}(d)\) dips and then climbs — the sweet spot is the bottom of that U, not the far right.

See it — turning the capacity dial: underfit to just-right to overfit

flowchart LR
    D["10 noisy points<br/>on a gentle curve"] --> L
    D --> M
    D --> H
    L["Degree 1<br/>stiff ruler"] --> LR["Misses the curve<br/>UNDERFIT"]
    M["Degree 3<br/>right bend"] --> MR["Tracks trend<br/>JUST RIGHT"]
    H["Degree 9<br/>very bendy"] --> HR["Train error to 0<br/>wiggles wildly<br/>OVERFIT"]

The one gotcha

More capacity is not more skill; it's more rope to overfit with. A bigger model does not automatically understand more — it just has more room to memorize the training data, quirks and noise included. That degree-9 curve with zero training error isn't smarter than the degree-3 one; it's confidently wrong on the next point. The goal is never maximum capacity — it's enough capacity to capture the real pattern, and no more than your data can support.

See it (visualization plan)

A MetricLineScene with x-axis = model capacity: a blue line for training error sliding toward zero, and the real star — an unseen-data error line that dips to a minimum and then climbs as capacity grows past the sweet spot. Pair it with a TilesScene of the three fits — Degree 1 "underfit", Degree 3 "just right", Degree 9 "overfit" — and a BigNumberScene counting the degree-9 training error down to a suspicious 0.

The one-line takeaway

More capacity is more rope, not more skill.

Where you'll meet it

Every capacity dial you'll touch: max_depth and min_samples_leaf in scikit-learn trees, degree in PolynomialFeatures, layer widths and counts in PyTorch, n_estimators and max_depth in XGBoost, and n_neighbors in kNN. Turning any of them up feels like progress — until the validation score turns around. The failure mode is its own video (Overfitting, AIM-01-021-P), and the fixes are too: regularization, cross-validation, and early stopping.

Named concepts: model capacity, hypothesis space