AIM-01-004-P · Primer — Parameters vs Hyperparameters

The idea in one sentence

Parameters are the numbers a model learns from data during training; hyperparameters are the numbers you choose beforehand to control how that learning happens.

Why it exists

Every model has two kinds of knobs, and they turn at different times by different hands. Think of baking bread: the oven temperature and the bake time are settings you dial in before anything goes in — those are hyperparameters. The final color, crumb, and rise are what the dough becomes in response — those are the parameters, and you don't set them by hand, the process produces them. We split the two because you cannot learn everything at once: something has to be fixed to give the learning a shape to work inside, and that "something" is the set of choices you make going in.

How it works

Training is an optimizer (usually gradient descent) nudging the parameters to reduce error on your data. For a linear model

\[ y = w \cdot x + b \]

the weight \(w\) and bias \(b\) are parameters — a small model might have two, GPT-scale models have hundreds of billions, and every one is fit by the algorithm, never typed in by a human. Hyperparameters sit one level up and govern that fitting: the learning rate (how big each step is, e.g. 0.01), the number of trees in a random forest (n_estimators=100), the regularization strength, the tree's max depth, the number of training epochs. You pick these before training starts; the optimizer never touches them. To choose them well you train multiple models — say learning rate 0.1 vs 0.01 vs 0.001 — score each on a held-out validation set, and keep the winner. So parameters are learned on the training set in one run; hyperparameters are chosen by comparing many runs.

See it — you choose hyperparameters; the optimizer learns parameters

flowchart TD
    YOU["YOU choose<br/>before training"] --> HP["Hyperparameters<br/>lr=0.01, max_depth=6"]
    HP --> TRAIN["Training loop<br/>gradient descent"]
    DATA["Training data"] --> TRAIN
    TRAIN --> PARAMS["Parameters learned<br/>weights w, bias b"]
    PARAMS --> VAL["Score on<br/>validation set"]
    VAL --> PICK["Compare many runs,<br/>keep best hyperparams"]
    PICK -.retry new lr.-> HP

The one gotcha

The trap is confusing which knob you're supposed to turn — and it quietly costs you weeks. You do not learn hyperparameters by gradient descent, and you do not hand-set parameters. Beginners try to "tune" their way to a better model by fiddling weights, or they let the training loop pick the learning rate and wonder why nothing converges. Worse, people tune hyperparameters against the test set, silently leaking it, then ship a model whose real-world score is far below the number they reported. Learn one set, choose the other, and never let the two roles blur.

See it (visualization plan)

A CompareScene/CardsScene split: left card "Parameters — learned" listing w, b, "175B weights," arrow from a TRAIN step; right card "Hyperparameters — chosen" listing learning_rate=0.01, max_depth=6, n_estimators=100, arrow from a YOU step. Then a MetricLineScene-style trio of validation scores for learning rates 0.1 / 0.01 / 0.001 with the middle one marked as the winner — showing hyperparameters chosen by comparing runs, not by gradient descent.

The one-line takeaway

You learn parameters; you choose hyperparameters.

Where you'll meet it

In scikit-learn, everything you pass to a constructor like RandomForestClassifier(n_estimators=100, max_depth=6) is a hyperparameter, while model.coef_ and model.feature_importances_ after .fit() are learned parameters. In PyTorch, nn.Linear layer weights are parameters updated by the optimizer, while the lr you hand torch.optim.Adam(lr=1e-3) is a hyperparameter. Tools like GridSearchCV, Optuna, and Ray Tune exist for exactly one job: searching the hyperparameter space for you.

Named concepts: parameters, hyperparameters