AIM-07-002-P · Primer — Ridge Regression (L2)

The idea in one sentence

Ridge regression adds a penalty proportional to the sum of squared coefficients to the least-squares objective, shrinking all coefficients toward zero to trade a little bias for a large drop in variance.

Why it exists

Plain OLS goes haywire when features are correlated (multicollinearity) or when you have many features relative to rows: coefficients explode, flip signs, and swing wildly from sample to sample. Ridge is like a governor on an engine — it caps how large any coefficient can grow, so no single feature can dominate on the strength of noise. The result is a model that predicts more reliably on new data even though it fits the training data slightly worse.

How it works

We minimize RSS + lambda * sum(beta_j^2), where lambda >= 0 controls the penalty strength. At lambda = 0 you recover OLS; as lambda grows, all coefficients shrink smoothly toward (but never exactly to) zero. Crucially you must standardize features first, because the penalty is scale-sensitive — a feature measured in dollars vs thousands of dollars would otherwise be penalized differently. Worked example: with two nearly identical features x1 and x2, OLS might land on beta1 = +8000, beta2 = -7990 (a knife-edge that cancels out and is pure noise); ridge instead spreads the load to something stable like beta1 = 5, beta2 = 5. Ridge has a closed-form solution: beta = (X'X + lambda*I)^-1 X'y — the added lambda*I on the diagonal is literally why it's numerically stable when X'X is near-singular. You pick lambda by cross-validation, choosing the value that minimizes held-out error. Because it shrinks correlated features together rather than picking one, ridge keeps every feature in the model — good when you believe many features each contribute a little.

See it

flowchart TD
  A["Standardize<br/>features"] --> B["OLS loss<br/>plus L2 penalty"]
  B --> C["lambda controls<br/>shrinkage"]
  C --> D["Small lambda<br/>near OLS"]
  C --> E["Large lambda<br/>coeffs shrink"]
  D --> F["Pick lambda by<br/>cross-validation"]
  E --> F
  F --> G["Stable, lower<br/>variance model"]

In practice

Credit risk scoring with dozens of correlated bureau variables (utilization, balances, limits across accounts): ridge produces stable, sensible coefficients that don't whipsaw when the model is refit monthly on fresh data.

The one thing to remember

Ridge never zeros out a coefficient — it keeps all features but tames them; standardize your features first or the penalty silently favors large-scale variables.