AIM-07-001-P · Primer — Linear Regression, Revisited

The idea in one sentence

Linear regression models the conditional mean of a continuous outcome as a weighted sum of features, and each coefficient is the expected change in the outcome per one-unit change in that feature, holding all others fixed.

Why it exists

It exists because most relationships are 'roughly linear over the range we care about,' and a linear model gives you the cheapest possible object that is also fully interpretable: one number per feature. Think of it as fitting the single flat plane that comes closest to a cloud of points — no plane will pass through every point, so we settle for the one whose vertical misses are smallest overall. The magic isn't the fit; it's that the slope of that plane has a precise, actionable meaning.

How it works

We assume y = beta0 + beta1*x1 + ... + betap*xp + error, and choose the betas to minimize the sum of squared residuals (OLS). The 'holding all others fixed' clause is everything: beta1 is the effect of x1 after the linear influence of every other feature has been partialled out, which is why adding a correlated feature can flip a coefficient's sign. Worked example: predict house price ($k) from size (sqft) and bedrooms. Fit gives price = 50 + 0.12*sqft + 15*bedrooms. So each extra square foot adds $120 to the predicted price at a fixed bedroom count, and each extra bedroom adds $15k at fixed size. If you regress on size alone, bedrooms' effect gets absorbed into the size slope because bigger houses have more bedrooms. The classic OLS assumptions — linearity, independent errors, constant error variance (homoscedasticity), and roughly normal errors — matter mostly for the standard errors and p-values, not the point estimates. Check residual-vs-fitted plots: a funnel shape means heteroscedasticity (your confidence intervals lie), a curve means you're missing a nonlinear term. R-squared tells you variance explained, but a high R-squared on training data says nothing about generalization or causation.

See it

flowchart LR
  A["Features<br/>size, bedrooms"] --> B["Weighted sum<br/>beta0 + beta*x"]
  B --> C["Predicted mean<br/>of price"]
  D["Actual prices"] --> E["Residuals<br/>actual minus pred"]
  C --> E
  E --> F["Minimize sum<br/>of squares OLS"]
  F --> B

In practice

Real-estate pricing: estimate how much an extra bathroom or an additional 100 sqft is worth in a given ZIP code, giving appraisers a defensible per-feature dollar figure rather than a black-box number.

The one thing to remember

A coefficient is a conditional effect ('all else equal'), not a marginal one — with correlated features, its sign and size depend on what else is in the model, so never read a single coefficient in isolation.