ML-01-013-P · Primer — Evaluating Regressors: MAE, RMSE, and R2

The idea in one sentence

Once a regression model predicts numbers, you measure how wrong those numbers are (MAE and RMSE) and how much of the pattern it captured (R2), so you know whether to trust it.

Why it exists

A classifier is either right or wrong on each row, but a regressor predicts a continuous number — so "wrong" comes in degrees. If the true house price is $300k and you predict $305k, that is a good miss; predicting $180k is a bad one. You need a single number that summarizes "how far off am I, on average?" and another that answers "is my model actually better than just guessing the mean?" MAE and RMSE answer the first question in the units of your target (dollars, degrees, minutes). R2 answers the second on a clean 0-to-1 scale, where 1.0 is perfect and 0.0 means "no better than always guessing the average." Together they turn a vague "seems okay" into evidence you can defend.

How it works

Every regression metric starts from the residual — the gap between the true value and the prediction, y_true - y_pred, for each row. MAE (mean absolute error) takes the size of each gap, ignores its sign, and averages them: it is the typical miss in the target's own units, and every mistake counts equally. RMSE (root mean squared error) squares each gap before averaging, then takes the square root — squaring makes big misses hurt much more than small ones, so RMSE is the metric to watch when one huge error is worse than several tiny ones. R2 (the coefficient of determination) rescales things: it compares your model's squared error to the error of a dumb baseline that always predicts the mean of y, giving 1.0 for a perfect fit and 0.0 for "no better than the mean." Here we fit a LinearRegression, predict on the same tiny dataset, and read all three at once.

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score

# Tiny inline dataset: hours studied -> exam score
X = np.array([[1], [2], [3], [4], [5], [6]])
y = np.array([52, 58, 67, 71, 78, 85])

model = LinearRegression().fit(X, y)
pred = model.predict(X)

mae  = mean_absolute_error(y, pred)
rmse = np.sqrt(mean_squared_error(y, pred))   # RMSE = sqrt of MSE
r2   = r2_score(y, pred)

print(f"MAE  = {mae:.2f}")    # MAE  = 0.64  (typical miss ~0.64 points)
print(f"RMSE = {rmse:.2f}")   # RMSE = 0.85  (a bit larger; big misses weigh more)
print(f"R2   = {r2:.3f}")     # R2   = 0.994 (captures ~99.4% of the pattern)

The one gotcha

MAE and RMSE are only meaningful in your target's units, so a "good" value is entirely context-dependent — an RMSE of 5 is superb for exam scores but catastrophic for a body-temperature model. Never judge them against a fixed threshold; compare them to the scale and spread of y itself. R2 avoids this trap because it is already normalized, but it hides its own surprise: on badly overfit or nonsensical models R2 can go negative (worse than guessing the mean), so a negative R2 is a loud signal that something is broken, not a rounding quirk.

See it — from predictions to trusted metrics

flowchart TD
    A["Trained regressor<br/>predicts y_pred"] --> B["Residuals<br/>y_true - y_pred"]
    B --> C["MAE:<br/>mean of absolute<br/>residuals"]
    B --> D["RMSE: sqrt of<br/>mean squared<br/>residuals"]
    B --> E["R2: 1 - error<br/>vs mean-baseline<br/>error"]
    C --> F["Typical miss<br/>(target units)"]
    D --> G["Miss, big errors<br/>penalized more"]
    E --> H["Fit quality<br/>0=baseline, 1=perfect"]
    F --> I["Trust the model?"]
    G --> I
    H --> I

The one-line takeaway

Report MAE and RMSE for the size of your errors in real units, and R2 for how much of the pattern you captured — read all three, never just one.

Where you'll meet it

Every regression task you ship: forecasting sales, estimating delivery times, pricing homes, predicting sensor readings. In interviews you will be asked "why RMSE over MAE?" (answer: when large errors are disproportionately costly) and "what does R2 = 0 mean?" These three metrics are your first honest report card, and the deep AIM track builds straight from here into cross-validation, error analysis, and choosing loss functions that match the business cost of being wrong.

Named concepts: mean_absolute_error, root mean squared error, r2_score