ML-01-011-P · Primer — KNN Regression: Predicting Numbers from Neighbors
The idea in one sentence
To guess a number for a new example — a house price, a temperature, a delivery time — find the handful of past examples that look most like it, and just average their answers.
Why it exists
Most "predict a number" methods start by assuming a shape for the relationship — a straight line, a curve, some equation you have to fit. That's powerful, but it forces you to commit up front to a formula that might be wrong. K-Nearest-Neighbors regression throws that assumption out. It says: I don't need a formula for how size relates to price; I just need to remember every example I've seen, and when you hand me a new one, I'll look up the most similar past cases and report the average of what actually happened to them. This is the same instinct a human appraiser uses — "houses like this one on this street sold for around $210k, so yours is probably close." It's called a non-parametric method because there's no fixed set of coefficients to learn; the training data is the model. That makes it a wonderful first regressor to meet: the logic is something you already do in daily life, there's almost no math to fear, and it builds the core intuition — similar inputs tend to have similar outputs — that underlies a huge amount of machine learning to come.
How it works
Training is almost a joke: the model just stores your example inputs X and their known target numbers y — no equation is solved. The work happens at prediction time. To predict for a new point, KNN measures the distance from it to every stored example, picks the k closest ones (k is a number you choose, like 3 or 5), and returns the average of those k neighbors' target values. That average is your prediction. Choosing k trades off two failure modes: a small k follows the data very closely but jitters with noise, while a large k gives a smoother, calmer prediction that can wash out real detail. In scikit-learn this is KNeighborsRegressor(n_neighbors=k), and the key difference from KNN classification is that here we average continuous targets instead of voting on labels. Watch it predict a house price, and note that the result is exactly the mean of the three nearest neighbors' prices:
import numpy as np
from sklearn.neighbors import KNeighborsRegressor
# Tiny inline dataset: house size (in hundreds of sqft) -> price ($k)
X = [[8], [11], [12], [14], [17], [19], [22]]
y = [150, 190, 210, 240, 275, 305, 340]
model = KNeighborsRegressor(n_neighbors=3) # look at the 3 closest houses
model.fit(X, y) # "training" = just remembering X, y
pred = model.predict([[13]]) # a new 1300-sqft house
print(pred) # [213.33333333]
# Why 213.33? The 3 nearest sizes to 13 are 12, 14, and 11,
# whose prices are 210, 240, 190 -> mean = 213.33 exactly.
print(np.mean([210, 240, 190])) # 213.33333333333334
print(round(model.score(X, y), 3)) # R^2 on training data: 0.888The one gotcha
The value of k makes or breaks the model, and both extremes fail in opposite ways. Set n_neighbors=1 and each prediction is just the single closest example's target — the model scores a perfect R² of 1.0 on the training data because every point is its own nearest neighbor, but that's memorization, not learning: the prediction surface is a jagged staircase that overreacts to every noisy point. Push k all the way up to the number of samples and the model averages everything every time, so it returns the global mean for any input you give it — a flat, useless line that ignores the very similarity KNN is supposed to exploit.
from sklearn.neighbors import KNeighborsRegressor
X = [[8], [11], [12], [14], [17], [19], [22]]
y = [150, 190, 210, 240, 275, 305, 340]
k1 = KNeighborsRegressor(n_neighbors=1).fit(X, y)
kAll = KNeighborsRegressor(n_neighbors=7).fit(X, y)
print(round(k1.score(X, y), 3)) # 1.0 -> perfect but memorized
print(round(kAll.predict([[13]])[0], 2)) # 244.29 <- just the global mean
print(round(kAll.predict([[22]])[0], 2)) # 244.29 <- same guess for ANY input!The sweet spot is in between, and you find it with a validation set rather than by eyeball. One more trap that hides inside "distance": if one feature is in dollars (0–500,000) and another in bedrooms (1–5), the dollar feature dominates every distance calculation and bedrooms are effectively ignored — so scale your features (e.g. StandardScaler) before any KNN.
See it — how KNN turns neighbors into a number
flowchart TD
A["New house:<br/>size = 13"] --> B["Measure distance to<br/>every stored house"]
B --> C["Sort by distance,<br/>keep the closest k=3"]
C --> D["Neighbors: sizes<br/>12, 14, 11"]
D --> E["Their prices:<br/>210, 240, 190"]
E --> F["Average the targets"]
F --> G["Prediction:<br/>213.33"]
The one-line takeaway
KNN regression predicts a number by averaging the target values of the k most-similar stored examples — so pick k with care and scale your features, because "similar" is doing all the work.
Where you'll meet it
You'll reach for KNeighborsRegressor as a fast, honest baseline whenever you're predicting a number and want a sanity check before trying anything fancier — if a linear model or a gradient-booster can't beat plain averaging-of-neighbors, that's a red flag worth investigating. It shines on problems where similar inputs genuinely mean similar outputs and the relationship is bumpy rather than a clean line: estimating a home's value from comparable nearby listings, filling a missing sensor reading from readings at similar times, or recommending a rating by blending the ratings of look-alike users. You'll also meet its ideas everywhere KNN's cousins live — KNeighborsClassifier for labels, KNNImputer for filling missing values, and the whole family of similarity-based methods. Most of all, tuning k here — feeling the pull between a twitchy model that memorizes and a lazy one that averages everything into mush — is your first real encounter with the bias–variance tradeoff, the central tension you'll navigate in every model you build from here into the deep AIM track.
Named concepts: KNeighborsRegressor, averaging neighbor targets, k for regression