ML-01-006-P · Primer — Feature Scaling: Why KNN Needs It

The idea in one sentence

If one column is measured in big numbers (dollars) and another in small ones (years), KNN's distance is silently decided by the big column alone — so you rescale every column onto the same footing first.

Why it exists

KNN classifies a new point by looking at its nearest neighbors, and "nearest" is measured with straight-line (Euclidean) distance. That distance just adds up the gaps across every feature. A gap of 20,000 in salary and a gap of 20 in age are both real, but 20,000 squared utterly dwarfs 20 squared — so the neighbor search reduces to "who has the closest salary?" and the age column stops mattering. This isn't a bug in KNN; it's what raw units do to a distance. Scaling exists to remove the accident of units so each feature gets an honest, equal vote in who counts as "close."

How it works

StandardScaler rescales each column to have mean 0 and standard deviation 1: it subtracts the column's mean and divides by its standard deviation, turning raw values into "how many standard deviations from average." After this, a one-standard-deviation move in age weighs the same as a one-standard-deviation move in salary, so distance reflects real similarity instead of unit size. The golden rule: fit the scaler on the training data only, then use that same fitted scaler to transform both train and test — the test set must never influence the mean and std it will be measured against. Below, the same KNN is run on raw columns and on scaled columns; watch accuracy jump from 0.00 to 1.00.

import numpy as np
from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score

# 0 = "won't buy": younger, lower salary.  1 = "will buy": older, higher salary.
X_train = np.array([
    [25, 30000], [28, 32000], [31, 34000], [34, 33000],  # -> 0
    [50, 60000], [53, 62000], [56, 64000], [59, 61000],  # -> 1
], dtype=float)
y_train = np.array([0, 0, 0, 0, 1, 1, 1, 1])

# Age is the honest signal; salary is nudged toward the WRONG class, so a
# distance dominated by raw (unscaled) salary gets fooled.
X_test = np.array([
    [29, 55000],   # young -> should be 0, but salary looks high
    [55, 40000],   # older -> should be 1, but salary looks low
])
y_test = np.array([0, 1])

# Raw features: salary's big numbers dominate the distance.
knn_raw = KNeighborsClassifier(n_neighbors=1).fit(X_train, y_train)
acc_raw = accuracy_score(y_test, knn_raw.predict(X_test))

# Scaled features: fit on TRAIN only, then transform BOTH.
scaler = StandardScaler().fit(X_train)
knn_scaled = KNeighborsClassifier(n_neighbors=1).fit(scaler.transform(X_train), y_train)
acc_scaled = accuracy_score(y_test, knn_scaled.predict(scaler.transform(X_test)))

print(f"Raw accuracy:    {acc_raw:.2f}")     # Raw accuracy:    0.00
print(f"Scaled accuracy: {acc_scaled:.2f}")  # Scaled accuracy: 1.00

The one gotcha

Never fit the scaler on your test set (or on the full dataset before splitting). If the test data helps compute the mean and standard deviation, information from data you're supposed to be predicting leaks into training, and your reported accuracy is a fantasy. Always fit on train, then transform both with that one fitted scaler — in real projects wrap this in a Pipeline so it can't go wrong.

See it — fit on train, transform both, then KNN

flowchart TD
    A["Split data<br/>train + test"] --> B["StandardScaler.fit<br/>(train only)"]
    B --> C["transform(train)<br/>mean 0, std 1"]
    B --> D["transform(test)<br/>same mean and std"]
    C --> E["KNN.fit<br/>(scaled train)"]
    D --> F["KNN.predict<br/>(scaled test)"]
    E --> F
    F --> G["Equal-weight distance<br/>honest accuracy"]

The one-line takeaway

KNN measures distance, so put every feature on the same scale — fit StandardScaler on train, transform both — or your biggest-unit column secretly decides everything.

Where you'll meet it

Any distance- or gradient-based model that mixes features of different units: KNN, K-means clustering, SVMs with RBF kernels, PCA, and neural nets all behave far better once features are scaled. You'll reach for StandardScaler (or MinMaxScaler) in nearly every tabular ML pipeline — and in the deep AIM track you'll see why the "fit on train, transform both" discipline is the difference between a model that generalizes and one that only looked good on paper.

Named concepts: StandardScaler, distance sensitivity to units, fit on train, transform both