ML-01-005-P · Primer — K-Nearest Neighbors: Your First Classifier
The idea in one sentence
K-Nearest Neighbors classifies a new point by finding the k examples it sits closest to and letting them vote — whatever label most of those neighbors carry becomes the prediction, the same way you'd guess a stranger's hobby from the people they hang out with.
Why it exists
Most of machine learning sounds intimidating because it hides math behind it — weights, gradients, loss curves. KNN exists as the friendly front door: it's the one classifier you can fully picture in your head, with zero equations, and still get real, working predictions. The whole idea is a hunch you already trust in daily life — things that are near each other tend to be alike. A fruit that's small and soft is probably in the same family as other small, soft fruits. An email that looks like the spam you've seen before is probably spam too.
KNN turns that hunch into an algorithm with almost no machinery. There's no training in the usual sense — the model just remembers every example you show it. When a new, unlabeled point arrives, it measures how far that point is from every example it remembers, keeps the k closest, and takes a majority vote. That's it. It exists to prove a point that carries through your whole ML journey: a classifier is, at heart, a rule for drawing boundaries between groups — and the simplest honest rule is "look at your neighbors." Master this and you already understand the shape of what every fancier classifier is trying to do.
How it works
Every example lives as a point in space, one axis per feature — a fruit measured by [sweetness, crunch] is a dot on a 2D grid. "Nearest" means literal distance: scikit-learn defaults to Euclidean distance, the straight-line gap you'd measure with a ruler. You call fit(X, y), which for KNN simply stores the points and their labels — the "learning" is memorization. When you call predict, the model computes the distance from your new point to every stored point, sorts them, keeps the closest k, and returns the label that appears most among those k (ties broken by the nearest one). The number k is the one dial you turn — the hyperparameter n_neighbors — and it controls how many voices get a say. Here's the whole thing on six fruits:
import numpy as np
from sklearn.neighbors import KNeighborsClassifier
# Tiny dataset. Two features per fruit: [sweetness, crunch].
# Labels: 0 = grape (small, soft), 1 = apple (big, crunchy).
X = np.array([[1, 1], [2, 1], [1, 2], # 3 grapes
[7, 7], [8, 7], [7, 8]]) # 3 apples
y = np.array([0, 0, 0, 1, 1, 1])
knn = KNeighborsClassifier(n_neighbors=3) # k = 3: ask the 3 closest
knn.fit(X, y) # "training" = just memorize
mystery = np.array([[7, 6]]) # big and crunchy -> looks apple-ish
print("prediction:", knn.predict(mystery)[0]) # prediction: 1 (apple)
# Peek at the vote: who are the 3 nearest neighbors?
dist, idx = knn.kneighbors(mystery)
print("neighbor labels:", y[idx][0]) # neighbor labels: [1 1 1]
print("train accuracy:", knn.score(X, y)) # train accuracy: 1.0The mystery fruit's three nearest neighbors are all apples, so the vote is unanimous and the prediction is apple (1). kneighbors lets you literally read off who voted — the transparency is the whole charm.
The one gotcha
KNN measures raw distance, so if one feature is on a bigger numeric scale than another, it silently drowns out the rest — you almost always need to scale your features first. Imagine adding a third feature, weight_in_grams, that ranges from 50 to 300, while sweetness only ranges from 1 to 10. Distance is dominated by the grams: a 40-gram difference swamps any difference in sweetness, so KNN effectively ignores sweetness entirely and you never see an error telling you so. Your accuracy just quietly suffers. The fix is to put every feature on a comparable scale before fitting — wrap StandardScaler and KNeighborsClassifier in a Pipeline so scaling happens automatically on both training and new data. The same trap hides in k itself: too small (k=1) and one weird outlier neighbor can flip a prediction (jagged, overfit boundaries); too large (k near your whole dataset) and every point just predicts the overall majority class (bland, underfit). Scale your features, then tune k — those two habits turn KNN from a toy into something that actually holds up.
See it — how KNN predicts one new point
flowchart TB
A["New point arrives<br/>(sweetness, crunch)"] --> B["Measure distance to<br/>every stored example"]
B --> C["Sort by distance,<br/>keep the k closest"]
C --> D{"Majority label<br/>among the k?"}
D -->|"most are grapes"| E["Predict: grape (0)"]
D -->|"most are apples"| F["Predict: apple (1)"]
The one-line takeaway
To classify a point, find its k nearest neighbors and take a majority vote — just remember to scale your features first so distance is fair.
Where you'll meet it
You'll meet KNN as the very first classifier in almost every course and tutorial, precisely because it makes the abstract idea of a "decision boundary" something you can see and reason about without any math. It's a genuinely useful baseline in the wild — recommendation systems ("users like you also bought..."), simple image and handwriting recognition, and anomaly detection all lean on nearest-neighbor thinking. Interviewers love it as an opener: expect "how does KNN make a prediction?", "what happens to the decision boundary as k grows?", "why do we scale features for KNN but not always for trees?", and the classic "is KNN's cost in training or in prediction?" (answer: prediction — it does no real work until you ask it something). Once distance, neighbors, and the k dial feel natural, you're ready for the deeper AIM track, where the same instinct — near things are alike — grows into decision boundaries, cross-validation, and the bias–variance trade-off that governs every model you'll ever tune.
Named concepts: KNeighborsClassifier, distance and neighbors, the k hyperparameter