ML-01-015-P · Primer — Comparing Models: Picking the Right Tool

The idea in one sentence

To choose the best model, you train several candidates on the same training data, score them all on the same held-out test data with the same metric, and pick whichever number comes out on top.

Why it exists

When you first learn machine learning, every new model feels like a shiny hammer — logistic regression, a decision tree, k-nearest-neighbors — and it's tempting to fall in love with one and use it for everything. But no single model is best for every dataset: a straight-line separator wins on some problems, a flexible tree wins on others, and the only honest way to know which fits your data is to try them and measure. The trap is comparing them unfairly — grading one model on an easy test and another on a hard one, or judging one by accuracy and another by "it felt good." That's like timing three runners on three different tracks and declaring a winner: the comparison is meaningless. Model comparison exists to make the contest fair. Fix one split of the data, fix one metric, run every candidate through that identical gauntlet, and the winning number actually means something. This one habit — same split, same metric — is the backbone of every serious model-selection decision you'll ever make.

How it works

Start by splitting your data once into a training set and a test set, and hold that split constant for every model. Put your candidate models in a list (or dictionary) so you can loop over them. Inside the loop, each model does the same three steps: fit on the training data, predict on the test data, and get scored by one shared metric — here, accuracy_score. Because the split and the metric never change between candidates, the resulting numbers sit on the same scale and can be compared directly. Then you simply pick the model with the best score. In the tiny example below, two overlapping clusters make the problem genuinely hard, so the models disagree — logistic regression comes out ahead, which is exactly the kind of clear signal a fair comparison is meant to reveal.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score

# Tiny inline dataset: two overlapping clusters (feature1, feature2)
X = np.array([
    [3.94, 2.44], [3.08, 4.46], [4.05, 0.92], [3.05, 1.83], [1.89, 2.45],
    [2.16, 3.60], [2.84, 2.13], [2.49, 2.37], [3.64, 1.77], [2.34, 1.06],
    [-0.81, 2.72], [2.95, 1.18], [4.50, 0.40], [2.05, 1.79], [3.69, 3.62],
    [4.17, 4.42], [3.02, 1.82], [3.62, 4.17], [5.35, 5.32], [3.57, 3.67],
    [2.85, 2.44], [2.12, 6.15], [3.44, 3.52], [2.62, 4.86], [2.22, 3.77],
    [3.01, 4.43], [3.44, 2.70], [3.97, 4.47], [4.07, 4.33], [3.30, 3.60],
])
y = np.array([0] * 15 + [1] * 15)

# ONE split, reused for every model (stratify keeps class balance even)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.4, random_state=42, stratify=y)

candidates = {
    "LogReg": LogisticRegression(),
    "Tree":   DecisionTreeClassifier(random_state=0),
    "KNN":    KNeighborsClassifier(n_neighbors=3),
}

scores = {}
for name, model in candidates.items():
    model.fit(X_train, y_train)                       # same training data
    preds = model.predict(X_test)                     # same test data
    scores[name] = accuracy_score(y_test, preds)      # same metric
    print(f"{name:7s} accuracy = {scores[name]:.2f}")
# LogReg  accuracy = 0.83
# Tree    accuracy = 0.50
# KNN     accuracy = 0.75

best = max(scores, key=scores.get)
print("Best model:", best, "->", round(scores[best], 2))
# Best model: LogReg -> 0.83

The one gotcha

The split and the metric must be identical for every model, or the comparison is worthless. The most common mistake is re-splitting the data inside the loop (or with a different random_state per model), so each candidate is quietly graded on a different test set — then a model can "win" purely because it drew easier examples, not because it's better. Split once, before the loop, and reuse those exact X_train/X_test for all candidates. The same discipline applies to the metric: don't judge one model by accuracy and another by "it looks about right." Pick one metric that matches your goal (accuracy for balanced classification, something like f1_score for imbalanced classes, or mean_squared_error / r2_score for regression) and score every candidate with that one number. And remember the deeper rule: the winner must be chosen on the held-out test set, never on the training data — a decision tree can memorize the training set and score a perfect 1.00 there while flopping on data it hasn't seen.

See it — fair contest: one split, one metric, pick the winner

flowchart TD
    A["All labeled data"] --> B["train_test_split ONCE<br/>(fix random_state)"]
    B --> C["X_train, y_train"]
    B --> D["X_test, y_test"]
    C --> E["For each candidate:<br/>fit on X_train"]
    D --> F["predict on X_test"]
    E --> F
    F --> G["Score with the SAME<br/>metric (e.g. accuracy)"]
    G --> H{"More candidates<br/>left?"}
    H -->|"Yes"| E
    H -->|"No"| I["Pick the model with<br/>the best test score"]

The one-line takeaway

Compare models fairly by running every candidate through the same split and the same metric, then pick the one with the best held-out score.

Where you'll meet it

You'll reach for this pattern the moment you have a real dataset and more than one idea about how to model it — which is basically always. A first pass on any new problem is often exactly this: throw three or four sensible models into a dictionary, loop, score, and see who's in front before you invest in tuning. It's also the shape of the two headline decisions in ML — matching the model to the task (classifier vs. regressor, linear vs. tree-based, simple vs. flexible) is decided by this kind of side-by-side test, not by taste. As you go deeper, the same skeleton grows up: you'll swap the single split for cross-validation to get a steadier average, use GridSearchCV to compare not just models but their settings, and wrap everything in a Pipeline so preprocessing gets compared fairly too. But the instinct never changes — same split, same metric, pick the winner — and building it now is the foundation the entire deep AIM track is poured on.

Named concepts: model comparison, same split, same metric, matching model to task