AIM-01-009-P · Primer — Train / Validation / Test Split

The idea in one sentence

You split your labeled data into three disjoint parts — training, validation, and test — and each part has exactly one job, so that the final number you report is an honest estimate of how the model does on data it has never seen.

Why it exists

A model graded on the same rows it learned from will always look brilliant, because with enough capacity it can simply memorize those rows — that score measures recall, not skill. The whole point of machine learning is generalization: performance on new, unseen data. So you must hold some data back and never let the model learn from it. Think of a student: the training set is the textbook they study, the validation set is the practice quizzes they use to adjust how they study, and the test set is the sealed final exam, taken once. If the student had seen the final in advance, a perfect score would tell you nothing.

How it works

You cut the data three ways so that every row lands in exactly one of the three disjoint sets:

\[ n = n_{\text{train}} + n_{\text{val}} + n_{\text{test}}, \qquad p_{\text{train}} + p_{\text{val}} + p_{\text{test}} = 1 \]

A common split is \(p_{\text{train}} = 60\%\), \(p_{\text{val}} = 20\%\), \(p_{\text{test}} = 20\%\) (with lots of data, people go \(98\%/1\%/1\%\)). The training set is what the model fits its parameters on. The validation set is where you make choices: try different hyperparameters, compare rival models, and decide when to stop — each choice judged on this held-out slice the model never trained on. The test set stays locked away until the very end and is touched exactly once, to produce a single honest estimate. Worked example: 10,000 labeled emails, spam or not, split 6,000 / 2,000 / 2,000:

\[ \begin{aligned} n_{\text{train}} &= 0.60 \times 10{,}000 = 6{,}000 \\ n_{\text{val}} &= 0.20 \times 10{,}000 = 2{,}000 \\ n_{\text{test}} &= 0.20 \times 10{,}000 = 2{,}000 \end{aligned} \]

You train on the 6,000; on the 2,000 validation emails you try a few settings and pick the best, say \(\text{Acc}_{\text{val}} = 94\%\); only then do you unlock the test set, where the model scores \(\text{Acc}_{\text{test}} = 92\%\) on 2,000 unseen emails. That 92% is the number you trust and report. Common variations: cross-validation rotates the validation slice across \(k\) folds when data is scarce; a time-based split trains on the past and tests on the future so there is no look-ahead; a stratified split preserves the class balance (e.g. \(5\%\) fraud) in every part.

See it — one dataset, three disjoint jobs

flowchart TD
    D["Labeled data<br/>10,000 emails"] --> T["Train 60%<br/>6,000 rows"]
    D --> V["Validation 20%<br/>2,000 rows"]
    D --> E["Test 20%<br/>2,000 rows"]
    T --> FT["Fit parameters"]
    V --> TU["Tune and compare<br/>pick best 94%"]
    E --> ON["Unlock once<br/>report 92%"]
    FT --> TU
    TU --> ON

The one gotcha

Touch the test set once and it stops being a test. The moment you look at the test score, change something, and check again, you are tuning to the test — which is the validation set's job, not the test set's. Each peek leaks a little information, and the test number creeps upward into a promise the model can't keep in production. If you find yourself checking the test score more than once, you no longer have a test set; you have a second validation set and no honest estimate left.

See it (visualization plan)

TilesScene for the 60/20/20 cut (Training 60% · Validation 20% · Test 20%) and again for the worked-example counts (6,000 / 2,000 / 2,000); a PipelineScene (Fit parameters → Tune & compare → Test once); a MetricLineScene where the reported test score creeps above the true ~92% line with each peek, turning red as it crosses; and a BigNumberScene counting the "honest test sets left after a second peek" down to 0.

The one-line takeaway

The test set is a one-shot final exam — peek and you've graded your own homework.

Where you'll meet it

scikit-learn's train_test_split and cross_val_score, the separate validation DataLoader in every PyTorch and TensorFlow project, the val-loss curve you watch in TensorBoard, and every Kaggle competition that hides its true test set on a private leaderboard for exactly this reason. The sneakier cousin — the same information leaking in without an obvious peek — is data leakage (AIM-01-010), and cross-validation gets its own treatment in AIM-01-024.

Named concepts: train/validation/test split, held-out data, generalization