AIM-01-003-P · Primer — Training vs Inference

The idea in one sentence

Training is the one-time, offline process where a model learns its parameters from labeled data; inference is the fast, repeated, live process where the finished model uses those parameters to make a prediction on one new input.

Why it exists

Learning and using are two different jobs, so we split them the way a student splits studying from taking the exam. Studying is slow, expensive, and done once with the textbook open — you can take hours, re-read, and see the answer key. The exam is fast, done many times, and you only get the question, never the answers. A model works the same way: training can chew through a million rows on a rack of GPUs for six hours; inference has to score one transaction in fifteen milliseconds with none of the labels present. Conflating the two — expecting the exam to feel like studying — is where most beginner intuition goes wrong.

How it works

The mechanism has two distinct passes. Training feeds batches of examples forward, measures how wrong the predictions are with a loss function, and then adjusts every parameter slightly to reduce that loss (gradient descent), repeating for many epochs until the numbers stop improving — the output is a frozen set of weights, say a 400-megabyte file. Inference loads those frozen weights and runs a single forward pass on a new input: no labels, no loss, no weight updates, just input in, prediction out. Concretely: an offline job might train a fraud model on 12 months of historical charges (with the fraud/not-fraud labels attached) overnight; the deployed service then loads that model and, when a card is swiped, computes the same features for that one charge and returns "risk 0.02" in a few milliseconds. Same math in the forward direction, but one side learns and the other only serves.

See it — train learns weights once; serve reuses them fast

flowchart LR
    A["many labeled rows<br/>(12 months history)"] --> B["forward pass<br/>+ loss + adjust<br/>(repeat epochs)"]
    B --> C["frozen weights<br/>(400 MB file)"]
    C --> D["load into<br/>live service"]
    E["one unlabeled row<br/>(card swiped now)"] --> F["single forward pass<br/>no loss, no update"]
    D --> F
    F --> G["risk 0.02<br/>(in ~15 ms)"]

The one gotcha

A model can be brilliant offline and useless live, because training and serving are different worlds. In training you have the whole dataset, clean historical features, unlimited time, and the labels; in production you have one record, features computed by a different code path, a hard latency budget, and no labels to check yourself against. When the feature you trained on ("average spend over 30 days," computed in a tidy notebook) is computed a slightly different way at serve time (a live SQL query that rounds differently or misses yesterday's data), the model quietly degrades — this mismatch is train-serve skew, and it doesn't throw an error. Your 0.95 offline accuracy becomes 0.71 in production and every dashboard stays green.

See it (visualization plan)

CompareScene or PipelineScene with two panels side by side: TRAIN — a stack of many labeled rows → "loss → adjust weights" loop → a frozen weights file (slow, offline, once); SERVE — a single unlabeled row → one forward pass → "risk 0.02" (fast, live, millions of times). Then a BigNumberScene counting offline accuracy 0.95 dropping to live 0.71 to make the skew concrete, tagged "same model, different world."

The one-line takeaway

Offline brilliant, live useless — training and serving are different worlds.

Where you'll meet it

Every framework draws this line for you: in scikit-learn it is literally model.fit(X, y) (train) versus model.predict(X_new) (infer); in PyTorch it is the training loop with loss.backward() and optimizer.step() versus model.eval() and torch.no_grad() at serving; the trained artifact gets saved (pickle, torch.save, ONNX) and loaded behind an API like FastAPI, TorchServe, or KServe/BentoML. The failure mode is dramatized in Drill 4 — The Stale Score, where a serving-side cache freezes the very features the model was trained to trust, and an offline-perfect fraud model goes silently blind in production.

Named concepts: training, inference, train-serve skew