AIM-04-002-P · Primer — The ML Lifecycle

The idea in one sentence

The ML lifecycle is the repeating loop — data, train, evaluate, deploy, monitor — that turns a model from a one-off notebook experiment into a system someone keeps alive in production.

Why it exists

Teams that treat ML like software shipping ("build it, deploy it, done") get burned the same way over and over: a fraud-detection model is trained on last year's transaction patterns, deployed, and never revisited. Six months later fraudsters have shifted tactics, the input distribution has drifted, and the model is quietly approving transactions it should flag — but the dashboard still shows "model: healthy" because nobody wired monitoring back into retraining. The lifecycle exists as an explicit loop, not a line, specifically to prevent this: the monitor stage's whole job is to notice decay and hand fresh data back to the data stage, so the model gets retrained before the losses pile up, not after.

How it works

The canonical lifecycle has five stages that run in a cycle, not a one-time sequence: data (collect and label training examples), train (fit the model), evaluate (score it against held-out data and a champion model), deploy (package and serve it in production, e.g. via BentoML or KServe), and monitor (watch live inputs and predictions for drift or degradation). In tooling terms, one pass through data→train→evaluate is typically logged as an MLflow (or Aim, ClearML) run; a model registry entry marks the deploy transition; and Evidently or a custom job continuously computes drift statistics during monitor. The loop closes when monitor decides the model is stale enough to justify going back to data — that decision is usually made with a concrete statistic, not a gut feeling.

A common one is the Population Stability Index (PSI), which compares the distribution of a feature (or the model's score) at training time against its distribution in current production traffic:

\[ PSI = \sum_{i=1}^{n} (Actual_i - Expected_i) \times \ln\left(\frac{Actual_i}{Expected_i}\right) \]

Worked example: a lending-risk model was trained on applicant income bucketed into five bins, with training-time proportions \(Expected = [0.10, 0.25, 0.30, 0.20, 0.15]\). Six months into production, the applicant pool has shifted toward lower incomes: \(Actual = [0.30, 0.15, 0.20, 0.20, 0.15]\). Computing each bin's contribution: bin 1 gives \((0.30-0.10)\ln(0.30/0.10) = 0.20 \times 1.0986 \approx 0.2197\); bin 2 gives \((0.15-0.25)\ln(0.15/0.25) = -0.10 \times (-0.5108) \approx 0.0511\); bin 3 gives \((0.20-0.30)\ln(0.20/0.30) = -0.10 \times (-0.4055) \approx 0.0406\); bins 4 and 5 are unchanged and contribute \(0\). Summing gives \(PSI \approx 0.311\). Since the standard reading is "PSI \(< 0.1\): no meaningful shift, \(0.1\)–\(0.25\): moderate, \(> 0.25\): significant," this \(0.311\) crosses the significant threshold — the monitor stage fires an alert, a Kubeflow Pipelines run (or MLflow, Ray) is triggered to re-enter the data stage, and the cycle restarts: pull fresh labeled applications, retrain, re-evaluate against the current champion, and redeploy only if it wins.

See it — The lifecycle is a closed loop: monitoring doesn't just report health, it re-opens the data stage.

flowchart TD
    Data["Data<br/>collect & label"] --> Train["Train<br/>fit model"]
    Train --> Evaluate["Evaluate<br/>test metrics"]
    Evaluate --> Deploy["Deploy<br/>serve in prod"]
    Deploy --> Monitor["Monitor<br/>watch for drift"]
    Monitor -->|"PSI above 0.25"| Data

The one gotcha

Teams draw this as a waterfall — data, train, evaluate, deploy — and stop there, treating "deploy" as the finish line. Without an explicit trigger (a PSI threshold, a scheduled retrain job, an accuracy floor) wired from monitor back into data, the loop never closes: the model keeps serving in production long after the world it was trained on has changed, and nothing in the system is watching for that except a human who has to remember to check.

The one-line takeaway

The lifecycle loops — monitoring restarts data, not ships once.

Where you'll meet it

MLflow (or Aim, ClearML) tracks the data→train→evaluate runs and registers the winning model; Kubeflow Pipelines orchestrates the stages end-to-end and can auto-trigger a new run; BentoML (or KServe) packages and serves the model at the deploy stage; Evidently computes drift metrics like PSI at the monitor stage and is what typically fires the signal that reopens the loop at data.

Named concepts: ml lifecycle