AIM-04-003-P · Primer — Experiment Tracking

The idea in one sentence

Experiment tracking is a system of record that logs every training run's parameters, metrics, code/data version, and artifacts so you can compare runs objectively and reproduce the one that won.

Why it exists

Picture a fraud-detection team that trains dozens of model variants over a month — different learning rates, tree depths, feature sets. Each analyst saves their "best" model as a local file: model_final.pkl, model_final_v2.pkl, model_final_v2_REALLY.pkl. Three months later the production model starts drifting and they need to retrain from the winning configuration — except nobody wrote down which learning rate, which random seed, or which snapshot of the training data produced it. The team burns two weeks re-running grid search from scratch, and still can't reproduce the original 0.918 AUC because the training data table has since been backfilled. Experiment tracking exists to prevent exactly this: every run — successful or not — gets a permanent, queryable record, so "which run was that" is a lookup, not an archaeology project.

How it works

An experiment-tracking system (MLflow Tracking is the reference implementation) sits between your training code and two stores: a metadata store (a database holding params, metrics, tags) and an artifact store (S3, GCS, or local disk holding model files, plots, checkpoints). When training code calls log_param, log_metric, or log_artifact, the client writes a row tied to a unique run_id, alongside auto-captured context like the git commit hash, the training script version, and a timestamp. Metrics can be logged repeatedly across training steps (loss at epoch 1, 2, 3...), producing a time series per run, while params and tags are typically fixed once per run. The tracking server exposes this as a queryable table, so "give me the run with the highest validation AUC" becomes a SELECT ... ORDER BY metric DESC LIMIT 1 instead of a search through Slack messages.

Concretely: the fraud team runs 40 hyperparameter combinations for a gradient-boosted classifier. Each run logs three params (learning_rate, max_depth, n_estimators) and one metric (val_auc). Run 7 logs lr=0.10, depth=4, trees=200 → val_auc=0.867. Run 27 logs lr=0.05, depth=6, trees=300 → val_auc=0.912. Run 33 logs lr=0.01, depth=8, trees=500 → val_auc=0.918. Run 41 logs lr=0.02, depth=5, trees=400 → val_auc=0.899. Querying the tracking store for the maximum val_auc returns run 33 in milliseconds — and because run 33's record also pins the git commit and the training-data snapshot ID, anyone can check out that exact commit, pull that exact data version, and reproduce the 0.918 result byte-for-byte.

The second half of the workflow is deciding whether the best run is good enough to ship. If production is currently running a baseline with \(m_{baseline} = 0.900\) AUC, and the team requires a minimum improvement margin \(\epsilon = 0.01\) before touching production (churn has a cost — redeploying a marginally-better model isn't free), the promotion rule is: compute \(\Delta = m_{best} - m_{baseline}\), and promote only if \(\Delta \ge \epsilon\). For run 33, \(\Delta = 0.918 - 0.900 = 0.018 \ge 0.01\), so it clears the bar and moves to the model registry for staging.

See it — A tracked run flows from training through logging into a queryable store, then the best run is compared against the promotion bar.

flowchart TD
  A["Training Run<br/>(40 configs)"] --> B["Log Params<br/>Metrics<br/>Artifacts"]
  B --> C["Tracking Store<br/>(runs table)"]
  C --> D["Query: Best<br/>Metric"]
  D --> E["Compare vs<br/>Baseline + Margin"]
  E --> F["Promote to<br/>Registry"]

The one gotcha

Logging the metric isn't enough — if you don't also pin the code commit, data version, environment, and random seed inside the same run record, "best_run" points at a number you can never regenerate. Teams that only track metrics end up with a leaderboard of ghosts: they know run_33 scored 0.918 AUC, but six months later nobody can rebuild the exact model that produced it.

The one-line takeaway

Track params, metrics, and artifacts together — not just scores.

Where you'll meet it

MLflow Tracking, Aim, ClearML, Kubeflow, Ray Tune

Named concepts: experiment tracking