AIM-04-001-P · Primer — What is MLOps

The idea in one sentence

MLOps is the set of engineering practices — versioning, automated pipelines, monitoring, and staged rollouts — that turns a machine learning model from a research notebook into a reliable, maintainable production system.

Why it exists

Zillow's home-buying algorithm is the cautionary tale: the pricing model that told Zillow Offers how much to bid on houses kept predicting rosy prices while the 2021 housing market cooled, and nobody had automated drift monitoring or a retraining trigger to catch it. By the time humans noticed, Zillow had overpaid for thousands of homes, took a $304 million inventory write-down, shut the unit down, and cut 25% of its workforce. MLOps exists to prevent exactly this failure mode: without versioned data, CI for training pipelines, drift monitors, and automated rollback, a model can be silently wrong for months while everyone assumes it still behaves like the day it was validated.

How it works

MLOps maturity is usually measured against a checklist of practices spanning the whole lifecycle: data/model versioning, a feature store, experiment tracking, an automated (CI) training pipeline, a model registry, automated validation gates before deploy, automated (CD) serving deploys, canary/shadow rollouts, production drift monitoring, an automated retraining trigger (CT), automated rollback, and infrastructure-as-code. A team's maturity score is simply \( \text{score} = \dfrac{\text{adopted}}{\text{total}} \) — the fraction of that checklist actually in place — and it maps onto three bands: \[ \text{level}(s) = \begin{cases} \text{manual} & s < 0.34 \\ \text{repeatable} & 0.34 \le s < 0.67 \\ \text{automated} & s \ge 0.67 \end{cases} \]

Concretely: Team A tracks the 12 practices above and has only adopted 4 — git for code, a shared spreadsheet of experiment results, manual training runs in a notebook, and a manual copy of the pickled model onto a server. Their score is \( 4/12 = 0.333 \), landing just under the 0.34 cutoff — still "manual," meaning every release is a person's judgment call. Team B adds a model registry (MLflow), an automated training pipeline (Kubeflow), and CI tests that check incoming data schema, bringing adopted practices to 7 of 12: \( 7/12 \approx 0.583 \), now "repeatable" — releases follow a defined process even though humans still trigger them. Team C wires up drift monitoring (Evidently), an automated retraining trigger, canary deploys through KServe, and automated rollback on regression, reaching 10 of 12: \( 10/12 \approx 0.833 \), "automated" — the system detects its own decay and responds without waiting for a person to notice.

In practice, teams recompute this score quarterly, track the trend in a dashboard (MLflow or an Aim experiment board), and use the gap to prioritize the next investment — usually whichever single missing practice removes the most operational risk, not whichever is cheapest to check off.

See it — MLOps maturity moves from manual, ad-hoc notebooks through a repeatable pipeline to a fully automated CI/CD/CT loop.

flowchart TD
    S0["score < 0.34"] --> L0["Level 0: Manual<br/>(notebooks, manual<br/>deploys)"]
    S1["0.34 to 0.67"] --> L1["Level 1: Repeatable<br/>(versioned data/models,<br/>CI for training)"]
    S2["score >= 0.67"] --> L2["Level 2: Automated<br/>(CI/CD/CT, drift<br/>monitor, auto-rollback)"]
    L0 --> L1
    L1 --> L2

The one gotcha

A high score can be gamed: a team can automate three easy practices (say, logging and IaC and a registry) while skipping the one that actually catches failures — drift monitoring — and still cross the "automated" threshold while shipping silently-wrong predictions.

The one-line takeaway

Automate what reduces risk, not everything.

Where you'll meet it

MLflow and Aim track experiments and model versions; Kubeflow and Ray automate training pipelines (CI/CT); BentoML and KServe endpoints serve models (CD); Evidently monitors drift once a model is live.

Named concepts: mlops maturity