PAPERS-01-010 · Paper — Compute-Optimal

The paper

Training Compute-Optimal Large Language Models, by Jordan Hoffmann and colleagues at DeepMind, published in 2022 and presented at NeurIPS (arXiv:2203.15556). It is the paper the field now shorthands as Chinchilla, after the model it trained to prove its point. The question it asks is deceptively narrow: given a fixed amount of compute, how big should the model be, and on how many tokens should you train it?

Before this paper

By 2022 the race was to build ever-larger models. Kaplan et al.'s 2020 scaling laws had shown that loss falls as a smooth power law in parameters, data, and compute — and the field read that result as bigger is better. When you gained more compute, you spent most of it on parameters and only modestly increased the data. That is how you got Gopher at 280 billion parameters and GPT-3 at 175 billion, each trained on a few hundred billion tokens. Almost everyone was scaling the same way, and almost everyone was, it turned out, undertraining.

The key idea

Hoffmann and colleagues re-ran the scaling analysis more carefully and reached a different conclusion: for a fixed compute budget, parameters and training tokens should grow in roughly equal proportion. Concretely, you want about 20 training tokens for every parameter. Under that rule, the giant models of the day were badly out of balance — far too many parameters for far too little data. They had spent their compute on size when they should have split it evenly between size and data.

How it works

Compute for a transformer is approximately proportional to parameters times tokens, so any budget defines a curve of (model size, data size) pairs you can afford. The paper estimates the loss-minimizing point on that curve three independent ways — fixing model sizes and varying data, fixing compute and varying the split, and fitting a parametric loss surface — and all three agree. The optimum is not the biggest model you can fit; it is a smaller model trained on far more tokens, with the two scaled together at roughly 20 tokens per parameter.

See it — the compute-optimal trade

flowchart LR
  A["Fixed compute<br/>budget"] --> B["Split evenly:<br/>params + tokens"]
  B --> C["~20 tokens<br/>per parameter"]
  C --> D["Smaller model,<br/>more data"]
  D --> E["Lower loss<br/>same budget"]

What they showed

They put the rule to the test. Chinchilla — 70 billion parameters, trained on 1.4 trillion tokens — uses the same compute budget as Gopher but reallocates it toward data. The much smaller Chinchilla outperforms Gopher at 280 billion parameters and GPT-3 at 175 billion across a broad battery of benchmarks. Same compute, a quarter of the size, better results — because the compute was spent in the right proportion. The lesson was not "train less"; it was "train the right-sized model on enough data."

What it changed

Chinchilla reframed the entire field. Overnight, "how big is your model" stopped being the headline number and "how many tokens did you train on" took its place — data, not parameters, became the scarce resource everyone chased. The compute-optimal ratio became the default sizing rule for the models that followed, and it explains why later flagship models are smaller and far more heavily trained than their 2020 predecessors. But a capable base model, however well-sized, still is not helpful or safe by default — it will happily complete a toxic prompt or ignore your instruction. Next: Learning from Feedback — aligning the model with human preferences through R.L.H.F.

The one-line takeaway

For a fixed compute budget, scale parameters and data together — about 20 tokens per parameter — because most big models were undertrained.

Read it / meet it

Read it at arxiv.org/abs/2203.15556. The compute-optimal rule now lives in every pretraining plan: when a lab decides how large a model to train and how much data to gather, it is running the Chinchilla calculation, explicitly or by instinct. "Chinchilla-optimal" is shorthand practitioners still use when they size a run.

Named concepts: compute-optimal scaling, Chinchilla, tokens-per-parameter ratio