PAPERS-01-009 · Paper — Scaling Laws

The paper

Scaling Laws for Neural Language Models, by Kaplan et al. at OpenAI, posted to arXiv in 2020 (arXiv:2001.08361). It is an empirical study, not a new architecture: the authors trained a large family of Transformer language models across a wide range of sizes and dataset sizes, and asked a single question — how does the model's loss actually behave as you make things bigger? The answer turned out to be startlingly clean, and it reframed how the whole field thinks about spending compute.

Before this paper

By 2020 the recipe for a better language model was largely folklore: make it bigger, feed it more text, and hope. Practitioners knew that scale helped, but there was no principled way to say how much it would help, or which knob to turn next. Given a fixed budget of GPU-hours, should you train a bigger model, gather more data, or just train longer? Teams guessed, and the guesses were expensive — a large training run could cost more than the experiment that was supposed to justify it.

The key idea

The loss of a language model is not a mystery to be discovered run by run — it is a smooth, predictable function of scale. Kaplan and colleagues showed that cross-entropy loss falls as a power law in each of three quantities held separately: the number of model parameters, the size of the training dataset, and the amount of compute. A power law is the thing that becomes a straight line on log-log axes, which means you can fit it from small, cheap runs and extrapolate to the behavior of a model you have not yet trained.

How it works

The method is deliberately mundane, and that is the point. Train many Transformers spanning orders of magnitude in parameters and data; for each, measure the cross-entropy loss on held-out text; then plot loss against parameters, against dataset size, and against compute, each on log-log axes. The points do not scatter — they land on straight lines. Each straight line is a power law with its own exponent, and because the fit is so tight, a handful of small runs predicts where a much larger run will end up. That predictability is what converts model training from art into forecasting.

See it — the scaling-law method

flowchart LR
  A["Train models<br/>across many sizes"] --> B["Measure held-out<br/>cross-entropy loss"]
  B --> C["Plot loss vs scale<br/>on log-log axes"]
  C --> D["Points fall on a<br/>straight line"]
  D --> E["Power law:<br/>extrapolate the loss"]

What they showed

The headline result is the regularity itself: cross-entropy loss decreases as a power law in each of parameters, dataset size, and compute, and the relationship holds over roughly seven orders of magnitude. That is not a local trend near one model size — it is a law that spans from tiny models to the largest the authors could afford, with no sign of bending. From these curves the paper drew a practical corollary about how to spend a fixed compute budget: at the time, the analysis implied that to be compute-optimal you should make the model much bigger and train it for a relatively short time — pour the budget into parameters, not into passes over data.

What it changed

This paper turned scaling into engineering. Instead of gambling a full training run, a team could fit the curves on cheap runs, read off the exponent, and forecast the payoff of the next order of magnitude before committing the compute. That confidence helped launch the era of ever-larger models — the same year's GPT-3 lived squarely inside this worldview. But the "bigger, briefly" recipe carried a flaw. That recommendation turns out to be wrong on the data axis: models built this way were badly undertrained, given far more parameters than tokens. Next: Compute-Optimal (Chinchilla).

The one-line takeaway

Loss is a law, not a lottery — it falls as a power law in scale, so you can predict it before you pay for it.

Read it / meet it

Read it at arxiv.org/abs/2001.08361. The idea lives today in every capability forecast and every training-budget plan: labs still fit scaling curves to decide model size before they burn the compute, and "scaling laws" is now the default language for arguing about what more scale will and will not buy. Its most important descendant is the correction that follows it — Chinchilla's compute-optimal rebalancing of parameters against data.

Named concepts: scaling laws, power-law scaling, compute-optimal training