In January 2020, a team at OpenAI led by Jared Kaplan published a paper that quietly shaped the next five years of AI. Scaling Laws for Neural Language Models showed that the performance of language models improves in a smooth, predictable way as you make them bigger, train them on more data and spend more compute. That predictability gave labs the confidence to invest enormous sums in ever-larger training runs.
The question
Before this paper, researchers knew that larger neural networks often performed better, but nobody could say by how much, or how to split a fixed budget between model size and training data. Training a large model is expensive; guessing wrong wastes millions of dollars.
What the authors did
The team trained many transformer language models of different sizes — from very small to over a billion parameters — on different amounts of text and with different compute budgets. For each run they measured the test loss: a number that captures how well the model predicts held-out text. Lower loss means better prediction.
They then looked for mathematical relationships between loss and three ingredients:
- N — the number of model parameters (excluding embeddings);
- D — the size of the dataset in tokens;
- C — the amount of compute used for training.
The key finding: power laws
When one ingredient is the bottleneck, loss falls as a power law of that ingredient. Plotted on a log-log chart, the relationship is close to a straight line that holds across several orders of magnitude.
In plain terms: every time you multiply model size by a fixed factor (say ten), loss drops by a roughly constant fraction. The improvements never stop abruptly, but each step requires multiplying resources again.
The paper also reported that:
- Architecture details matter less than scale. Within a reasonable range, the depth and width of the network had much less effect on loss than total parameter count.
- Bigger models are more sample-efficient. A larger model reaches a given loss after seeing fewer tokens than a small one.
- For a fixed compute budget, it is better to train a very large model and stop before it fully converges than to train a small model to convergence.
Why it mattered
Scaling laws turned model development into something closer to engineering. If you can predict the loss of a large model from experiments on small ones, you can plan a training run, estimate its cost and justify the investment. The paper was published months before GPT-3, the 175-billion-parameter model whose capabilities — including learning tasks from a few examples in the prompt — seemed to confirm that scale itself unlocks new behavior.
The correction: Chinchilla
Two years later, DeepMind researchers led by Jordan Hoffmann revisited the question in Training Compute-Optimal Large Language Models (2022). With a different experimental setup, they found that for a given compute budget, model size and training tokens should grow roughly in proportion — about 20 training tokens per parameter in their experiments. That implied many large models of the time were undertrained: too many parameters, too little data.
Their 70-billion-parameter model Chinchilla, trained on 1.4 trillion tokens, outperformed much larger models trained on less data. The industry adjusted: later models were trained on far more tokens relative to their size.
Limits and caveats
Loss is not capability. Scaling laws predict how well a model predicts text, not whether it will be good at a specific task like coding or arithmetic. Some abilities improve smoothly; others seem to appear only at certain scales, though researchers debate how much of this “emergence” depends on how abilities are measured.
Data is finite. The laws assume more high-quality data is available. As models consume most of the public text on the web, the supply of fresh data becomes a constraint, pushing labs toward synthetic data and other modalities.
Inference costs. Training-optimal is not the same as deployment-optimal. A smaller model trained on more data is cheaper to run for millions of users, so many labs now deliberately “overtrain” smaller models beyond the Chinchilla ratio.
New axes of scaling. Recent reasoning models show that spending more compute at inference time — letting the model think longer before answering — can also improve results predictably, opening a second dimension of scaling.
The takeaway
The 2020 scaling-laws paper did not invent large models, but it gave the field a map. Its central message — performance improves predictably with resources — explains much of the investment boom that followed, and its revisions explain why today’s models are trained on vastly more data than their predecessors.
Frequently asked questions
What is a power law?
A relationship in which one quantity changes as a fixed power of another. In scaling laws, it means that multiplying model size, data or compute by a constant factor reduces the loss by a roughly constant fraction, which appears as a straight line on a log-log chart.
Do scaling laws mean progress will continue forever?
No. They describe trends observed within the tested range. Running out of high-quality data, rising costs and physical limits on energy and chips can all slow progress, and loss improvements do not guarantee improvements on every task people care about.
How do labs use scaling laws in practice?
They train a series of smaller models, fit a curve to their results and extrapolate to predict the loss of a much larger run. This helps choose the model size and data budget before committing to an expensive training job.