Every model launch comes with a table of benchmark scores. Those numbers drive headlines, investment and purchasing decisions — yet few people know how benchmarks are built or why a test that seemed impossible two years ago is now “solved”. This guide explains how to read them.
What a benchmark is
A benchmark is a fixed set of tasks with known correct answers (or a fixed way of judging answers), plus a scoring rule. Because everyone uses the same tasks, scores can be compared across models and over time.
Benchmarks fall into a few families.
Multiple-choice knowledge tests. MMLU (Measuring Massive Multitask Language Understanding, 2020) contains thousands of questions across 57 subjects, from law to medicine to mathematics. Its successors use harder, expert-written questions to stay ahead of models.
Task completion with automatic checks. SWE-bench gives a model a real GitHub repository and an issue to fix; the patch passes only if the project’s tests pass. Coding and math benchmarks are attractive because answers can be verified automatically.
Abstract reasoning puzzles. The ARC (Abstraction and Reasoning Corpus) benchmarks, now run by the ARC Prize Foundation, present small visual puzzles designed to be easy for people but hard for systems that rely on memorized patterns.
Human preference. LMArena (originally Chatbot Arena) shows two anonymous model responses side by side and asks users which is better. Votes are aggregated into an Elo-style rating, similar to chess rankings.
Agentic and long-horizon tasks. Newer benchmarks measure whether models can complete multi-step tasks on computers, in browsers or across long periods of work.
Why benchmarks saturate
A benchmark is “saturated” when top models score near the maximum, so it no longer separates them. This happens surprisingly fast, for several reasons.
- Genuine progress. Models really do get better, often faster than benchmark designers expect.
- Optimization pressure. Once a benchmark becomes famous, labs tune training data and methods to do well on it. Improvements may transfer only partly to real tasks.
- Contamination. Benchmark questions published online can end up in training data scraped from the web. A model that has seen the answers is not reasoning; it is remembering.
- Label noise. Some benchmark questions have wrong or ambiguous answers. Near the top of the scale, the remaining errors may reflect flaws in the test rather than in the model.
The field responds with harder benchmarks, private held-out test sets, regularly refreshed questions (as in LiveBench) and human-judged evaluations.
How to read a benchmark table
Check the settings. Scores depend on how the model was prompted, how many attempts it was allowed, whether tools were available and how long it could “think”. Two numbers for the same benchmark may not be comparable if the settings differ.
Who ran the test? Self-reported numbers from a model’s developer are useful but should be confirmed by independent evaluators or leaderboards.
Look at the spread across benchmarks. A model that leads on one test and lags on others has a specialty, not general superiority.
Mind the margin. Differences of one or two points are often within noise, especially on smaller benchmarks.
Remember cost and speed. A model that scores slightly higher but costs ten times more per task may be the worse choice for your use case. Independent analysts increasingly report quality alongside price and latency.
Preference leaderboards have their own biases
Human-preference rankings capture something benchmarks miss — what people actually like — but they can reward style over substance. Longer, more confident, nicely formatted answers tend to win votes even when they are not more accurate. Leaderboard operators have introduced controls for style and length to reduce this effect.
What benchmarks cannot tell you
No public benchmark knows your data, your users or your tolerance for errors. The most reliable evaluation is a small set of your own real tasks, with clear criteria for success, run against the models you are considering. Public benchmarks narrow the list; private tests make the decision.
See the latest numbers in our Benchmarks Snapshot.
Frequently asked questions
Which benchmark should I trust the most?
None on its own. Look for agreement across several benchmarks that test different skills, prefer results reproduced by independent evaluators, and give extra weight to benchmarks close to your own use case, such as coding tasks if you write code.
What does “contamination” mean?
It means test questions, or close copies of them, appeared in the model’s training data. The model may then recall answers instead of solving the problems, inflating its score. Private test sets and regularly refreshed questions reduce the risk.
Why do companies report different numbers for the same model?
Because evaluation settings differ: the prompt format, the number of attempts allowed, whether tools or extended reasoning were enabled, and how answers were graded. Small changes in settings can move scores by several points.
How can I benchmark models for my own work?
Collect twenty to fifty real tasks from your work, define what a good answer looks like, and run each candidate model on them under the same conditions. Score the results blind if possible. This small private benchmark is usually more decisive than any public leaderboard.