Every model launch comes with a chart: new state of the art on this benchmark, top of that leaderboard. It’s tempting to pick whichever model is at the top. Our take: benchmarks are useful signals for researchers, but they are a poor way to choose a model for your actual work.

What benchmarks measure

Benchmarks such as MMLU test knowledge across subjects with multiple-choice questions; others test coding, math, reasoning or tool use. Crowd-sourced arenas such as LMArena rank models by human votes comparing anonymous responses. These are valuable: they track progress and expose weaknesses.

Why they mislead buyers

Saturation. Top models score so close to each other on popular benchmarks that differences are within noise.

Contamination. Benchmark questions can leak into training data, inflating scores.

Optimization. Developers naturally tune models and report results on benchmarks that make them look good.

Mismatch. Your tasks — summarizing your contracts, answering your customers, writing in your brand voice, working with your codebase — aren’t on any leaderboard.

Preference ≠ correctness. Arena votes reward answers people like, which can favor length, formatting or confident tone over accuracy.

Missing dimensions. Benchmarks rarely capture latency, cost, reliability across repeated runs, context handling, safety behavior or how well a model follows your specific instructions.

What to do instead

Run a small evaluation on your own work. It takes an afternoon:

  1. Collect 20–30 real examples of the task, including hard and edge cases.
  2. Write down what a good answer looks like for each.
  3. Run two or three candidate models with the same prompt.
  4. Score outputs blind, ideally with colleagues, on accuracy, usefulness and format.
  5. Compare cost and speed alongside quality.
  6. Repeat when models update.

See our guide on building an evaluation set for LLM apps.

When benchmarks do help

  • Shortlisting: narrowing dozens of models to a few candidates.
  • Spotting big gaps: a model far behind on coding benchmarks is unlikely to be the best coding assistant.
  • Tracking trends over time.
  • Specialized benchmarks close to your domain can be informative.

The bottom line

Use leaderboards to decide what to test, not what to buy. The best model is the one that performs best on your tasks, at a cost and speed that fit your needs — and you’ll only know by trying.

A worked example

Imagine a small legal team choosing an assistant to summarize contracts. The leaderboard leader scores a few points higher on general knowledge benchmarks than two cheaper alternatives. The team runs 25 of its own contracts through all three models with the same prompt and asks two lawyers to score the summaries blind.

The result: one of the cheaper models misses fewer key clauses, follows the requested format more reliably and costs a fraction as much per document. The leaderboard leader writes more fluent summaries but occasionally adds interpretations that aren’t in the contract. For this team, the “best” model on paper is not the best choice.

This pattern is common. General benchmarks reward broad knowledge and reasoning; your task may depend more on instruction following, faithfulness to source documents, format consistency or speed.

What to score in your own test

  • Accuracy: are facts and conclusions correct?
  • Faithfulness: does the output stick to the provided material without inventing?
  • Completeness: are important points missing?
  • Format: does it follow your template reliably?
  • Consistency: do repeated runs give similar quality?
  • Cost and latency: what does each output cost, and how long does it take?

Weight these according to what matters for your use case, and record results in a simple spreadsheet so you can rerun the comparison when new models arrive.

What to watch next

  • New benchmarks designed to resist contamination.
  • Domain-specific evaluations for industries.
  • Tools that make private evaluations easier for non-experts.
  • Changes in how labs report results at launch.

Key takeaways

  • Benchmarks track progress but rarely match your actual tasks.
  • Saturation, contamination and optimization limit their value for buyers.
  • Arena rankings reflect preferences, not necessarily accuracy.
  • A small evaluation on 20–30 real examples beats leaderboard browsing.
  • Use benchmarks to shortlist, then test on your own work.

Frequently asked questions

Which benchmark is most reliable?

None is reliable for every purpose. Choose benchmarks closest to your tasks and complement them with your own tests.

Is the top model on a leaderboard always best?

No. Differences at the top are often small, and other models may be better for your tasks, budget or speed requirements.

How often should I re-evaluate?

Whenever you consider switching models or providers update models significantly.