Humanity’s Last Exam (HLE) was first released in January 2025 by the Center for AI Safety (CAIS) and Scale AI. The name is deliberately provocative: it was meant to be the final closed-ended academic exam needed before models simply outgrow tests of this kind.
What it measures
HLE tests knowledge and reasoning at the very edge of what human specialists know, across dozens of subjects including mathematics, the natural sciences and the humanities. Its creators built it because, as the paper notes, frontier models were already exceeding 90% on popular benchmarks such as MMLU, leaving no room to measure further progress.
How it works
The questions were written by nearly 1,000 subject experts from more than 500 institutions in 50 countries. The final set was fixed at 2,500 public questions in April 2025. According to Scale AI’s leaderboard, about 24% of questions are multiple choice and the rest need an exact short answer; about 14% require reading a diagram or figure, which makes part of the test multimodal. A text-only subset is used for models that cannot read images.
Candidate questions were first tested against several frontier models and rejected if the models could answer them; about 13,000 survivors then went through two rounds of expert review. Answers are graded automatically: a judge model (o3-mini, according to the official site) compares the model’s final answer with the reference. Alongside accuracy, the leaderboard reports calibration error: models state how confident they are, and a well-calibrated model that is wrong should also be unsure. A typical question might ask for a specific numeric result in an advanced proof or a precise detail of an obscure linguistic or biological system.
Why it matters
HLE is one of the few broad knowledge benchmarks where frontier models still have plenty of room to improve. It has become a headline number in model launches and is often run both with and without tools such as web search and code execution. The paper has been revised many times since its first version, reflecting ongoing corrections to the dataset.
Limitations and criticisms
- Adversarial selection. Questions were filtered to be ones that models of early 2025 got wrong, which can exaggerate how much newer models appear to improve.
- Answer-key errors. At this level of difficulty, some reference answers have been disputed, and the dataset has gone through corrections. In September 2026 CAIS released HLE-Rolling, a fork intended as a migration path as the original ages.
- Tool use changes everything. Scores with search and code tools are not comparable with scores without them; always check which setting is reported.
- Narrow format. Short, closed-form answers test knowledge and reasoning, not open-ended research ability.
Reading the scores
Random-chance performance is close to zero for exact-answer questions. The scores above come from Scale AI’s Humanity’s Last Exam leaderboard, evaluated on all 2,500 public questions, as checked on 23 October 2026; each comes with a confidence interval of roughly ±2 points, so the entries ranked second to fourth are effectively tied. Scale also maintains separate text-only and preview leaderboards. Results move quickly, so check the official leaderboard for the latest figures.
For context on why scores on tests like this tend to climb fast, see how AI benchmarks work and why they saturate.