MMLU (Measuring Massive Multitask Language Understanding) was introduced in September 2020 by Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song and Jacob Steinhardt. For several years it was the number most often quoted when a lab launched a new large language model.
What it measures
MMLU tests how much general knowledge and problem-solving ability a model picked up during pretraining. It covers 57 tasks, including elementary mathematics, US history, computer science, law, medicine and many more. The aim was to go beyond narrow language tasks and check whether a model could handle the range of subjects a well-educated person meets at school, university and in professional exams.
How it works
Every question is multiple choice with four options, and the model has to pick one letter. The 15,908 questions were collected by graduate and undergraduate students from freely available sources, including practice questions for the GRE and the US Medical Licensing Examination, and are split into a small few-shot set (5 per subject), a 1,540-question validation set and a 14,079-question test set. Models are usually tested “few-shot”: the prompt shows five solved examples from the same subject, followed by the question to answer. The score is plain accuracy, reported per subject and averaged across all of them.
A typical item might give a short clinical description and ask which of four diagnoses is most likely, or set out a legal scenario and ask which doctrine applies. Because the answer is a single letter, grading is fully automatic.
Why it matters
When it came out, MMLU was hard. The paper reports that the largest GPT-3 model beat random chance (25%) by almost 20 percentage points on average, while most smaller models scored close to chance. That gap left plenty of room to track progress, and MMLU became a common yardstick for comparing models such as GPT-4o with open-weight rivals. Our explainer of the original paper covers the design choices in more depth.
Limitations and criticisms
- Saturation. Frontier models now score so high that the remaining differences are mostly noise. The authors of Humanity’s Last Exam point out that leading models exceed 90% accuracy on popular benchmarks like MMLU, which is why harder successors such as MMLU-Pro and GPQA were built.
- Label errors. The 2024 paper “Are We Done with MMLU?” manually re-annotated 5,700 questions and estimated that 6.49% of MMLU questions contain errors, including 57% of the Virology questions it analysed. That puts a ceiling on meaningful accuracy and blurs comparisons at the top.
- Contamination. The questions have been public since 2020 and are widely copied online, so they may well appear in training data.
- Prompt sensitivity. Small changes to the prompt format, the number of examples or how the answer is extracted can move scores by several points. Numbers from different labs are not always comparable.
- Format. Choosing among four options is easier than producing an answer from scratch, and it says little about writing, coding or agentic work.
Reading the scores
Random guessing gives 25%. In the original paper, crowdworkers on Amazon Mechanical Turk scored 34.5%, and the authors estimated expert-level accuracy at about 89.8%. Today, a score in the high 80s or above is typical of strong models and no longer separates the frontier. For choosing a model, MMLU is best treated as a minimum bar for general knowledge. See how AI benchmarks work and why they saturate for the bigger picture.
Scores move quickly and are mostly self-reported by model developers, so check the official repository and each model’s technical report for current figures.