SimpleQA was released by OpenAI in November 2024 in the paper “Measuring short-form factuality in large language models”, by Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman and William Fedus. The authors describe it as a test of whether models “know what they know”.
What it measures
SimpleQA targets one specific kind of hallucination: confidently stating a wrong fact. Checking the truth of long, open-ended answers is hard, so the benchmark deliberately narrows the problem to short questions with a single correct answer. A good model should answer as many as possible correctly and, crucially, decline to answer when it is not confident rather than guessing.
How it works
The dataset contains 4,326 questions covering a wide range of topics, including history, science and technology, art, geography and TV shows. Questions were written by human annotators (“AI trainers”) under strict rules:
- each question must have a single, indisputable answer, with its scope spelled out (asking “which city”, not “where”);
- the answer must not change over time;
- the answer must be supported by evidence, with a link provided by the writer;
- the question must be hard: at least one of four OpenAI model answers had to be wrong.
A second trainer then answered each question independently, and only questions where both trainers agreed were kept. Because questions were collected adversarially against GPT-4-class models, the set is challenging for frontier systems.
Each model answer is graded by a prompted ChatGPT grader as correct, incorrect or not attempted. The headline figure is usually the share correct, but the paper also reports the share correct among attempted answers and an F-score that combines the two. A typical question asks for a precise, obscure detail, such as the year a particular minor award was first given or the name of a specific character in one season of a TV series.
Why it matters
Factual reliability is one of the main barriers to trusting AI assistants. SimpleQA gives a cheap, low-variance way to track it: with over 4,000 questions and very short answers, it is fast to run and grade. In the original paper, both GPT-4o and Claude models scored under 50%. The paper also showed that Claude models declined to answer more often: Claude 3.5 Sonnet got far fewer questions right than GPT-4o but attempted far fewer, ending with a similar F-score. It also studied calibration: models consistently overstated their confidence, though larger models were better calibrated.
Limitations and criticisms
- Narrow scope. Short questions with one answer say nothing about long-form factuality, where most real hallucinations occur. The authors flag that as an open question.
- Rewards memorising trivia. Without tools, the score mostly reflects how much obscure knowledge a model has stored, which favours larger models; with web search enabled, the test becomes a measure of search, not memory. Check which setting was used.
- Adversarial against one family. Questions were filtered against OpenAI models, which might make the set slightly harder for them than for others; the paper checked this with Claude and found it challenging too.
- Archived reference code. OpenAI’s simple-evals repository was deprecated in July 2025 for new results, though it still hosts the SimpleQA reference implementation.
Reading the scores
The authors estimate the dataset’s own error rate at around 3%, which sets a practical ceiling. Look at both accuracy and the not-attempted rate: a model that says “I don’t know” more often may have lower accuracy but far fewer hallucinations, which can be the better trade-off. Scores without tools and with search are not comparable. There is no single official live leaderboard; check model cards and independent evaluators for current figures.