HumanEval was introduced by OpenAI in July 2021 in “Evaluating Large Language Models Trained on Code”, the paper that presented Codex, the model behind the original GitHub Copilot. Its first author is Mark Chen. For years it was the default way to quote a model’s coding ability in a single number.
What it measures
HumanEval tests “functional correctness”: can a model write a short Python function that actually works? Instead of comparing the generated code with a reference text, which says little about whether it runs, the benchmark executes the code against unit tests. That simple idea, judging code by running it, shaped almost every coding benchmark that followed.
How it works
The benchmark contains 164 hand-written programming problems. Each gives the model a Python function signature and a docstring that describes what the function should do, sometimes with examples. The model completes the function body. The completion is then run against a set of hidden unit tests, and it counts as correct only if all tests pass.
The paper introduced the pass@k metric: the probability that at least one of k generated samples passes the tests. The most commonly reported number today is pass@1, essentially the share of problems solved on the first try. The official repository warns that the evaluation runs untrusted, model-generated code and should only be executed inside a robust security sandbox.
A typical problem might ask for a function that takes a list of numbers and returns whether any two of them are closer to each other than a given threshold, or one that reverses the words in a string while keeping certain characters in place.
Why it matters
HumanEval made progress in AI coding measurable. In the original paper, Codex solved 28.8% of problems on the first attempt, GPT-3 solved 0% and GPT-J 11.4%; drawing 100 samples per problem and picking the right one raised Codex’s success to 70.2%. Those results helped launch the AI coding-assistant industry, and the pass@k idea is still used to report results on harder benchmarks.
Limitations and criticisms
- Saturation. Frontier models now solve nearly every problem, so HumanEval no longer separates strong coding models.
- Small and simple. 164 short, self-contained functions are far from real software work, which involves large codebases, dependencies and ambiguous requirements. Benchmarks such as SWE-bench Verified and LiveCodeBench fill that gap.
- Weak tests. Some problems have few unit tests, so incorrect solutions can pass. EvalPlus’s HumanEval+ (2023) expanded the tests 80-fold and found that pass@k fell by up to 19.3–28.9% under the stricter checks.
- Contamination. The problems have been public since 2021 and are widely copied, so they are likely to appear in training data.
- Python only. The original set covers just one language, though translated variants exist.
Reading the scores
A pass@1 above 90% is now common for capable models, including many open-weight models. At that level, differences of a few points are mostly noise and test quirks. Use HumanEval as a basic sanity check, and rely on harder, fresher benchmarks for serious comparisons.
The official repository is no longer maintained, and there is no single official leaderboard; scores are mostly self-reported by model developers, so compare them with care.