SWE-bench was introduced in October 2023 by Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press and Karthik Narasimhan, and accepted at ICLR 2024. SWE-bench Verified is a cleaned-up 500-task subset released in 2024 in collaboration with OpenAI. It became the most quoted measure of AI coding agents.
What it measures
Instead of asking a model to write a small function from scratch, as HumanEval does, SWE-bench asks it to do what software engineers actually spend their days on: understand an existing codebase, find the cause of a reported problem and change the code to fix it. That requires navigating many files, reading tests and making edits that do not break anything else.
How it works
The original SWE-bench contains 2,294 task instances drawn from real GitHub issues and the pull requests that resolved them, across 12 popular Python repositories. For each task, the system receives the repository at the commit before the fix and the text of the issue. It must produce a patch. The patch is applied and the repository’s tests are run: the task counts as resolved only if the tests that the real fix made pass now pass, and previously passing tests still pass.
Many original tasks turned out to be underspecified or to have tests that rejected valid solutions. SWE-bench Verified keeps 500 issue–pull request pairs that human annotators validated for quality: the issue is clear enough to solve and the tests are fair. The score is the percentage of the 500 tasks resolved.
A typical task might be a bug report saying that a date-parsing function in a well-known library mishandles a particular time-zone format, with the system expected to locate the function, fix the logic and leave the rest of the test suite green.
Results depend heavily on the “scaffold”, the agent software that lets the model browse files, run commands and test its changes. The official site therefore lists several leaderboards, including a “bash only” view that compares models inside a minimal common agent.
Why it matters
When SWE-bench launched, the best model in the paper, Claude 2, resolved just 1.96% of issues. The climb from there to most of the Verified set became one of the defining storylines of AI coding agents, and the benchmark drove the design of many coding tools.
Limitations and criticisms
- Saturation and archiving. Top systems now resolve a large majority of Verified tasks. Independent evaluator Vals AI has archived its SWE-bench Verified leaderboard, and Epoch AI has published methodology changes, including the exclusion of some problematic samples.
- Contamination. The issues and fixes are public on GitHub, and many predate current models’ training data.
- Python only, narrow repositories. Twelve Python projects are not representative of all software. Newer variants, such as SWE-bench Multimodal, broaden the scope.
- Scaffold confusion. A headline number mixes the model’s skill with the agent framework, the number of attempts and the compute budget. Compare like with like.
Reading the scores
Check which leaderboard a number comes from (bash-only, a vendor’s own agent, or another harness) and whether multiple attempts were allowed. Small differences near the top are within noise for a 500-task set, where each task is worth 0.2 points. Scores change constantly; see the official SWE-bench leaderboards for current results.
For background on benchmark saturation, read how AI benchmarks work and why they saturate.