HellaSwag was introduced in May 2019 by Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi and Yejin Choi, and presented at ACL 2019. Its title asks: “Can a Machine Really Finish Your Sentence?” At the time, the answer was clearly no.
What it measures
HellaSwag tests commonsense inference: given the start of a description of an everyday activity, can a model tell which continuation makes sense? Humans find this almost trivial because we know how the physical and social world works. Models that rely on surface patterns in text, the paper argued, can be fooled by endings that look fluent but are nonsensical.
How it works
Each item gives a short context and four possible endings, one correct and three wrong. The contexts come from two sources: video captions from ActivityNet, describing activities such as washing a car or doing a gymnastics routine, and step-by-step how-to articles from WikiHow. Validation and test sets are split into “in-domain” and “zero-shot” categories, depending on whether the activity type appears in the training data.
The wrong endings are produced by adversarial filtering: a series of discriminator models iteratively selects machine-generated endings that are ridiculous to humans but are often misclassified by state-of-the-art models. This makes the benchmark hard for models while staying easy for people. The score is accuracy, and random guessing gives 25%.
A typical item might describe a person on a ladder starting to clean a window, then offer endings such as “wipes the glass with a cloth”, alongside generated distractors like “climbs into the bucket and drives away”.
Why it matters
When HellaSwag came out, humans scored above 95% while state-of-the-art models scored under 48%, a dramatic gap. It became a standard part of language model evaluation suites, and for several years it was one of the most widely reported commonsense numbers in model releases, including on Hugging Face’s original Open LLM Leaderboard.
Limitations and criticisms
- Saturation. The official site lists GPT-4 (base, 10-shot) at 95.3% on the test set, roughly matching humans. Modern models score at or near the ceiling, so HellaSwag no longer differentiates them.
- Closed leaderboard. The test-set leaderboard closed in November 2024, because most people evaluated on the public validation set instead.
- Contamination. The validation set is public and widely used, so it has very likely appeared in training data.
- Dated distractors. The adversarial endings were selected against 2019-era models. They are not adversarial for today’s far stronger systems.
Reading the scores
Human performance is above 95%. Random chance is 25%. Scores in the high 80s or 90s are typical for modern models, and differences at that level carry little meaning. HellaSwag remains useful mainly for evaluating small models, for example during pretraining research, where it can still show whether a model has acquired basic commonsense knowledge.
For how benchmarks like this reach their ceiling, see how AI benchmarks work and why they saturate.