# HellaSwag

> A commonsense natural language inference benchmark where models must choose the most plausible continuation of an everyday scenario. It was designed using adversarial filtering to be hard for models at the time of creation.

## Claim

- **headline**: HellaSwag accuracy supports only a check that small models prefer plausible continuations of everyday activity descriptions; it carries no information for ranking current LLMs.
- **construct**: Commonsense inference about how everyday activities unfold
- **population**: Four-way sentence-completion items built from ActivityNet captions and WikiHow articles, with machine-generated distractors filtered against 2019-era models
- **measures**: Commonsense understanding of everyday physical and social scenarios; Ability to distinguish plausible from implausible continuations of activity descriptions; Grounded language understanding connected to real-world knowledge of how activities proceed
- **doesNotMeasure**: Any meaningful capability in modern LLMs — the benchmark is completely saturated above 95%; Genuine commonsense reasoning, as high scores can be achieved through surface-level language patterns; Ability to reason about novel situations, edge cases, or scenarios outside the training distribution; Complex multi-step reasoning or planning that real commonsense often requires; Understanding of physical causality beyond the statistical regularities in the training data

## Instrument

- **itemFormat**: multiple-choice
- **specification**: An item is correct when the model picks the human-written continuation over three adversarially filtered, machine-generated endings.
- **anchor**: Fixed answer key: the original human-written continuation
- **evidenceView**: The chosen ending; in most LLM harnesses this is the ending with the highest length-normalized log-likelihood, so no generated text is inspected
- **mechanism**: deterministic
- **mechanismDetail**: Exact match of the chosen ending against the key.
- **signal**: Binary correct/incorrect per item
- **resolution**: Final choice per item
- **aggregation**: Accuracy; most LLM harnesses report the roughly 10,000-item validation split
- **metrics**: accuracy
- **size**: ~70,000 questions (10,000 test)
- **methodology**: Contains approximately 70,000 multiple-choice questions derived from ActivityNet and WikiHow descriptions. Each question presents a scenario beginning and four possible continuations, one correct and three adversarially generated by a language model and filtered to fool earlier models. Evaluation is accuracy on the correct continuation choice.

## Regime

- **typicalUses**: measure
- **exposure**: Public since 2019; ActivityNet and WikiHow text is likely well represented in training corpora. It was part of the Hugging Face Open LLM Leaderboard (v1), which made it a hill-climbing target for open models.

## Validation

- **summary**: HellaSwag's adversarial-filtering design was influential, but its evidence no longer supports claims about current models: near-perfect scores across frontier models say nothing about relative quality, and its survival on leaderboards lowers evaluation standards. Commonsense reasoning claims need harder, more recent tests.
- **evidence**: Separates current LLMs; Adversarial filtering keeps items hard for the models being evaluated; Measures commonsense reasoning rather than familiarity with activity scripts
- **status**: draft

## Lifecycle

- **saturation**: saturated
- **contamination**: high
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/hellaswag