Skip to content
Benchmaker

Reasoning · Evaluator card

HellaSwag

2019Rowan Zellers et al.Paper (opens in a new tab)

A commonsense natural language inference benchmark where models must choose the most plausible continuation of an everyday scenario. It was designed using adversarial filtering to be hard for models at the time of creation.

View as Markdown

Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.

Headline claim

HellaSwag accuracy supports only a check that small models prefer plausible continuations of everyday activity descriptions; it carries no information for ranking current LLMs.

At a glance

Mechanism
Deterministic checkerPrior: execution feedback, rung 2 of 4
Item format
Multiple choice
Size
~70,000 questions (10,000 test)
Typical uses
Measure
Saturation
Saturated
Contamination
High risk
Year
2019

1Claim

What property are we trying to infer, over what world, and for what decision?

Construct
Commonsense inference about how everyday activities unfold
Population
Four-way sentence-completion items built from ActivityNet captions and WikiHow articles, with machine-generated distractors filtered against 2019-era models
Measures
  • Commonsense understanding of everyday physical and social scenarios
  • Ability to distinguish plausible from implausible continuations of activity descriptions
  • Grounded language understanding connected to real-world knowledge of how activities proceed
Does not measure
  • Any meaningful capability in modern LLMs — the benchmark is completely saturated above 95%
  • Genuine commonsense reasoning, as high scores can be achieved through surface-level language patterns
  • Ability to reason about novel situations, edge cases, or scenarios outside the training distribution
  • Complex multi-step reasoning or planning that real commonsense often requires
  • Understanding of physical causality beyond the statistical regularities in the training data

2Instrument

How is the judgment actually produced?

Mechanism

Deterministic checker

Exact match of the chosen ending against the key.

Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format
Multiple choice
Specification
An item is correct when the model picks the human-written continuation over three adversarially filtered, machine-generated endings.
Anchor
Fixed answer key: the original human-written continuation
Evidence view
The chosen ending; in most LLM harnesses this is the ending with the highest length-normalized log-likelihood, so no generated text is inspected
Signal
Binary correct/incorrect per item
Resolution
Final choice per item
Aggregation
Accuracy; most LLM harnesses report the roughly 10,000-item validation split
Metrics
accuracy
Size
~70,000 questions (10,000 test)
Methodology
Contains approximately 70,000 multiple-choice questions derived from ActivityNet and WikiHow descriptions. Each question presents a scenario beginning and four possible continuations, one correct and three adversarially generated by a language model and filtered to fool earlier models. Evaluation is accuracy on the correct continuation choice.

3Regime

What causal role does the signal play, and what happens when a system adapts to it?

Typical uses
  • Measure · supports inference and decision
Exposure
Public since 2019; ActivityNet and WikiHow text is likely well represented in training corpora. It was part of the Hugging Face Open LLM Leaderboard (v1), which made it a hill-climbing target for open models.

4ValidationDraft

Why should we believe that the signal means what we claim it means — before and after the system adapts to it?

HellaSwag's adversarial-filtering design was influential, but its evidence no longer supports claims about current models: near-perfect scores across frontier models say nothing about relative quality, and its survival on leaderboards lowers evaluation standards. Commonsense reasoning claims need harder, more recent tests.

Evidence notes1 contested · 2 not supported

  1. Not supported

    Separates current LLMs

    Modern LLMs consistently score above 95%.

  2. Not supported

    Adversarial filtering keeps items hard for the models being evaluated

    Distractors were filtered against 2019-era models and no longer produce hard examples.

  3. Contested

    Measures commonsense reasoning rather than familiarity with activity scripts

    High scores are reachable through surface-level language patterns and familiarity with common activity descriptions, which the format cannot separate from commonsense inference.

5Lifecycle

Does the validity survive over time, and are we still evaluating the right thing?

Saturation
Saturated

Whether top systems still differ, or crowd the ceiling.

Contamination
High risk

How likely test items, or close variants, have leaked into training data.

Successors
None listed
Last reviewed
Not yet reviewed · card updated October 12, 2026

Limitations

  • Fully saturated for all modern LLMs, with scores consistently above 95%
  • Adversarial filtering was calibrated against 2019-era models and no longer produces hard examples
  • ActivityNet and WikiHow source data is likely well-represented in modern training corpora
  • The 4-choice format is lenient and allows partial credit from elimination
  • Measures familiarity with common activity scripts more than genuine commonsense reasoning

When to use

  • Evaluating very small models where commonsense language understanding is not guaranteed
  • Historical comparison with older systems for tracking progress over time

When not to use

  • Any evaluation involving models from 2023 onward — scores are at ceiling
  • Making claims about a model's commonsense reasoning ability
  • Differentiating between competitive modern language models

Related benchmarks