Knowledge · Evaluator card
TriviaQA
2017Mandar Joshi et al.Paper (opens in a new tab)
A large-scale reading comprehension and trivia knowledge dataset with questions authored by trivia enthusiasts. Originally designed for QA systems with evidence documents, it is now mostly used as a closed-book factual recall benchmark.
Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.
Headline claim
Closed-book TriviaQA accuracy supports only a basic check that small or older models can recall common trivia facts; it carries no information for ranking current LLMs.
At a glance
- Mechanism
- Deterministic checkerPrior: execution feedback, rung 2 of 4
- Item format
- Short answer
- Size
- 95,000+ question-answer pairs
- Typical uses
- Measure
- Saturation
- Saturated
- Contamination
- High risk
- Year
- 2017
1Claim
What property are we trying to infer, over what world, and for what decision?
- Construct
- Recall of trivia-style facts from parametric memory
- Population
- Questions written by trivia enthusiasts (trivia websites and quiz leagues), answered closed-book, without evidence documents
- Measures
- Factual recall of trivia-style knowledge across general domains
- Pattern matching between question phrasing and memorized facts
- Ability to retrieve specific named entities, dates, and facts from parametric memory
- Does not measure
- Understanding, reasoning, or any form of deeper cognition — this is pure recall
- Ability to handle nuanced, ambiguous, or context-dependent questions
- Modern LLM capabilities, as frontier models achieve near-perfect scores
- Real-world information retrieval skills since the closed-book setting is artificial
2Instrument
How is the judgment actually produced?
- Mechanism
Deterministic checker
String normalization (case, punctuation, articles), then exact match or token F1 against the alias list: a weak comparator for correct answers phrased differently from every alias.
Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.
- Item format
- Short answer
- Specification
- An answer is correct when, after normalization, it matches one of the accepted answer aliases (exact match); token-overlap F1 gives partial credit.
- Anchor
- A reference answer with a list of accepted aliases per question
- Evidence view
- The model's short answer string
- Signal
- Exact-match verdict and F1 score per question
- Resolution
- Final answer per question
- Aggregation
- Mean exact match and mean F1 over questions
- Metrics
- exact-match, F1
- Size
- 95,000+ question-answer pairs
- Methodology
- Contains over 95,000 question-answer pairs collected from trivia websites and quiz leagues. In the closed-book setting (most common for LLM evaluation), models must answer without any provided context. Evaluation uses exact-match and F1 score against a set of accepted answer aliases. The open-book version provides Wikipedia or web documents as evidence.
3Regime
What causal role does the signal play, and what happens when a system adapts to it?
- Typical uses
- Measure · supports inference and decision
- Exposure
- Public since 2017 and drawn from trivia websites whose content is abundant in pretraining corpora, so contamination is likely for any modern model. Closed-book use repurposes a dataset designed for reading comprehension over evidence documents.
4ValidationDraft
Why should we believe that the signal means what we claim it means — before and after the system adapts to it?
TriviaQA's evidence supports, at most, a closed-book check of trivia recall in small or older models and comparison with pre-LLM QA systems. It does not support claims about knowledge depth, understanding, or reasoning, and near-perfect scores from frontier models mean it no longer separates them; its continued presence in evaluation suites reflects inertia more than information.
Evidence notes1 partially supported · 1 not supported · 1 untested
- Not supported
Separates current frontier models
Frontier models achieve near-perfect scores, so differences carry no capability information.
- Partially supported
Exact match credits every correct answer
Alias lists are incomplete, so correct answers phrased differently can be marked wrong; F1 softens but does not remove this.
- Untested
Closed-book accuracy indicates useful real-world knowledge
The items reward recall of isolated trivia and the closed-book setup does not match how models are used; nothing links the score to knowledge use in realistic tasks.
5Lifecycle
Does the validity survive over time, and are we still evaluating the right thing?
- Saturation
- Saturated
Whether top systems still differ, or crowd the ceiling.
- Contamination
- High risk
How likely test items, or close variants, have leaked into training data.
- Successors
- None listed
- Last reviewed
- Not yet reviewed · card updated October 12, 2026
Limitations
- Completely saturated for modern LLMs, providing no discriminative signal between frontier models
- Massive contamination risk since trivia questions are abundant in training data
- Trivia-style questions reward memorization of arbitrary facts over useful knowledge
- Answer aliases are incomplete, sometimes penalizing correct but differently-phrased answers
- The closed-book evaluation setup no longer reflects how models are actually used
When to use
- Testing basic factual recall in smaller or older models as a sanity check
- Historical comparison with pre-LLM QA systems where TriviaQA was a standard benchmark
When not to use
- Any modern LLM evaluation — the benchmark is fully saturated and uninformative
- Assessing genuine knowledge depth, understanding, or reasoning ability
- Making claims about a model's real-world knowledge capabilities
Related benchmarks
- MMLUKnowledge · 2021Massive Multitask Language Understanding evaluates broad academic knowledge across 57 subjects ranging from STEM to humanities. It became the de facto standard for measuring general knowledge in LLMs but is now largely superseded by harder variants.
- ARCKnowledge · 2018The AI2 Reasoning Challenge consists of grade-school science questions split into Easy and Challenge sets. The Challenge set filters for questions that simple retrieval and co-occurrence methods fail on.