# TriviaQA

> A large-scale reading comprehension and trivia knowledge dataset with questions authored by trivia enthusiasts. Originally designed for QA systems with evidence documents, it is now mostly used as a closed-book factual recall benchmark.

## Claim

- **headline**: Closed-book TriviaQA accuracy supports only a basic check that small or older models can recall common trivia facts; it carries no information for ranking current LLMs.
- **construct**: Recall of trivia-style facts from parametric memory
- **population**: Questions written by trivia enthusiasts (trivia websites and quiz leagues), answered closed-book, without evidence documents
- **measures**: Factual recall of trivia-style knowledge across general domains; Pattern matching between question phrasing and memorized facts; Ability to retrieve specific named entities, dates, and facts from parametric memory
- **doesNotMeasure**: Understanding, reasoning, or any form of deeper cognition — this is pure recall; Ability to handle nuanced, ambiguous, or context-dependent questions; Modern LLM capabilities, as frontier models achieve near-perfect scores; Real-world information retrieval skills since the closed-book setting is artificial

## Instrument

- **itemFormat**: short-answer
- **specification**: An answer is correct when, after normalization, it matches one of the accepted answer aliases (exact match); token-overlap F1 gives partial credit.
- **anchor**: A reference answer with a list of accepted aliases per question
- **evidenceView**: The model's short answer string
- **mechanism**: deterministic
- **mechanismDetail**: String normalization (case, punctuation, articles), then exact match or token F1 against the alias list: a weak comparator for correct answers phrased differently from every alias.
- **signal**: Exact-match verdict and F1 score per question
- **resolution**: Final answer per question
- **aggregation**: Mean exact match and mean F1 over questions
- **metrics**: exact-match; F1
- **size**: 95,000+ question-answer pairs
- **methodology**: Contains over 95,000 question-answer pairs collected from trivia websites and quiz leagues. In the closed-book setting (most common for LLM evaluation), models must answer without any provided context. Evaluation uses exact-match and F1 score against a set of accepted answer aliases. The open-book version provides Wikipedia or web documents as evidence.

## Regime

- **typicalUses**: measure
- **exposure**: Public since 2017 and drawn from trivia websites whose content is abundant in pretraining corpora, so contamination is likely for any modern model. Closed-book use repurposes a dataset designed for reading comprehension over evidence documents.

## Validation

- **summary**: TriviaQA's evidence supports, at most, a closed-book check of trivia recall in small or older models and comparison with pre-LLM QA systems. It does not support claims about knowledge depth, understanding, or reasoning, and near-perfect scores from frontier models mean it no longer separates them; its continued presence in evaluation suites reflects inertia more than information.
- **evidence**: Separates current frontier models; Exact match credits every correct answer; Closed-book accuracy indicates useful real-world knowledge
- **status**: draft

## Lifecycle

- **saturation**: saturated
- **contamination**: high
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/triviaqa