# ARC

> The AI2 Reasoning Challenge consists of grade-school science questions split into Easy and Challenge sets. The Challenge set filters for questions that simple retrieval and co-occurrence methods fail on.

## Claim

- **headline**: ARC-Challenge accuracy supports only a basic check that very small or constrained models can answer grade-school science multiple-choice questions; it carries no information for ranking current LLMs.
- **construct**: Grade-school science knowledge and simple reasoning
- **population**: Multiple-choice questions from standardized science exams for grades 3–9; the Challenge set keeps questions that 2018 retrieval and word co-occurrence baselines answered incorrectly
- **measures**: Basic scientific knowledge at an elementary to middle school level; Ability to handle science questions that require more than simple keyword matching; Performance on questions designed to be hard for retrieval-based baselines
- **doesNotMeasure**: Advanced scientific reasoning — these are grade-school questions that frontier models trivially solve; Any meaningful differentiation between modern LLMs, all of which score above 95% on ARC-Challenge; Genuine reasoning ability — despite the name, most questions can be answered through knowledge recall; Real-world science competence or ability to apply scientific thinking

## Instrument

- **itemFormat**: multiple-choice
- **specification**: A question is answered correctly when the selected option matches the keyed answer (most questions have four options).
- **anchor**: Fixed answer key from the source exams
- **evidenceView**: The selected option only (a generated letter or the highest-likelihood option, depending on the harness)
- **mechanism**: deterministic
- **mechanismDetail**: Exact match of the selected option against the key.
- **signal**: Binary correct/incorrect per question
- **resolution**: Final answer per question
- **aggregation**: Accuracy, usually on the Challenge test split; the Easy set is reported separately
- **metrics**: accuracy
- **size**: 7,787 questions (Challenge set: 2,590; Challenge test split: 1,172)
- **methodology**: Drawn from standardized science exams for grades 3 through 9, containing 7,787 questions. The Challenge set (2,590 questions, 1,172 of them in the test split) keeps only those that are answered incorrectly by both a retrieval algorithm and a word co-occurrence algorithm. Evaluation is multiple-choice accuracy, typically reported on the Challenge set.

## Regime

- **typicalUses**: measure
- **exposure**: Public since 2018 and built from standardized test questions widely available online, so heavy contamination is likely. It was part of the Hugging Face Open LLM Leaderboard (v1), which made it a hill-climbing target for open models.
- **notes**: Not to be confused with ARC-AGI (Chollet, 2019), an unrelated abstraction-and-reasoning benchmark. Check which one a reported "ARC" score refers to, and whether it is the Easy or the Challenge set.

## Validation

- **summary**: ARC's evidence supported grade-school science question answering for 2018-era systems, but it no longer supports claims about modern LLMs: grade-school difficulty cannot separate systems that all score above 95%, and its continued inclusion in suites such as the Open LLM Leaderboard reflects tradition rather than information. GPQA or MMLU-Pro support more useful knowledge claims.
- **evidence**: Separates current LLMs; The Challenge filter selects questions that are hard for the systems being evaluated; Measures scientific reasoning beyond recall
- **status**: draft

## Lifecycle

- **saturation**: saturated
- **contamination**: high
- **successors**: gpqa; mmlu-pro
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/arc