ARC

CConstruct

Grade-school science knowledge and simple reasoning

PPopulation

Multiple-choice questions from standardized science exams for grades 3–9; the Challenge set keeps questions that 2018 retrieval and word co-occurrence baselines answered incorrectly

Measures

• Basic scientific knowledge at an elementary to middle school level
• Ability to handle science questions that require more than simple keyword matching
• Performance on questions designed to be hard for retrieval-based baselines

Does not measure

• Advanced scientific reasoning — these are grade-school questions that frontier models trivially solve
• Any meaningful differentiation between modern LLMs, all of which score above 95% on ARC-Challenge
• Genuine reasoning ability — despite the name, most questions can be answered through knowledge recall
• Real-world science competence or ability to apply scientific thinking

MMechanism

Deterministic checker

Exact match of the selected option against the key.

Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format

Multiple choice

ΨSpecification

A question is answered correctly when the selected option matches the keyed answer (most questions have four options).

AAnchor

Fixed answer key from the source exams

VEvidence view

The selected option only (a generated letter or the highest-likelihood option, depending on the harness)

ZSignal

Binary correct/incorrect per question

RResolution

Final answer per question

ΓAggregation

Accuracy, usually on the Challenge test split; the Easy set is reported separately

Metrics

accuracy

Size

7,787 questions (Challenge set: 2,590; Challenge test split: 1,172)

Methodology

Drawn from standardized science exams for grades 3 through 9, containing 7,787 questions. The Challenge set (2,590 questions, 1,172 of them in the test split) keeps only those that are answered incorrectly by both a retrieval algorithm and a word co-occurrence algorithm. Evaluation is multiple-choice accuracy, typically reported on the Challenge set.

Typical uses

• Measure · supports inference and decision

Exposure

Public since 2018 and built from standardized test questions widely available online, so heavy contamination is likely. It was part of the Hugging Face Open LLM Leaderboard (v1), which made it a hill-climbing target for open models.

Notes

Not to be confused with ARC-AGI (Chollet, 2019), an unrelated abstraction-and-reasoning benchmark. Check which one a reported "ARC" score refers to, and whether it is the Easy or the Challenge set.

ARC's evidence supported grade-school science question answering for 2018-era systems, but it no longer supports claims about modern LLMs: grade-school difficulty cannot separate systems that all score above 95%, and its continued inclusion in suites such as the Open LLM Leaderboard reflects tradition rather than information. GPQA or MMLU-Pro support more useful knowledge claims.

Evidence notes3 not supported

1. Not supported

   Separates current LLMs

   Competitive modern LLMs all score above 95% on ARC-Challenge.

2. Not supported

   The Challenge filter selects questions that are hard for the systems being evaluated

   The filter removed only questions that a 2018 retrieval baseline and a word co-occurrence baseline answered correctly; it was never calibrated against LLMs.

   Source: Clark et al., 2018

3. Not supported

   Measures scientific reasoning beyond recall

   Despite the name, most questions can be answered through knowledge recall.

Saturation

Saturated

Whether top systems still differ, or crowd the ceiling.

Contamination

High risk

How likely test items, or close variants, have leaked into training data.

Successors

• GPQA
• MMLU-Pro

Last reviewed

Not yet reviewed · card updated October 12, 2026

• MMLUKnowledge · 2021Massive Multitask Language Understanding evaluates broad academic knowledge across 57 subjects ranging from STEM to humanities. It became the de facto standard for measuring general knowledge in LLMs but is now largely superseded by harder variants.
• MMLU-ProKnowledge · 2024A harder, more discriminating successor to MMLU that expands answer choices to 10 options, increases reasoning demands, and reduces noise from trivially easy questions. It addresses several known MMLU weaknesses while retaining broad subject coverage.
• GPQAKnowledge · 2023Graduate-Level Google-Proof Questions & Answers is a dataset of extremely difficult questions written and validated by domain experts. Questions are designed so that even skilled non-experts with internet access struggle to answer correctly.