GPQA

CConstruct

Graduate-level scientific knowledge and reasoning in biology, physics, and chemistry

PPopulation

Four-option questions written by domain experts (PhD students and researchers) and kept when skilled non-experts with web access mostly failed them; frontier models

Measures

• Deep expert-level knowledge in biology, physics, and chemistry
• Ability to reason through graduate-level scientific problems that resist simple lookup
• Performance on questions validated to be genuinely difficult even for resourceful humans
• Whether a model has internalized deep domain understanding versus surface-level associations

Does not measure

• Breadth of knowledge — the dataset is narrow, covering only a few scientific domains
• Practical scientific reasoning like experimental design, data interpretation, or literature synthesis
• Performance on applied or interdisciplinary problems that cross domain boundaries
• Ability to explain reasoning or communicate scientific concepts clearly

MMechanism

Deterministic checker

Exact match of the selected option against the key.

Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format

Multiple choice

ΨSpecification

A question is answered correctly when the selected option matches the expert-keyed answer among four choices.

AAnchor

Fixed answer key written by the question's author and checked by expert validators

VEvidence view

The selected option only

ZSignal

Binary correct/incorrect per question

RResolution

Final answer per question

ΓAggregation

Accuracy over the chosen subset; Diamond is the usual reporting set

Metrics

accuracy

Size

448 questions (Diamond subset: 198)

Methodology

Domain experts (PhD students and researchers) write challenging multiple-choice questions in their area of expertise. Questions are validated by having non-expert humans with internet access attempt them, keeping only questions where non-experts perform poorly. The Diamond subset contains the hardest, most carefully validated questions. Evaluation is exact-match accuracy.

Typical uses

• Measure · supports inference and decision

Exposure

Public since 2023 and distributed with a request not to post examples online, which lowers but does not remove contamination risk. GPQA Diamond is a headline number in frontier-model releases and was part of the Hugging Face Open LLM Leaderboard (v2), so a small item pool is under selection pressure.

GPQA's evidence supports it as a carefully constructed measure of expert-level scientific question answering within three domains: experts write the questions, and only questions that resist skilled non-experts with web access are kept. Its main limit is scale — with a few hundred items, score differences of a few points may not be meaningful — and its claims stop at biology, physics, and chemistry.

Evidence notes1 supported · 1 partially supported · 1 not supported · 1 untested

1. Supported

   Questions resist answer lookup by skilled non-experts

   In the authors' validation, experts in the question's field answered far more questions correctly than highly skilled non-experts who spent substantial time with unrestricted web access.

   Source: Rein et al., 2023

2. Not supported

   Gaps of a few points reflect capability differences

   Binomial arithmetic: with 198 Diamond questions, a 95% interval around 50% accuracy spans about ±7 points, so small gaps are within sampling noise.

3. Partially supported

   The answer key is unambiguous

   Expert validators did not always agree with the key; the Diamond subset applies stricter expert-agreement filters, which reduces but need not eliminate ambiguous items.

   Source: Rein et al., 2023

4. Untested

   Generalizes to scientific work beyond question answering

   Experimental design, data interpretation, literature synthesis, and interdisciplinary problems are outside the item pool.

Saturation

Active

Whether top systems still differ, or crowd the ceiling.

Contamination

Medium risk

How likely test items, or close variants, have leaked into training data.

Successors

None listed

Last reviewed

Not yet reviewed · card updated October 12, 2026

• MMLUKnowledge · 2021Massive Multitask Language Understanding evaluates broad academic knowledge across 57 subjects ranging from STEM to humanities. It became the de facto standard for measuring general knowledge in LLMs but is now largely superseded by harder variants.
• MMLU-ProKnowledge · 2024A harder, more discriminating successor to MMLU that expands answer choices to 10 options, increases reasoning demands, and reduces noise from trivially easy questions. It addresses several known MMLU weaknesses while retaining broad subject coverage.
• ARCKnowledge · 2018The AI2 Reasoning Challenge consists of grade-school science questions split into Easy and Challenge sets. The Challenge set filters for questions that simple retrieval and co-occurrence methods fail on.