# GPQA

> Graduate-Level Google-Proof Questions & Answers is a dataset of extremely difficult questions written and validated by domain experts. Questions are designed so that even skilled non-experts with internet access struggle to answer correctly.

## Claim

- **headline**: GPQA accuracy, usually on the 198-question Diamond subset, supports coarse comparisons of frontier models on expert-written biology, physics, and chemistry questions that resist web lookup; gaps of a few points are within sampling noise.
- **construct**: Graduate-level scientific knowledge and reasoning in biology, physics, and chemistry
- **population**: Four-option questions written by domain experts (PhD students and researchers) and kept when skilled non-experts with web access mostly failed them; frontier models
- **measures**: Deep expert-level knowledge in biology, physics, and chemistry; Ability to reason through graduate-level scientific problems that resist simple lookup; Performance on questions validated to be genuinely difficult even for resourceful humans; Whether a model has internalized deep domain understanding versus surface-level associations
- **doesNotMeasure**: Breadth of knowledge — the dataset is narrow, covering only a few scientific domains; Practical scientific reasoning like experimental design, data interpretation, or literature synthesis; Performance on applied or interdisciplinary problems that cross domain boundaries; Ability to explain reasoning or communicate scientific concepts clearly

## Instrument

- **itemFormat**: multiple-choice
- **specification**: A question is answered correctly when the selected option matches the expert-keyed answer among four choices.
- **anchor**: Fixed answer key written by the question's author and checked by expert validators
- **evidenceView**: The selected option only
- **mechanism**: deterministic
- **mechanismDetail**: Exact match of the selected option against the key.
- **signal**: Binary correct/incorrect per question
- **resolution**: Final answer per question
- **aggregation**: Accuracy over the chosen subset; Diamond is the usual reporting set
- **metrics**: accuracy
- **size**: 448 questions (Diamond subset: 198)
- **methodology**: Domain experts (PhD students and researchers) write challenging multiple-choice questions in their area of expertise. Questions are validated by having non-expert humans with internet access attempt them, keeping only questions where non-experts perform poorly. The Diamond subset contains the hardest, most carefully validated questions. Evaluation is exact-match accuracy.

## Regime

- **typicalUses**: measure
- **exposure**: Public since 2023 and distributed with a request not to post examples online, which lowers but does not remove contamination risk. GPQA Diamond is a headline number in frontier-model releases and was part of the Hugging Face Open LLM Leaderboard (v2), so a small item pool is under selection pressure.

## Validation

- **summary**: GPQA's evidence supports it as a carefully constructed measure of expert-level scientific question answering within three domains: experts write the questions, and only questions that resist skilled non-experts with web access are kept. Its main limit is scale — with a few hundred items, score differences of a few points may not be meaningful — and its claims stop at biology, physics, and chemistry.
- **evidence**: Questions resist answer lookup by skilled non-experts; Gaps of a few points reflect capability differences; The answer key is unambiguous; Generalizes to scientific work beyond question answering
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: medium
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/gpqa