MMLU

CConstruct

Breadth of academic knowledge, operationalized as recognizing the keyed option on exam-style questions

PPopulation

English four-option questions from practice exams, textbooks, and academic materials (elementary to professional level), answered zero- or few-shot by base and chat models

Measures

• Breadth of factual knowledge across academic disciplines
• Ability to apply domain-specific knowledge to multiple-choice questions
• Performance consistency across diverse subject areas from abstract algebra to world religions
• Baseline competence at undergraduate and professional-level exam questions

Does not measure

• Genuine understanding or reasoning — models can score highly through pattern matching and memorization of training data that overlaps with test questions
• Ability to generate, explain, or apply knowledge in open-ended contexts
• Robustness to question rephrasing — small perturbations can cause large score swings
• Real-world problem-solving ability, which requires integrating knowledge with judgment

MMechanism

Deterministic checker

Exact match of the selected option against the key, with no learned judgment. The prompt format, number of shots, and answer-extraction rule belong to the instrument and change scores.

Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format

Multiple choice

ΨSpecification

A question is answered correctly when the model's selected option matches the keyed answer among four choices.

AAnchor

Fixed answer key: one keyed option per question, taken from the source materials

VEvidence view

The selected option only (a generated letter or the highest-likelihood option, depending on the harness); no reasoning is inspected

ZSignal

Binary correct/incorrect per question

RResolution

Final answer per question

ΓAggregation

Accuracy over all questions, with per-subject breakdowns; some harnesses report the mean of per-subject accuracies instead

Metrics

accuracy

Size

14,042 questions across 57 subjects

Methodology

Presents 14,042 multiple-choice questions across 57 subjects in a zero-shot or few-shot setting. Questions are sourced from practice exams, textbooks, and academic materials at varying difficulty levels from elementary to professional. Evaluation is exact-match accuracy on the selected answer choice.

Typical uses

• Measure · supports inference and decision
• Optimize · shapes the candidate or policy distribution

Exposure

Public test set with answer keys, and questions widely reproduced on the web; contamination is well documented. For years MMLU was the headline knowledge number in model releases and on the Hugging Face Open LLM Leaderboard (v1), which made it a hill-climbing target, so its anchor has been spent many times over.

Notes

Compare scores only within a fixed harness: prompt format, few-shot setup, and whether the answer is scored by letter likelihood or by generation all move the number.

MMLU's evidence supports it as an early, broad screen of exam-style knowledge and as a historical reference point. It does not support claims about understanding or general intelligence, or rankings among frontier models: contamination is well documented, the multiple-choice format is forgiving, and near-ceiling scores leave little range to discriminate.

Evidence notes1 partially supported · 1 contested · 2 not supported

1. Not supported

   Separates frontier models on broad academic knowledge

   Leading models cluster near the ceiling, where differences above roughly 85% are mostly noise rather than capability gaps.

2. Partially supported

   The answer key is a reliable anchor across all 57 subjects

   A manual re-annotation (MMLU-Redux) found wrong keys, unclear questions, and missing correct options, concentrated in some subjects; over half of the re-annotated Virology questions had errors.

   Source: Gema et al., 2024 (Are We Done with MMLU?)

3. Not supported

   Scores are comparable across evaluation harnesses

   Popular implementations differed in prompt format and answer scoring, and gave materially different scores, and different rankings, for the same models.

   Source: Hugging Face, 2023 (What's going on with the Open LLM Leaderboard?)

4. Contested

   Scores reflect knowledge rather than memorized test items

   Questions appear verbatim in web-scraped corpora and models can score through memorization and pattern matching; how much this inflates a given model's score is rarely measured.

Saturation

Near saturation

Whether top systems still differ, or crowd the ceiling.

Contamination

High risk

How likely test items, or close variants, have leaked into training data.

Successors

• MMLU-Pro

Last reviewed

Not yet reviewed · card updated October 12, 2026

• MMLU-ProKnowledge · 2024A harder, more discriminating successor to MMLU that expands answer choices to 10 options, increases reasoning demands, and reduces noise from trivially easy questions. It addresses several known MMLU weaknesses while retaining broad subject coverage.
• GPQAKnowledge · 2023Graduate-Level Google-Proof Questions & Answers is a dataset of extremely difficult questions written and validated by domain experts. Questions are designed so that even skilled non-experts with internet access struggle to answer correctly.
• ARCKnowledge · 2018The AI2 Reasoning Challenge consists of grade-school science questions split into Easy and Challenge sets. The Challenge set filters for questions that simple retrieval and co-occurrence methods fail on.