Knowledge · Evaluator card
MMLU-Pro
2024Yubo Wang et al.Paper (opens in a new tab)
A harder, more discriminating successor to MMLU that expands answer choices to 10 options, increases reasoning demands, and reduces noise from trivially easy questions. It addresses several known MMLU weaknesses while retaining broad subject coverage.
Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.
Headline claim
MMLU-Pro accuracy supports comparing strong models on STEM-weighted, reasoning-heavy ten-option knowledge questions under a fixed chain-of-thought protocol; it does not support claims about balanced knowledge breadth or open-ended expertise.
At a glance
- Mechanism
- Deterministic checkerPrior: execution feedback, rung 2 of 4
- Item format
- Multiple choice
- Size
- ~12,000 questions
- Typical uses
- Measure
- Saturation
- Active
- Contamination
- Medium risk
- Year
- 2024
1Claim
What property are we trying to infer, over what world, and for what decision?
- Construct
- Knowledge application that requires reasoning beyond recall, across a STEM-weighted range of academic subjects
- Population
- About 12,000 ten-option questions filtered from MMLU and augmented from additional sources, answered with chain-of-thought prompting by strong models
- Measures
- Knowledge application under conditions requiring more reasoning than standard MMLU
- Robustness against elimination-based guessing thanks to 10 answer choices instead of 4
- Ability to distinguish between closely related concepts in STEM-heavy domains
- Performance on questions that require multi-step thinking, not just recall
- Does not measure
- Open-ended generation or explanation ability — still fundamentally a multiple-choice test
- True depth of expertise — questions are harder than MMLU but still bounded by the MCQ format
- Practical problem-solving or the ability to use knowledge in realistic workflows
- Calibration or uncertainty awareness in the model's responses
2Instrument
How is the judgment actually produced?
- Mechanism
Deterministic checker
Rule-based extraction of the final option letter from the response, then exact match against the key.
Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.
- Item format
- Multiple choice
- Specification
- A question is answered correctly when the option extracted from the model's chain-of-thought response matches the keyed answer among ten choices.
- Anchor
- Fixed answer key: one keyed option among ten per question
- Evidence view
- The final option extracted from the response; the reasoning is not graded
- Signal
- Binary correct/incorrect per question
- Resolution
- Final answer per question
- Aggregation
- Accuracy over all questions, with per-subject breakdowns
- Metrics
- accuracy
- Size
- ~12,000 questions
- Methodology
- Contains approximately 12,000 questions filtered and augmented from MMLU and additional sources. Each question has 10 answer choices to reduce guessing advantage. Questions skew toward STEM subjects and require more reasoning steps than the original MMLU. Evaluation uses chain-of-thought prompting with exact-match accuracy.
3Regime
What causal role does the signal play, and what happens when a system adapts to it?
- Typical uses
- Measure · supports inference and decision
- Exposure
- Public test set with answer keys, partly derived from MMLU, so some items share MMLU's exposure. It replaced MMLU on the Hugging Face Open LLM Leaderboard (v2) and is a common headline number in model releases, which puts it under selection pressure.
- Notes
- Fix the prompting protocol (chain-of-thought or direct answer, number of shots) before comparing; the choice moves scores substantially.
4ValidationDraft
Why should we believe that the signal means what we claim it means — before and after the system adapts to it?
MMLU-Pro's evidence supports it as a more discriminating successor to MMLU for broad knowledge evaluation: ten answer options and harder, reasoning-heavy questions restore spread among strong models. It inherits the structural limits of multiple-choice testing, and its STEM skew means claims about overall knowledge breadth need complementary evaluations.
Evidence notes1 supported · 2 partially supported
- Supported
Restores spread among models that saturate MMLU
The same models scored substantially lower than on MMLU at release, and ten options lower the chance baseline from 25% to 10%.
Source: Wang et al., 2024
- Partially supported
Scores are stable across prompt variations
The authors report lower sensitivity to prompt wording than MMLU, but chain-of-thought versus direct answering still shifts scores substantially, so the protocol must be fixed.
Source: Wang et al., 2024
- Partially supported
Represents knowledge breadth across disciplines
The question pool skews toward STEM, so the aggregate under-weights the humanities and other non-STEM domains.
5Lifecycle
Does the validity survive over time, and are we still evaluating the right thing?
- Saturation
- Active
Whether top systems still differ, or crowd the ceiling.
- Contamination
- Medium risk
How likely test items, or close variants, have leaked into training data.
- Successors
- None listed
- Last reviewed
- Not yet reviewed · card updated October 12, 2026
Limitations
- STEM-heavy distribution means it is less balanced across knowledge domains than the original MMLU
- Still susceptible to contamination, though the expanded options and newer questions help
- Chain-of-thought dependence means scores vary significantly based on prompting strategy
- 10-option MCQ is still a constrained format that does not test generative knowledge
When to use
- Comparing frontier models on knowledge tasks where original MMLU scores are saturated
- Evaluating whether a model can handle questions requiring reasoning beyond pure recall
- Benchmarking chain-of-thought prompting effectiveness across knowledge domains
When not to use
- Assessing model capability on non-STEM or humanities-heavy knowledge domains
- When you need a measure of open-ended generation or explanation quality
Related benchmarks
- MMLUKnowledge · 2021Massive Multitask Language Understanding evaluates broad academic knowledge across 57 subjects ranging from STEM to humanities. It became the de facto standard for measuring general knowledge in LLMs but is now largely superseded by harder variants.
- GPQAKnowledge · 2023Graduate-Level Google-Proof Questions & Answers is a dataset of extremely difficult questions written and validated by domain experts. Questions are designed so that even skilled non-experts with internet access struggle to answer correctly.
- ARCKnowledge · 2018The AI2 Reasoning Challenge consists of grade-school science questions split into Easy and Challenge sets. The Challenge set filters for questions that simple retrieval and co-occurrence methods fail on.