# MMLU-Pro

> A harder, more discriminating successor to MMLU that expands answer choices to 10 options, increases reasoning demands, and reduces noise from trivially easy questions. It addresses several known MMLU weaknesses while retaining broad subject coverage.

## Claim

- **headline**: MMLU-Pro accuracy supports comparing strong models on STEM-weighted, reasoning-heavy ten-option knowledge questions under a fixed chain-of-thought protocol; it does not support claims about balanced knowledge breadth or open-ended expertise.
- **construct**: Knowledge application that requires reasoning beyond recall, across a STEM-weighted range of academic subjects
- **population**: About 12,000 ten-option questions filtered from MMLU and augmented from additional sources, answered with chain-of-thought prompting by strong models
- **measures**: Knowledge application under conditions requiring more reasoning than standard MMLU; Robustness against elimination-based guessing thanks to 10 answer choices instead of 4; Ability to distinguish between closely related concepts in STEM-heavy domains; Performance on questions that require multi-step thinking, not just recall
- **doesNotMeasure**: Open-ended generation or explanation ability — still fundamentally a multiple-choice test; True depth of expertise — questions are harder than MMLU but still bounded by the MCQ format; Practical problem-solving or the ability to use knowledge in realistic workflows; Calibration or uncertainty awareness in the model's responses

## Instrument

- **itemFormat**: multiple-choice
- **specification**: A question is answered correctly when the option extracted from the model's chain-of-thought response matches the keyed answer among ten choices.
- **anchor**: Fixed answer key: one keyed option among ten per question
- **evidenceView**: The final option extracted from the response; the reasoning is not graded
- **mechanism**: deterministic
- **mechanismDetail**: Rule-based extraction of the final option letter from the response, then exact match against the key.
- **signal**: Binary correct/incorrect per question
- **resolution**: Final answer per question
- **aggregation**: Accuracy over all questions, with per-subject breakdowns
- **metrics**: accuracy
- **size**: ~12,000 questions
- **methodology**: Contains approximately 12,000 questions filtered and augmented from MMLU and additional sources. Each question has 10 answer choices to reduce guessing advantage. Questions skew toward STEM subjects and require more reasoning steps than the original MMLU. Evaluation uses chain-of-thought prompting with exact-match accuracy.

## Regime

- **typicalUses**: measure
- **exposure**: Public test set with answer keys, partly derived from MMLU, so some items share MMLU's exposure. It replaced MMLU on the Hugging Face Open LLM Leaderboard (v2) and is a common headline number in model releases, which puts it under selection pressure.
- **notes**: Fix the prompting protocol (chain-of-thought or direct answer, number of shots) before comparing; the choice moves scores substantially.

## Validation

- **summary**: MMLU-Pro's evidence supports it as a more discriminating successor to MMLU for broad knowledge evaluation: ten answer options and harder, reasoning-heavy questions restore spread among strong models. It inherits the structural limits of multiple-choice testing, and its STEM skew means claims about overall knowledge breadth need complementary evaluations.
- **evidence**: Restores spread among models that saturate MMLU; Scores are stable across prompt variations; Represents knowledge breadth across disciplines
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: medium
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/mmlu-pro