# MMLU

> Massive Multitask Language Understanding evaluates broad academic knowledge across 57 subjects ranging from STEM to humanities. It became the de facto standard for measuring general knowledge in LLMs but is now largely superseded by harder variants.

## Claim

- **headline**: MMLU accuracy supports a coarse screen of whether a model can recognize correct answers to English-language academic exam questions across 57 subjects; it no longer supports ranking frontier models or claims about understanding.
- **construct**: Breadth of academic knowledge, operationalized as recognizing the keyed option on exam-style questions
- **population**: English four-option questions from practice exams, textbooks, and academic materials (elementary to professional level), answered zero- or few-shot by base and chat models
- **measures**: Breadth of factual knowledge across academic disciplines; Ability to apply domain-specific knowledge to multiple-choice questions; Performance consistency across diverse subject areas from abstract algebra to world religions; Baseline competence at undergraduate and professional-level exam questions
- **doesNotMeasure**: Genuine understanding or reasoning — models can score highly through pattern matching and memorization of training data that overlaps with test questions; Ability to generate, explain, or apply knowledge in open-ended contexts; Robustness to question rephrasing — small perturbations can cause large score swings; Real-world problem-solving ability, which requires integrating knowledge with judgment

## Instrument

- **itemFormat**: multiple-choice
- **specification**: A question is answered correctly when the model's selected option matches the keyed answer among four choices.
- **anchor**: Fixed answer key: one keyed option per question, taken from the source materials
- **evidenceView**: The selected option only (a generated letter or the highest-likelihood option, depending on the harness); no reasoning is inspected
- **mechanism**: deterministic
- **mechanismDetail**: Exact match of the selected option against the key, with no learned judgment. The prompt format, number of shots, and answer-extraction rule belong to the instrument and change scores.
- **signal**: Binary correct/incorrect per question
- **resolution**: Final answer per question
- **aggregation**: Accuracy over all questions, with per-subject breakdowns; some harnesses report the mean of per-subject accuracies instead
- **metrics**: accuracy
- **size**: 14,042 questions across 57 subjects
- **methodology**: Presents 14,042 multiple-choice questions across 57 subjects in a zero-shot or few-shot setting. Questions are sourced from practice exams, textbooks, and academic materials at varying difficulty levels from elementary to professional. Evaluation is exact-match accuracy on the selected answer choice.

## Regime

- **typicalUses**: measure; optimize
- **exposure**: Public test set with answer keys, and questions widely reproduced on the web; contamination is well documented. For years MMLU was the headline knowledge number in model releases and on the Hugging Face Open LLM Leaderboard (v1), which made it a hill-climbing target, so its anchor has been spent many times over.
- **notes**: Compare scores only within a fixed harness: prompt format, few-shot setup, and whether the answer is scored by letter likelihood or by generation all move the number.

## Validation

- **summary**: MMLU's evidence supports it as an early, broad screen of exam-style knowledge and as a historical reference point. It does not support claims about understanding or general intelligence, or rankings among frontier models: contamination is well documented, the multiple-choice format is forgiving, and near-ceiling scores leave little range to discriminate.
- **evidence**: Separates frontier models on broad academic knowledge; The answer key is a reliable anchor across all 57 subjects; Scores are comparable across evaluation harnesses; Scores reflect knowledge rather than memorized test items
- **status**: draft

## Lifecycle

- **saturation**: near-saturated
- **contamination**: high
- **successors**: mmlu-pro
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/mmlu