# MMMU

> Massive Multi-discipline Multimodal Understanding evaluates multimodal models on college-level subject knowledge requiring both image understanding and domain expertise. It spans 30 subjects across art, business, science, health, humanities, and engineering.

## Claim

- **headline**: MMMU accuracy supports comparing vision-language models on college-level, image-based exam questions across 30 subjects; some items are answerable from text alone, so it overstates visual reasoning for text-strong models.
- **construct**: College-level subject knowledge integrated with image understanding
- **population**: Questions from college exams, quizzes, and textbooks in 30 subjects across six disciplines, paired with diagrams, charts, figures, and photographs
- **measures**: Ability to reason about images in the context of college-level domain knowledge; Integration of visual understanding with subject matter expertise across 30 academic disciplines; Performance on questions requiring interpretation of diagrams, charts, scientific figures, and artwork; Multimodal reasoning across a diverse and well-calibrated set of academic subjects
- **doesNotMeasure**: Fine-grained visual perception — many questions could be answered from rough visual descriptions; Real-world visual reasoning in naturalistic, uncurated settings; Ability to generate, edit, or manipulate visual content; Video understanding, spatial reasoning, or 3D scene comprehension

## Instrument

- **itemFormat**: multimodal
- **specification**: A question is answered correctly when the extracted answer matches the key: the keyed option for multiple-choice items, or the reference answer for the minority of open-ended items.
- **anchor**: Fixed answer key; test-split answers are withheld
- **evidenceView**: The answer extracted from the response; the reasoning and visual grounding are not inspected
- **mechanism**: deterministic
- **mechanismDetail**: Rule-based answer extraction followed by matching against the key.
- **signal**: Binary correct/incorrect per question
- **resolution**: Final answer per question
- **aggregation**: Accuracy overall and per discipline; most reports use the 900-question validation split
- **metrics**: accuracy
- **size**: 11,500 questions across 30 subjects
- **methodology**: Contains 11,500 multimodal questions collected from college exams, quizzes, and textbooks across 30 subjects in 6 broad disciplines. Questions are presented with images (diagrams, figures, charts, photographs) and require both visual understanding and domain knowledge to answer. Format is predominantly multiple-choice. Evaluation is accuracy on the selected answer.

## Regime

- **typicalUses**: measure
- **exposure**: Development and validation answers are public and most reports use the validation split; test-split answers are withheld behind an evaluation server. Questions drawn from widely available textbooks and exams carry contamination risk. It is a headline multimodal number in model releases.

## Validation

- **summary**: MMMU's evidence supports claims about multimodal academic knowledge: breadth across 30 subjects and a requirement for image-text integration set it apart from benchmarks with a superficial visual component. Some questions do not truly require deep visual understanding, so text-strong models with basic image description can score above their visual reasoning, which limits claims about visual understanding specifically.
- **evidence**: Every question requires the image; Accuracy reflects depth of multimodal reasoning; Per-discipline scores support discipline-level comparisons
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: medium
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/mmmu