Multimodal · Evaluator card
MMMU
2023Xiang Yue et al.Paper (opens in a new tab)
Massive Multi-discipline Multimodal Understanding evaluates multimodal models on college-level subject knowledge requiring both image understanding and domain expertise. It spans 30 subjects across art, business, science, health, humanities, and engineering.
Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.
Headline claim
MMMU accuracy supports comparing vision-language models on college-level, image-based exam questions across 30 subjects; some items are answerable from text alone, so it overstates visual reasoning for text-strong models.
At a glance
- Mechanism
- Deterministic checkerPrior: execution feedback, rung 2 of 4
- Item format
- Multimodal
- Size
- 11,500 questions across 30 subjects
- Typical uses
- Measure
- Saturation
- Active
- Contamination
- Medium risk
- Year
- 2023
1Claim
What property are we trying to infer, over what world, and for what decision?
- Construct
- College-level subject knowledge integrated with image understanding
- Population
- Questions from college exams, quizzes, and textbooks in 30 subjects across six disciplines, paired with diagrams, charts, figures, and photographs
- Measures
- Ability to reason about images in the context of college-level domain knowledge
- Integration of visual understanding with subject matter expertise across 30 academic disciplines
- Performance on questions requiring interpretation of diagrams, charts, scientific figures, and artwork
- Multimodal reasoning across a diverse and well-calibrated set of academic subjects
- Does not measure
- Fine-grained visual perception — many questions could be answered from rough visual descriptions
- Real-world visual reasoning in naturalistic, uncurated settings
- Ability to generate, edit, or manipulate visual content
- Video understanding, spatial reasoning, or 3D scene comprehension
2Instrument
How is the judgment actually produced?
- Mechanism
Deterministic checker
Rule-based answer extraction followed by matching against the key.
Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.
- Item format
- Multimodal
- Specification
- A question is answered correctly when the extracted answer matches the key: the keyed option for multiple-choice items, or the reference answer for the minority of open-ended items.
- Anchor
- Fixed answer key; test-split answers are withheld
- Evidence view
- The answer extracted from the response; the reasoning and visual grounding are not inspected
- Signal
- Binary correct/incorrect per question
- Resolution
- Final answer per question
- Aggregation
- Accuracy overall and per discipline; most reports use the 900-question validation split
- Metrics
- accuracy
- Size
- 11,500 questions across 30 subjects
- Methodology
- Contains 11,500 multimodal questions collected from college exams, quizzes, and textbooks across 30 subjects in 6 broad disciplines. Questions are presented with images (diagrams, figures, charts, photographs) and require both visual understanding and domain knowledge to answer. Format is predominantly multiple-choice. Evaluation is accuracy on the selected answer.
3Regime
What causal role does the signal play, and what happens when a system adapts to it?
- Typical uses
- Measure · supports inference and decision
- Exposure
- Development and validation answers are public and most reports use the validation split; test-split answers are withheld behind an evaluation server. Questions drawn from widely available textbooks and exams carry contamination risk. It is a headline multimodal number in model releases.
4ValidationDraft
Why should we believe that the signal means what we claim it means — before and after the system adapts to it?
MMMU's evidence supports claims about multimodal academic knowledge: breadth across 30 subjects and a requirement for image-text integration set it apart from benchmarks with a superficial visual component. Some questions do not truly require deep visual understanding, so text-strong models with basic image description can score above their visual reasoning, which limits claims about visual understanding specifically.
Evidence notes3 partially supported
- Partially supported
Every question requires the image
Some items can be answered from text alone. The authors' follow-up (MMMU-Pro) removed questions that text-only models could answer, added answer options, and introduced a vision-only setting; model scores fell substantially.
Source: Yue et al., 2024 (MMMU-Pro)
- Partially supported
Accuracy reflects depth of multimodal reasoning
The predominantly multiple-choice format limits the depth of reasoning tested, and variable image quality can obscure visual details.
- Partially supported
Per-discipline scores support discipline-level comparisons
Coverage is uneven: some disciplines have substantially more questions than others, so smaller disciplines give noisier scores.
5Lifecycle
Does the validity survive over time, and are we still evaluating the right thing?
- Saturation
- Active
Whether top systems still differ, or crowd the ceiling.
- Contamination
- Medium risk
How likely test items, or close variants, have leaked into training data.
- Successors
- None listed
- Last reviewed
- Not yet reviewed · card updated October 12, 2026
Limitations
- Multiple-choice format limits the depth of multimodal reasoning being tested
- Image quality and resolution vary, sometimes making visual details hard to distinguish
- Some questions can be partially answered from text alone, reducing the multimodal signal
- Subject coverage is uneven, with some disciplines having substantially more questions than others
- Contamination risk exists for questions drawn from widely-available textbook materials
When to use
- Evaluating multimodal models on knowledge-intensive visual reasoning across diverse academic domains
- Comparing vision-language models on tasks requiring both image understanding and domain expertise
- Assessing readiness of multimodal systems for education or professional knowledge applications
When not to use
- Testing pure visual perception or fine-grained image understanding — use specialized vision benchmarks
- Evaluating creative or generative multimodal capabilities
Related benchmarks
- MathVistaMultimodal · 2023A consolidated benchmark for mathematical reasoning in visual contexts, combining problems that require understanding charts, plots, geometry figures, and scientific diagrams to solve mathematical questions.
- VQAv2Multimodal · 2017Visual Question Answering v2 is a large-scale benchmark where models answer open-ended questions about natural images. It was the foundational VQA benchmark but is now saturated and limited by its simplistic question types.