Multimodal · Evaluator card
MathVista
2023Pan Lu et al.Paper (opens in a new tab)
A consolidated benchmark for mathematical reasoning in visual contexts, combining problems that require understanding charts, plots, geometry figures, and scientific diagrams to solve mathematical questions.
Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.
Headline claim
MathVista accuracy supports comparing vision-language models on mathematical reasoning grounded in charts, plots, geometry figures, and scientific diagrams; its aggregate mixes 31 source datasets of uneven quality, so per-type results support more specific claims.
At a glance
- Mechanism
- Hybrid portfolioPrior: mixed, one per channel
- Item format
- Multimodal
- Size
- 6,141 examples across 31 datasets
- Typical uses
- Measure
- Saturation
- Active
- Contamination
- Medium risk
- Year
- 2023
1Claim
What property are we trying to infer, over what world, and for what decision?
- Construct
- Mathematical reasoning that requires reading quantitative or geometric information from images
- Population
- Examples aggregated from 28 existing multimodal datasets and 3 new ones, spanning figure QA, geometry problem solving, math word problems, textbook QA, and visual QA
- Measures
- Mathematical reasoning grounded in visual information like charts, plots, and geometric figures
- Ability to extract quantitative data from visual representations and perform calculations
- Integration of spatial reasoning with mathematical problem-solving
- Performance across diverse visual math tasks from geometry to statistical chart interpretation
- Whether models can genuinely read and reason about data visualizations
- Does not measure
- Pure mathematical ability without a visual component — use MATH for that
- Visual reasoning in non-mathematical contexts
- Ability to create or modify mathematical visualizations
- Advanced mathematical reasoning beyond what can be tested with visual inputs
2Instrument
How is the judgment actually produced?
- Mechanism
Hybrid portfolio
A language model extracts the final answer from the free-form response (a learned step); a rule-based comparison against the reference, with numerical tolerance, computes the verdict (deterministic). Extraction errors become scoring errors.
Hierarchy prior: mixed. Each channel carries the prior of its own mechanism. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.
- Item format
- Multimodal
- Specification
- An example is correct when the answer extracted from the response matches the reference: the keyed option, or a number within the allowed precision.
- Anchor
- Reference answers from the source datasets; most reports use the 1,000-example testmini split, whose answers are public
- Evidence view
- The final answer extracted from the model's free-form response
- Signal
- Binary correct/incorrect per example
- Resolution
- Final answer per example
- Aggregation
- Accuracy overall and broken down by task type and reasoning skill
- Metrics
- accuracy
- Size
- 6,141 examples across 31 datasets
- Methodology
- Aggregates 6,141 mathematical reasoning examples from 28 existing multimodal datasets and 3 newly created datasets. Problems span five task types: figure question answering, geometry problem solving, math word problems with visual context, textbook question answering, and visual question answering requiring calculation. Evaluation uses exact-match accuracy with some tolerance for numerical precision.
3Regime
What causal role does the signal play, and what happens when a system adapts to it?
- Typical uses
- Measure · supports inference and decision
- Exposure
- The testmini split, with answers, is public and is what most reports use. Many source datasets are years old and public, so older problems carry contamination risk.
4ValidationDraft
Why should we believe that the signal means what we claim it means — before and after the system adapts to it?
MathVista's evidence supports claims about doing mathematics from visual inputs, with task types ranging from chart reading to geometry, and it still separates current models. Its aggregated construction means quality and difficulty vary across sources, so per-type results are more informative than the headline number.
Evidence notes3 partially supported
- Partially supported
Requires reading the image
Some chart and plot items can be solved from captions or alt-text without visual understanding, and some problems from older datasets have simpler visual demands than intended.
- Partially supported
The aggregate measures one coherent construct
Problems come from 31 datasets of uneven quality and difficulty, and the aggregate hides per-type performance.
- Partially supported
Answer scoring is accurate
Final-answer matching ignores the reasoning, and the model-based answer-extraction step can misread responses.
Source: Lu et al., 2023
5Lifecycle
Does the validity survive over time, and are we still evaluating the right thing?
- Saturation
- Active
Whether top systems still differ, or crowd the ceiling.
- Contamination
- Medium risk
How likely test items, or close variants, have leaked into training data.
- Successors
- None listed
- Last reviewed
- Not yet reviewed · card updated October 12, 2026
Limitations
- Aggregation from many source datasets creates uneven quality and difficulty distribution
- Some problems from older datasets may have simpler visual requirements than intended
- Exact-match evaluation on numerical answers does not capture reasoning quality
- Chart and plot interpretation tasks may be solvable from image captions or alt-text without true visual understanding
When to use
- Evaluating multimodal mathematical reasoning with a diverse set of visual problem types
- Assessing whether a model can extract and reason about quantitative information from visual representations
- Comparing vision-language models on the intersection of math and visual understanding
When not to use
- Testing pure mathematical reasoning without visual components
- When you need to evaluate a specific type of visual math (e.g., only geometry) — the aggregated nature obscures per-type performance
Related benchmarks
- MMMUMultimodal · 2023Massive Multi-discipline Multimodal Understanding evaluates multimodal models on college-level subject knowledge requiring both image understanding and domain expertise. It spans 30 subjects across art, business, science, health, humanities, and engineering.
- MATHReasoning · 2021A dataset of 12,500 competition-level mathematics problems spanning seven subjects from prealgebra to number theory. It remains more discriminating than GSM8K but is approaching saturation for the strongest frontier models.
- VQAv2Multimodal · 2017Visual Question Answering v2 is a large-scale benchmark where models answer open-ended questions about natural images. It was the foundational VQA benchmark but is now saturated and limited by its simplistic question types.