# MathVista

> A consolidated benchmark for mathematical reasoning in visual contexts, combining problems that require understanding charts, plots, geometry figures, and scientific diagrams to solve mathematical questions.

## Claim

- **headline**: MathVista accuracy supports comparing vision-language models on mathematical reasoning grounded in charts, plots, geometry figures, and scientific diagrams; its aggregate mixes 31 source datasets of uneven quality, so per-type results support more specific claims.
- **construct**: Mathematical reasoning that requires reading quantitative or geometric information from images
- **population**: Examples aggregated from 28 existing multimodal datasets and 3 new ones, spanning figure QA, geometry problem solving, math word problems, textbook QA, and visual QA
- **measures**: Mathematical reasoning grounded in visual information like charts, plots, and geometric figures; Ability to extract quantitative data from visual representations and perform calculations; Integration of spatial reasoning with mathematical problem-solving; Performance across diverse visual math tasks from geometry to statistical chart interpretation; Whether models can genuinely read and reason about data visualizations
- **doesNotMeasure**: Pure mathematical ability without a visual component — use MATH for that; Visual reasoning in non-mathematical contexts; Ability to create or modify mathematical visualizations; Advanced mathematical reasoning beyond what can be tested with visual inputs

## Instrument

- **itemFormat**: multimodal
- **specification**: An example is correct when the answer extracted from the response matches the reference: the keyed option, or a number within the allowed precision.
- **anchor**: Reference answers from the source datasets; most reports use the 1,000-example testmini split, whose answers are public
- **evidenceView**: The final answer extracted from the model's free-form response
- **mechanism**: hybrid
- **mechanismDetail**: A language model extracts the final answer from the free-form response (a learned step); a rule-based comparison against the reference, with numerical tolerance, computes the verdict (deterministic). Extraction errors become scoring errors.
- **signal**: Binary correct/incorrect per example
- **resolution**: Final answer per example
- **aggregation**: Accuracy overall and broken down by task type and reasoning skill
- **metrics**: accuracy
- **size**: 6,141 examples across 31 datasets
- **methodology**: Aggregates 6,141 mathematical reasoning examples from 28 existing multimodal datasets and 3 newly created datasets. Problems span five task types: figure question answering, geometry problem solving, math word problems with visual context, textbook question answering, and visual question answering requiring calculation. Evaluation uses exact-match accuracy with some tolerance for numerical precision.

## Regime

- **typicalUses**: measure
- **exposure**: The testmini split, with answers, is public and is what most reports use. Many source datasets are years old and public, so older problems carry contamination risk.

## Validation

- **summary**: MathVista's evidence supports claims about doing mathematics from visual inputs, with task types ranging from chart reading to geometry, and it still separates current models. Its aggregated construction means quality and difficulty vary across sources, so per-type results are more informative than the headline number.
- **evidence**: Requires reading the image; The aggregate measures one coherent construct; Answer scoring is accurate
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: medium
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/mathvista