# MATH

> A dataset of 12,500 competition-level mathematics problems spanning seven subjects from prealgebra to number theory. It remains more discriminating than GSM8K but is approaching saturation for the strongest frontier models.

## Claim

- **headline**: MATH accuracy supports comparing models' final-answer performance on competition-style problems across five difficulty levels, with most remaining signal at Level 5; it does not support claims about mathematical ability beyond competition math.
- **construct**: Competition-mathematics problem solving, prealgebra through number theory
- **population**: Problems from AMC, AIME, and other competitions in seven subjects, labeled Levels 1–5; the 5,000-problem test split or the 500-problem MATH-500 subset
- **measures**: Mathematical problem-solving across algebra, counting, geometry, number theory, and more; Ability to apply competition-math techniques requiring genuine multi-step reasoning; Performance across a well-calibrated difficulty spectrum from Level 1 (easy) to Level 5 (AMC/AIME difficulty); Symbolic manipulation and formal mathematical reasoning beyond simple arithmetic
- **doesNotMeasure**: Research-level mathematical ability — competition math is a specific and somewhat narrow skill; Ability to formalize proofs, work with abstract structures, or do open-ended mathematical exploration; Real-world quantitative reasoning that involves modeling, estimation, or data interpretation; Mathematical creativity or the ability to develop novel approaches to unseen problem types

## Instrument

- **itemFormat**: short-answer
- **specification**: A problem is solved when the model's final answer (conventionally the boxed expression) is equivalent to the reference answer.
- **anchor**: Reference final answer from the competition solution
- **evidenceView**: The extracted final answer only; the solution steps are not graded
- **mechanism**: deterministic
- **mechanismDetail**: Rule-based answer extraction and normalization, then an equivalence check that accepts different valid representations of the same answer (symbolic equivalence in common graders). Checker implementations differ across harnesses, which changes scores.
- **signal**: Binary correct/incorrect per problem
- **resolution**: Final answer per problem
- **aggregation**: Accuracy over the test split, often broken down by subject and difficulty level
- **metrics**: accuracy
- **size**: 12,500 problems (5,000 test)
- **methodology**: Contains 12,500 problems (7,500 train, 5,000 test) from AMC, AIME, and other competition math sources across seven subjects. Problems have LaTeX-formatted solutions. Final answers are extracted and compared using a symbolic equivalence checker that handles different valid representations of the same answer. Difficulty is labeled Level 1 through 5.

## Regime

- **typicalUses**: measure; optimize
- **exposure**: Public train and test splits since 2021, with problems from public competition archives, so contamination is a real concern. The training split is a standard source of fine-tuning and reinforcement-learning data for reasoning models, and MATH-500 is a headline number in reasoning-model releases, so the test problems are under sustained optimization pressure.
- **notes**: Most current reports use MATH-500, a 500-problem test subset introduced by Lightman et al. (2023). State the split, the answer checker, and the sampling budget (single sample, best-of-N, or tool use) with every score.

## Validation

- **summary**: MATH's evidence supports comparing final-answer accuracy on competition mathematics, with the harder levels still carrying signal, and its equivalence-based checker credits valid answers that naive exact match would reject. It does not support claims of broad mathematical competence, contamination from public competition archives is a real concern, and top scores are approaching the ceiling.
- **evidence**: Discriminates among the strongest models; The answer checker credits every correct answer; Competition-math accuracy indicates broader quantitative ability
- **status**: draft

## Lifecycle

- **saturation**: near-saturated
- **contamination**: high
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/math