MATH

CConstruct

Competition-mathematics problem solving, prealgebra through number theory

PPopulation

Problems from AMC, AIME, and other competitions in seven subjects, labeled Levels 1–5; the 5,000-problem test split or the 500-problem MATH-500 subset

Measures

• Mathematical problem-solving across algebra, counting, geometry, number theory, and more
• Ability to apply competition-math techniques requiring genuine multi-step reasoning
• Performance across a well-calibrated difficulty spectrum from Level 1 (easy) to Level 5 (AMC/AIME difficulty)
• Symbolic manipulation and formal mathematical reasoning beyond simple arithmetic

Does not measure

• Research-level mathematical ability — competition math is a specific and somewhat narrow skill
• Ability to formalize proofs, work with abstract structures, or do open-ended mathematical exploration
• Real-world quantitative reasoning that involves modeling, estimation, or data interpretation
• Mathematical creativity or the ability to develop novel approaches to unseen problem types

MMechanism

Deterministic checker

Rule-based answer extraction and normalization, then an equivalence check that accepts different valid representations of the same answer (symbolic equivalence in common graders). Checker implementations differ across harnesses, which changes scores.

Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format

Short answer

ΨSpecification

A problem is solved when the model's final answer (conventionally the boxed expression) is equivalent to the reference answer.

AAnchor

Reference final answer from the competition solution

VEvidence view

The extracted final answer only; the solution steps are not graded

ZSignal

Binary correct/incorrect per problem

RResolution

Final answer per problem

ΓAggregation

Accuracy over the test split, often broken down by subject and difficulty level

Metrics

accuracy

Size

12,500 problems (5,000 test)

Methodology

Contains 12,500 problems (7,500 train, 5,000 test) from AMC, AIME, and other competition math sources across seven subjects. Problems have LaTeX-formatted solutions. Final answers are extracted and compared using a symbolic equivalence checker that handles different valid representations of the same answer. Difficulty is labeled Level 1 through 5.

Typical uses

• Measure · supports inference and decision
• Optimize · shapes the candidate or policy distribution

Exposure

Public train and test splits since 2021, with problems from public competition archives, so contamination is a real concern. The training split is a standard source of fine-tuning and reinforcement-learning data for reasoning models, and MATH-500 is a headline number in reasoning-model releases, so the test problems are under sustained optimization pressure.

Notes

Most current reports use MATH-500, a 500-problem test subset introduced by Lightman et al. (2023). State the split, the answer checker, and the sampling budget (single sample, best-of-N, or tool use) with every score.

MATH's evidence supports comparing final-answer accuracy on competition mathematics, with the harder levels still carrying signal, and its equivalence-based checker credits valid answers that naive exact match would reject. It does not support claims of broad mathematical competence, contamination from public competition archives is a real concern, and top scores are approaching the ceiling.

Evidence notes2 partially supported · 1 untested

1. Partially supported

   Discriminates among the strongest models

   Level 5 problems still spread models, but leading models with best-of-N or tool use exceed 90% overall, and the uneven level distribution gives Level 5 outsized weight in any separation.

2. Partially supported

   The answer checker credits every correct answer

   Equivalence checking accepts many valid forms that exact match would reject, but it still misses some valid answers, and graders differ in what they accept.

3. Untested

   Competition-math accuracy indicates broader quantitative ability

   Modeling, estimation, data interpretation, proof, and open-ended exploration are outside the item pool, and nothing links the score to them.

Saturation

Near saturation

Whether top systems still differ, or crowd the ceiling.

Contamination

High risk

How likely test items, or close variants, have leaked into training data.

Successors

None listed

Last reviewed

Not yet reviewed · card updated October 12, 2026

• GSM8KReasoning · 2021Grade School Math 8K is a dataset of 8,792 grade-school-level math word problems requiring multi-step arithmetic reasoning. Once a key reasoning benchmark, it is now fully saturated and largely serves as a historical reference point.
• BBHReasoning · 2022BIG-Bench Hard is a curated subset of 23 tasks from the BIG-Bench suite where language models initially performed below average human raters. It tests diverse reasoning skills including logical deduction, causal reasoning, and algorithmic thinking.

• A: Mathematical reasoningAnswer checkers, proof assistants, PRMs and mathematicians: one construct, four mechanisms, three regimes, and why a PRM is not "above" the checker.
• E: Reasoning-model trainingDense RLVR and process feedback for development, an untouched certification channel, and judge error re-estimated at every checkpoint.