GSM8K

CConstruct

Multi-step arithmetic reasoning on natural-language word problems

PPopulation

Linguistically diverse grade-school word problems needing 2–8 steps of elementary calculation; the 1,319-problem test split

Measures

• Basic multi-step arithmetic reasoning with natural language word problems
• Ability to translate simple story problems into a sequence of calculations
• Consistency of getting correct final numerical answers through chain-of-thought reasoning

Does not measure

• Mathematical reasoning beyond elementary arithmetic — no algebra, geometry, or higher math
• Genuine problem-solving ability, as the problems follow highly predictable templates
• Robustness to problem variation — models that score 95%+ can fail on trivially rephrased versions
• Anything about a model's capability that is still informative, given full saturation by frontier models

MMechanism

Deterministic checker

Rule-based extraction of the final numeric answer (after an answer marker, or the last number in the output), normalization, and exact match against the reference.

Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format

Short answer

ΨSpecification

A problem is solved when the final number extracted from the model's (usually chain-of-thought) response equals the reference answer.

AAnchor

Reference final answer from a human-written solution

VEvidence view

The final number extracted from the response; intermediate reasoning is not checked

ZSignal

Binary correct/incorrect per problem

RResolution

Final answer per problem

ΓAggregation

Accuracy over the test split

Metrics

accuracy

Size

8,792 problems (1,319 test)

Methodology

Contains 8,792 linguistically diverse grade-school math problems (7,473 train, 1,319 test) that require 2 to 8 steps of elementary calculation. Models generate chain-of-thought solutions and the final numerical answer is extracted and compared to the gold answer. Evaluation is exact-match accuracy on the final number.

Typical uses

• Measure · supports inference and decision
• Optimize · shapes the candidate or policy distribution

Exposure

Public train and test splits since 2021. The training split and close variants of the test problems are standard fine-tuning and reinforcement-learning data for reasoning, so the test set has been under direct optimization pressure for years, and contamination across major model families is documented. It was also part of the Hugging Face Open LLM Leaderboard (v1).

Notes

Used as a training reward, the final-answer check rewards any route to the right number, including flawed reasoning; it carries no information about the solution steps.

GSM8K's evidence once supported claims about multi-step arithmetic reasoning, but saturation, documented contamination, and a grade-school difficulty ceiling mean it no longer supports claims about modern models' reasoning. Models scoring above 95% can fail on slightly modified versions of the same problems, which shows the score does not hold up under construct-irrelevant changes; MATH or competition-level benchmarks support stronger claims.

Evidence notes1 partially supported · 1 contested · 1 not supported

1. Not supported

   Separates frontier models on mathematical reasoning

   All frontier models score above 95%, leaving no discriminating range.

2. Contested

   Accuracy reflects reasoning rather than familiarity with the test problems

   On freshly written look-alike problems (GSM1k), some model families scored markedly lower while frontier models showed little drop; templated variants that change names and numbers or add an irrelevant clause (GSM-Symbolic) lowered and destabilized accuracy for many models.

   Sources: Zhang et al., 2024 (GSM1k); Mirzadeh et al., 2024 (GSM-Symbolic)

3. Partially supported

   A correct final number implies a correct solution

   The checker sees only the final number: a sound method with one arithmetic slip scores zero, and a flawed method that lands on the right number gets full credit.

Saturation

Saturated

Whether top systems still differ, or crowd the ceiling.

Contamination

High risk

How likely test items, or close variants, have leaked into training data.

Successors

• MATH

Last reviewed

Not yet reviewed · card updated October 12, 2026

• MATHReasoning · 2021A dataset of 12,500 competition-level mathematics problems spanning seven subjects from prealgebra to number theory. It remains more discriminating than GSM8K but is approaching saturation for the strongest frontier models.
• BBHReasoning · 2022BIG-Bench Hard is a curated subset of 23 tasks from the BIG-Bench suite where language models initially performed below average human raters. It tests diverse reasoning skills including logical deduction, causal reasoning, and algorithmic thinking.

• A: Mathematical reasoningAnswer checkers, proof assistants, PRMs and mathematicians: one construct, four mechanisms, three regimes, and why a PRM is not "above" the checker.
• E: Reasoning-model trainingDense RLVR and process feedback for development, an untouched certification channel, and judge error re-estimated at every checkpoint.