# GSM8K

> Grade School Math 8K is a dataset of 8,792 grade-school-level math word problems requiring multi-step arithmetic reasoning. Once a key reasoning benchmark, it is now fully saturated and largely serves as a historical reference point.

## Claim

- **headline**: GSM8K accuracy supports a smoke test that small or fine-tuned models can solve two- to eight-step grade-school arithmetic word problems; it supports no claim about the mathematical reasoning of current frontier models.
- **construct**: Multi-step arithmetic reasoning on natural-language word problems
- **population**: Linguistically diverse grade-school word problems needing 2–8 steps of elementary calculation; the 1,319-problem test split
- **measures**: Basic multi-step arithmetic reasoning with natural language word problems; Ability to translate simple story problems into a sequence of calculations; Consistency of getting correct final numerical answers through chain-of-thought reasoning
- **doesNotMeasure**: Mathematical reasoning beyond elementary arithmetic — no algebra, geometry, or higher math; Genuine problem-solving ability, as the problems follow highly predictable templates; Robustness to problem variation — models that score 95%+ can fail on trivially rephrased versions; Anything about a model's capability that is still informative, given full saturation by frontier models

## Instrument

- **itemFormat**: short-answer
- **specification**: A problem is solved when the final number extracted from the model's (usually chain-of-thought) response equals the reference answer.
- **anchor**: Reference final answer from a human-written solution
- **evidenceView**: The final number extracted from the response; intermediate reasoning is not checked
- **mechanism**: deterministic
- **mechanismDetail**: Rule-based extraction of the final numeric answer (after an answer marker, or the last number in the output), normalization, and exact match against the reference.
- **signal**: Binary correct/incorrect per problem
- **resolution**: Final answer per problem
- **aggregation**: Accuracy over the test split
- **metrics**: accuracy
- **size**: 8,792 problems (1,319 test)
- **methodology**: Contains 8,792 linguistically diverse grade-school math problems (7,473 train, 1,319 test) that require 2 to 8 steps of elementary calculation. Models generate chain-of-thought solutions and the final numerical answer is extracted and compared to the gold answer. Evaluation is exact-match accuracy on the final number.

## Regime

- **typicalUses**: measure; optimize
- **exposure**: Public train and test splits since 2021. The training split and close variants of the test problems are standard fine-tuning and reinforcement-learning data for reasoning, so the test set has been under direct optimization pressure for years, and contamination across major model families is documented. It was also part of the Hugging Face Open LLM Leaderboard (v1).
- **notes**: Used as a training reward, the final-answer check rewards any route to the right number, including flawed reasoning; it carries no information about the solution steps.

## Validation

- **summary**: GSM8K's evidence once supported claims about multi-step arithmetic reasoning, but saturation, documented contamination, and a grade-school difficulty ceiling mean it no longer supports claims about modern models' reasoning. Models scoring above 95% can fail on slightly modified versions of the same problems, which shows the score does not hold up under construct-irrelevant changes; MATH or competition-level benchmarks support stronger claims.
- **evidence**: Separates frontier models on mathematical reasoning; Accuracy reflects reasoning rather than familiarity with the test problems; A correct final number implies a correct solution
- **status**: draft

## Lifecycle

- **saturation**: saturated
- **contamination**: high
- **successors**: math
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/gsm8k