# MBPP

> Mostly Basic Python Programming contains 974 crowd-sourced Python programming tasks designed to be solvable by entry-level programmers. It complements HumanEval with more problems but at a similar or lower difficulty level.

## Claim

- **headline**: MBPP pass@1 supports a basic check that small or specialized models can write short, entry-level Python functions that pass three example tests; it does not support ranking current models or broader claims about programming skill.
- **construct**: Entry-level Python function generation from a short natural-language description
- **population**: 974 crowd-sourced, entry-level Python tasks (427 in the hand-verified sanitized subset)
- **measures**: Basic Python programming ability on short, self-contained tasks; Code generation consistency across a larger set of problems than HumanEval; Ability to translate natural language task descriptions into working Python functions
- **doesNotMeasure**: Software engineering competence — tasks are elementary and isolated from real-world complexity; Any capability beyond basic function-level Python code generation; Code quality, design patterns, error handling, or production-readiness; Ability to work with existing code, understand APIs, or navigate large codebases

## Instrument

- **itemFormat**: code
- **specification**: A sample passes when the generated function passes all three assert-based test cases for the task.
- **anchor**: Three crowd-written assert test cases per task
- **evidenceView**: The generated function, executed against the three test cases, which the standard prompt also shows to the model
- **mechanism**: execution
- **mechanismDetail**: The generated code runs against the three asserts; any failure, error, or timeout fails the sample. Three cases rule out few wrong programs.
- **signal**: Binary pass/fail per sample
- **resolution**: Final function per sample
- **aggregation**: pass@1, or pass@k (at least one of k samples passes), averaged over tasks; the sanitized subset is reported separately
- **metrics**: pass@1; pass@10
- **size**: 974 problems (sanitized: 427)
- **methodology**: Contains 974 programming tasks described in natural language, each with 3 automated test cases. The sanitized subset (MBPP-S) of 427 problems filters for quality and correctness of the original test cases. Models generate Python functions and are evaluated by running the generated code against the test suite. Primary metric is pass@1 accuracy.

## Regime

- **typicalUses**: measure
- **exposure**: Public since 2021 with train, validation, test, and few-shot prompt splits; tasks and tests are widely available online, so contamination risk is significant. In the standard prompt format the grading tests are shown to the model, so the anchor is exposed on every item.

## Validation

- **summary**: MBPP's evidence supports a larger version of HumanEval's claim, not a harder or more relevant one. Crowd-sourced task quality is lower than HumanEval's hand-written problems and three-test suites make pass rates unreliable, so it no longer supports claims about modern models; SWE-bench and continuously refreshed benchmarks such as LiveCodeBench support more relevant coding claims.
- **evidence**: Passing the three tests implies a correct general solution; Task statements and tests are correct; Separates current models
- **status**: draft

## Lifecycle

- **saturation**: saturated
- **contamination**: high
- **successors**: swe-bench; livecodebench
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/mbpp