MBPP

CConstruct

Entry-level Python function generation from a short natural-language description

PPopulation

974 crowd-sourced, entry-level Python tasks (427 in the hand-verified sanitized subset)

Measures

• Basic Python programming ability on short, self-contained tasks
• Code generation consistency across a larger set of problems than HumanEval
• Ability to translate natural language task descriptions into working Python functions

Does not measure

• Software engineering competence — tasks are elementary and isolated from real-world complexity
• Any capability beyond basic function-level Python code generation
• Code quality, design patterns, error handling, or production-readiness
• Ability to work with existing code, understand APIs, or navigate large codebases

MMechanism

Execution / effect checker

The generated code runs against the three asserts; any failure, error, or timeout fails the sample. Three cases rule out few wrong programs.

Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format

Code

ΨSpecification

A sample passes when the generated function passes all three assert-based test cases for the task.

AAnchor

Three crowd-written assert test cases per task

VEvidence view

The generated function, executed against the three test cases, which the standard prompt also shows to the model

ZSignal

Binary pass/fail per sample

RResolution

Final function per sample

ΓAggregation

pass@1, or pass@k (at least one of k samples passes), averaged over tasks; the sanitized subset is reported separately

Metrics

pass@1, pass@10

Size

974 problems (sanitized: 427)

Methodology

Contains 974 programming tasks described in natural language, each with 3 automated test cases. The sanitized subset (MBPP-S) of 427 problems filters for quality and correctness of the original test cases. Models generate Python functions and are evaluated by running the generated code against the test suite. Primary metric is pass@1 accuracy.

Typical uses

• Measure · supports inference and decision

Exposure

Public since 2021 with train, validation, test, and few-shot prompt splits; tasks and tests are widely available online, so contamination risk is significant. In the standard prompt format the grading tests are shown to the model, so the anchor is exposed on every item.

MBPP's evidence supports a larger version of HumanEval's claim, not a harder or more relevant one. Crowd-sourced task quality is lower than HumanEval's hand-written problems and three-test suites make pass rates unreliable, so it no longer supports claims about modern models; SWE-bench and continuously refreshed benchmarks such as LiveCodeBench support more relevant coding claims.

Evidence notes2 partially supported · 1 not supported

1. Partially supported

   Passing the three tests implies a correct general solution

   Three cases rule out few wrong programs, and the standard prompt shows them to the model, so a solution can special-case the visible tests; extended test suites (MBPP+) lowered pass rates.

   Sources: Austin et al., 2021; Liu et al., 2023 (EvalPlus)

2. Partially supported

   Task statements and tests are correct

   Some crowd-sourced tasks are ambiguous or have incorrect test cases; the hand-verified sanitized subset (427 tasks) reduces this.

   Source: Austin et al., 2021

3. Not supported

   Separates current models

   Frontier systems score above 90% even on the sanitized subset.

Saturation

Saturated

Whether top systems still differ, or crowd the ceiling.

Contamination

High risk

How likely test items, or close variants, have leaked into training data.

Successors

• SWE-bench
• LiveCodeBench

Last reviewed

Not yet reviewed · card updated October 12, 2026

• HumanEvalCoding · 2021A set of 164 hand-written Python programming problems with function signatures, docstrings, and unit tests. It was the first widely-adopted code generation benchmark but is now far too easy and small to be informative.
• SWE-benchCoding · 2023A benchmark of real-world software engineering tasks drawn from actual GitHub issues and pull requests in popular Python repositories. Models must generate patches that resolve genuine issues and pass the corresponding test suites.
• LiveCodeBenchCoding · 2024A code generation benchmark built to resist contamination: it continuously collects new problems from competitive programming platforms (LeetCode, Codeforces, AtCoder) and scores models on problems released after their training cutoffs. It evaluates code generation, self-repair, code execution, and test output prediction.

• B: Coding agentTests, static analysis, sandbox invariants, trajectory checks and a calibrated readability judge: an agent eval as a portfolio, not a score.