Coding · Evaluator card
HumanEval
2021Mark Chen et al. (OpenAI)Paper (opens in a new tab)
A set of 164 hand-written Python programming problems with function signatures, docstrings, and unit tests. It was the first widely-adopted code generation benchmark but is now far too easy and small to be informative.
Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.
Headline claim
HumanEval pass@1 supports a basic check that a model can turn a docstring into a short, standalone Python function that passes a few unit tests; it does not support comparing frontier models or claims about software engineering.
At a glance
- Mechanism
- Execution / effect checkerPrior: execution feedback, rung 2 of 4
- Item format
- Code
- Size
- 164 programming problems
- Typical uses
- Measure, Optimize
- Saturation
- Saturated
- Contamination
- High risk
- Year
- 2021
1Claim
What property are we trying to infer, over what world, and for what decision?
- Construct
- Function-level Python code generation from a natural-language specification
- Population
- 164 hand-written, self-contained Python problems (function signature plus docstring), short and mostly algorithmic
- Measures
- Ability to generate correct Python functions from docstring specifications
- Basic algorithmic competence on self-contained, well-specified programming tasks
- Pass rate across multiple sampling attempts (pass@k: the chance that at least one of k samples passes)
- Does not measure
- Real-world software engineering ability — the problems are short, isolated, and fully specified
- Ability to work with existing codebases, understand context, debug, or refactor
- Performance in any language other than Python, or on problems requiring libraries/frameworks
- Code quality, readability, efficiency, or any dimension beyond functional correctness
- Meaningful model differentiation, as frontier models all achieve pass@1 above 90%
2Instrument
How is the judgment actually produced?
- Mechanism
Execution / effect checker
The completion runs in a sandbox against the problem's unit tests; any failing test, error, or timeout fails the sample. Passing means no tested counterexample, not proven correctness.
Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.
- Item format
- Code
- Specification
- A sample passes when the completed function passes every unit test for the problem.
- Anchor
- Hand-written unit tests with expected outputs (7.7 per problem on average)
- Evidence view
- The generated function, executed against tests the model does not see
- Signal
- Binary pass/fail per sample
- Resolution
- Final function per sample
- Aggregation
- pass@k (the probability that at least one of k samples passes all tests), estimated without bias from n ≥ k samples per problem and averaged over problems; pass@1 is the usual headline
- Metrics
- pass@1, pass@10
- Size
- 164 programming problems
- Methodology
- Models are given a function signature and docstring and must generate the function body. Correctness is evaluated by running the generated code against a hidden set of unit tests (typically 5-10 per problem). The primary metric is pass@k — the probability that at least one of k generated samples passes all tests. pass@1 is the most commonly reported figure.
3Regime
What causal role does the signal play, and what happens when a system adapts to it?
- Typical uses
- Measure · supports inference and decision
- Optimize · shapes the candidate or policy distribution
- Exposure
- Public since 2021 and widely reproduced and discussed online, so contamination risk is high. For years it was the headline code-generation number in model releases, which made it a direct hill-climbing target. All tests are public; there is no hidden split.
4ValidationDraft
Why should we believe that the signal means what we claim it means — before and after the system adapts to it?
HumanEval's evidence supported early claims about function-level code generation, but it no longer supports claims about modern models: with 164 problems, extreme contamination, and ceiling-level scores, conclusions drawn from it deserve deep skepticism. SWE-bench and LiveCodeBench support more relevant coding claims.
Evidence notes1 partially supported · 2 not supported
- Partially supported
Passing the tests implies the function is correct
Test suites are small: extending them (HumanEval+) caught previously accepted incorrect solutions and lowered pass rates across models.
Source: Liu et al., 2023 (EvalPlus)
- Not supported
Separates frontier models
Frontier models exceed 95% pass@1. With 164 problems, each problem is worth 0.6 points, and a 95% interval around 90% spans roughly 84–94%.
- Not supported
Indicates real-world programming ability
The problems are short, isolated, fully specified functions; working in existing codebases, debugging, and refactoring are outside the item pool.
5Lifecycle
Does the validity survive over time, and are we still evaluating the right thing?
- Saturation
- Saturated
Whether top systems still differ, or crowd the ceiling.
- Contamination
- High risk
How likely test items, or close variants, have leaked into training data.
- Successors
- Last reviewed
- Not yet reviewed · card updated October 12, 2026
Limitations
- Only 164 problems — far too small for statistically meaningful comparisons between strong models
- Completely saturated with frontier models exceeding 95% pass@1
- Massive contamination risk as the problems have been widely reproduced and discussed online
- Problems are trivial standalone functions that bear little resemblance to real programming work
- Unit test suites are sometimes insufficient, allowing incorrect solutions to pass
When to use
- Quick sanity check that a code generation model can produce basic correct Python
- Historical reference point for tracking code generation progress over time
When not to use
- Evaluating any model's real-world coding ability or software engineering competence
- Comparing frontier models — the benchmark cannot differentiate between them
- Making procurement or deployment decisions about coding assistants
Related benchmarks
- MBPPCoding · 2021Mostly Basic Python Programming contains 974 crowd-sourced Python programming tasks designed to be solvable by entry-level programmers. It complements HumanEval with more problems but at a similar or lower difficulty level.
- SWE-benchCoding · 2023A benchmark of real-world software engineering tasks drawn from actual GitHub issues and pull requests in popular Python repositories. Models must generate patches that resolve genuine issues and pass the corresponding test suites.
- LiveCodeBenchCoding · 2024A code generation benchmark built to resist contamination: it continuously collects new problems from competitive programming platforms (LeetCode, Codeforces, AtCoder) and scores models on problems released after their training cutoffs. It evaluates code generation, self-repair, code execution, and test output prediction.