# HumanEval

> A set of 164 hand-written Python programming problems with function signatures, docstrings, and unit tests. It was the first widely-adopted code generation benchmark but is now far too easy and small to be informative.

## Claim

- **headline**: HumanEval pass@1 supports a basic check that a model can turn a docstring into a short, standalone Python function that passes a few unit tests; it does not support comparing frontier models or claims about software engineering.
- **construct**: Function-level Python code generation from a natural-language specification
- **population**: 164 hand-written, self-contained Python problems (function signature plus docstring), short and mostly algorithmic
- **measures**: Ability to generate correct Python functions from docstring specifications; Basic algorithmic competence on self-contained, well-specified programming tasks; Pass rate across multiple sampling attempts (pass@k: the chance that at least one of k samples passes)
- **doesNotMeasure**: Real-world software engineering ability — the problems are short, isolated, and fully specified; Ability to work with existing codebases, understand context, debug, or refactor; Performance in any language other than Python, or on problems requiring libraries/frameworks; Code quality, readability, efficiency, or any dimension beyond functional correctness; Meaningful model differentiation, as frontier models all achieve pass@1 above 90%

## Instrument

- **itemFormat**: code
- **specification**: A sample passes when the completed function passes every unit test for the problem.
- **anchor**: Hand-written unit tests with expected outputs (7.7 per problem on average)
- **evidenceView**: The generated function, executed against tests the model does not see
- **mechanism**: execution
- **mechanismDetail**: The completion runs in a sandbox against the problem's unit tests; any failing test, error, or timeout fails the sample. Passing means no tested counterexample, not proven correctness.
- **signal**: Binary pass/fail per sample
- **resolution**: Final function per sample
- **aggregation**: pass@k (the probability that at least one of k samples passes all tests), estimated without bias from n ≥ k samples per problem and averaged over problems; pass@1 is the usual headline
- **metrics**: pass@1; pass@10
- **size**: 164 programming problems
- **methodology**: Models are given a function signature and docstring and must generate the function body. Correctness is evaluated by running the generated code against a hidden set of unit tests (typically 5-10 per problem). The primary metric is pass@k — the probability that at least one of k generated samples passes all tests. pass@1 is the most commonly reported figure.

## Regime

- **typicalUses**: measure; optimize
- **exposure**: Public since 2021 and widely reproduced and discussed online, so contamination risk is high. For years it was the headline code-generation number in model releases, which made it a direct hill-climbing target. All tests are public; there is no hidden split.

## Validation

- **summary**: HumanEval's evidence supported early claims about function-level code generation, but it no longer supports claims about modern models: with 164 problems, extreme contamination, and ceiling-level scores, conclusions drawn from it deserve deep skepticism. SWE-bench and LiveCodeBench support more relevant coding claims.
- **evidence**: Passing the tests implies the function is correct; Separates frontier models; Indicates real-world programming ability
- **status**: draft

## Lifecycle

- **saturation**: saturated
- **contamination**: high
- **successors**: swe-bench; livecodebench
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/humaneval