SWE-bench

CConstruct

Practical software maintenance: diagnosing a reported issue in a real codebase and producing a working fix

PPopulation

Real GitHub issues and the merged pull requests that fixed them in 12 popular Python repositories; the 500-task, human-screened Verified subset is the usual reporting set

Measures

• Ability to understand real codebases, diagnose issues, and produce working fixes
• Software engineering competence including code navigation, debugging, and patch generation
• Performance on authentic tasks sourced from open-source Python project histories
• End-to-end problem-solving from issue description to validated code change
• Practical utility as a coding assistant on realistic software maintenance tasks

Does not measure

• Greenfield software development or architectural design ability
• Performance on languages other than Python or on non-open-source codebases
• Code review quality, documentation writing, or collaborative engineering skills
• Efficiency or resource usage of the generated solutions

MMechanism

Execution / effect checker

The patch is applied in a containerized copy of the repository at the pre-fix commit and the specified tests are run. Passing means no tested counterexample; behavior the tests do not cover goes unchecked.

Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format

Code

ΨSpecification

A task is resolved when, after the generated patch is applied, the tests that the reference pull request made pass now pass and the previously passing tests still pass.

AAnchor

Tests from the merged reference pull request (fail-to-pass) plus the repository's existing passing tests (pass-to-pass)

VEvidence view

The repository state after the patch, observed only through test outcomes; the agent's trajectory is not graded

ZSignal

Binary resolved/unresolved per task

RResolution

Final patch per task

ΓAggregation

Percentage of tasks resolved

Metrics

% resolved

Size

2,294 task instances (Verified: 500)

Methodology

Collects task instances from 12 popular Python repositories by identifying merged pull requests that fix issues and have associated test changes. Each instance provides the issue description, repository state before the fix, and test cases added by the PR. Models must generate a patch that, when applied, makes the failing tests pass without breaking existing tests. SWE-bench Verified is a human-validated subset of 500 instances with confirmed solvability.

Typical uses

• Measure · supports inference and decision
• Optimize · shapes the candidate or policy distribution

Exposure

Tasks are public and the reference fixes sit in public repository histories, so pretraining contamination is likely for fixes that predate a model's cutoff. Verified is the headline agentic-coding number in model releases and on a public leaderboard, and scaffolds are tuned against it; the same issue-plus-tests format is widely used to build training environments. All grading tests are public.

Notes

Scores describe whole systems: the same model can score very differently across scaffolds, retrieval strategies, tool budgets, and attempt counts. Report all of them with every score.

SWE-bench's evidence supports claims about real software-maintenance capability more directly than function-level benchmarks, because it grades fixes to real issues with the repositories' own tests; the human-screened Verified subset addresses quality problems in the original set. Scores reflect the entire agent system — scaffolding, retrieval, and tool use — as much as the underlying model, so they support system comparisons more readily than pure model comparisons.

Evidence notes2 partially supported · 1 not supported · 1 untested

1. Partially supported

   Tasks are solvable and fairly graded

   Human screening of a sample of the original set flagged 38.3% of tasks as underspecified and 61.1% as having tests that could reject valid fixes; 68.3% were filtered out, leaving the 500-task Verified subset, on which the same model scored about twice as high. Invalid tasks can understate capability.

   Source: OpenAI, 2024 (Introducing SWE-bench Verified)

2. Partially supported

   Passing the tests means the issue is correctly fixed

   The reference pull request's tests ground the verdict in executed behavior, but they cover only what the original fix tested; a patch can pass while being incomplete or wrong elsewhere.

3. Not supported

   Scores isolate model capability

   Scaffolding and agent infrastructure differ across submissions, so results compare systems rather than models.

4. Untested

   Results transfer to other languages and closed codebases

   All tasks are Python, from well-maintained, well-tested open-source projects.

Saturation

Active

Whether top systems still differ, or crowd the ceiling.

Contamination

High risk

How likely test items, or close variants, have leaked into training data.

Successors

None listed

Last reviewed

Not yet reviewed · card updated October 12, 2026

• HumanEvalCoding · 2021A set of 164 hand-written Python programming problems with function signatures, docstrings, and unit tests. It was the first widely-adopted code generation benchmark but is now far too easy and small to be informative.
• MBPPCoding · 2021Mostly Basic Python Programming contains 974 crowd-sourced Python programming tasks designed to be solvable by entry-level programmers. It complements HumanEval with more problems but at a similar or lower difficulty level.
• LiveCodeBenchCoding · 2024A code generation benchmark built to resist contamination: it continuously collects new problems from competitive programming platforms (LeetCode, Codeforces, AtCoder) and scores models on problems released after their training cutoffs. It evaluates code generation, self-repair, code execution, and test output prediction.

• B: Coding agentTests, static analysis, sandbox invariants, trajectory checks and a calibrated readability judge: an agent eval as a portfolio, not a score.