# SWE-bench

> A benchmark of real-world software engineering tasks drawn from actual GitHub issues and pull requests in popular Python repositories. Models must generate patches that resolve genuine issues and pass the corresponding test suites.

## Claim

- **headline**: SWE-bench Verified resolve rates support comparing complete agent systems (model plus scaffold) on fixing real issues in 12 popular open-source Python repositories, as judged by the fix's own tests; they do not isolate model capability or extend to other languages or closed codebases.
- **construct**: Practical software maintenance: diagnosing a reported issue in a real codebase and producing a working fix
- **population**: Real GitHub issues and the merged pull requests that fixed them in 12 popular Python repositories; the 500-task, human-screened Verified subset is the usual reporting set
- **measures**: Ability to understand real codebases, diagnose issues, and produce working fixes; Software engineering competence including code navigation, debugging, and patch generation; Performance on authentic tasks sourced from open-source Python project histories; End-to-end problem-solving from issue description to validated code change; Practical utility as a coding assistant on realistic software maintenance tasks
- **doesNotMeasure**: Greenfield software development or architectural design ability; Performance on languages other than Python or on non-open-source codebases; Code review quality, documentation writing, or collaborative engineering skills; Efficiency or resource usage of the generated solutions

## Instrument

- **itemFormat**: code
- **specification**: A task is resolved when, after the generated patch is applied, the tests that the reference pull request made pass now pass and the previously passing tests still pass.
- **anchor**: Tests from the merged reference pull request (fail-to-pass) plus the repository's existing passing tests (pass-to-pass)
- **evidenceView**: The repository state after the patch, observed only through test outcomes; the agent's trajectory is not graded
- **mechanism**: execution
- **mechanismDetail**: The patch is applied in a containerized copy of the repository at the pre-fix commit and the specified tests are run. Passing means no tested counterexample; behavior the tests do not cover goes unchecked.
- **signal**: Binary resolved/unresolved per task
- **resolution**: Final patch per task
- **aggregation**: Percentage of tasks resolved
- **metrics**: % resolved
- **size**: 2,294 task instances (Verified: 500)
- **methodology**: Collects task instances from 12 popular Python repositories by identifying merged pull requests that fix issues and have associated test changes. Each instance provides the issue description, repository state before the fix, and test cases added by the PR. Models must generate a patch that, when applied, makes the failing tests pass without breaking existing tests. SWE-bench Verified is a human-validated subset of 500 instances with confirmed solvability.

## Regime

- **typicalUses**: measure; optimize
- **exposure**: Tasks are public and the reference fixes sit in public repository histories, so pretraining contamination is likely for fixes that predate a model's cutoff. Verified is the headline agentic-coding number in model releases and on a public leaderboard, and scaffolds are tuned against it; the same issue-plus-tests format is widely used to build training environments. All grading tests are public.
- **notes**: Scores describe whole systems: the same model can score very differently across scaffolds, retrieval strategies, tool budgets, and attempt counts. Report all of them with every score.

## Validation

- **summary**: SWE-bench's evidence supports claims about real software-maintenance capability more directly than function-level benchmarks, because it grades fixes to real issues with the repositories' own tests; the human-screened Verified subset addresses quality problems in the original set. Scores reflect the entire agent system — scaffolding, retrieval, and tool use — as much as the underlying model, so they support system comparisons more readily than pure model comparisons.
- **evidence**: Tasks are solvable and fairly graded; Passing the tests means the issue is correctly fixed; Scores isolate model capability; Results transfer to other languages and closed codebases
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: high
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/swe-bench