# LiveCodeBench

> A code generation benchmark built to resist contamination: it continuously collects new problems from competitive programming platforms (LeetCode, Codeforces, AtCoder) and scores models on problems released after their training cutoffs. It evaluates code generation, self-repair, code execution, and test output prediction.

## Claim

- **headline**: LiveCodeBench pass@1 supports contamination-resistant comparisons of algorithmic code generation on competitive-programming problems released after a model's training cutoff; it does not support claims about real-world software engineering or code quality.
- **construct**: Algorithmic problem solving in code, with related skills: self-repair and predicting program behavior
- **population**: Competitive-programming problems from LeetCode, AtCoder, and Codeforces, filtered by release date relative to each model's training cutoff
- **measures**: Algorithmic problem-solving on genuinely unseen competitive programming problems; Code generation ability with minimal contamination risk due to continuous temporal updates; Self-repair capability — whether models can fix their own incorrect solutions given test feedback; Test output prediction — understanding program behavior without executing code; Performance trends over time as new problems are continuously added
- **doesNotMeasure**: Real-world software engineering — competitive programming is a specific and narrow skill; Ability to work with large codebases, APIs, or existing software systems; Code quality beyond functional correctness — competitive code is notoriously unreadable; Collaborative development, code review, or software design capabilities

## Instrument

- **itemFormat**: code
- **specification**: A generated program passes when it produces the expected output on every test case for the problem within the time limit.
- **anchor**: Test inputs with expected outputs for each problem, collected from the platforms or generated
- **evidenceView**: The program's outputs on the test inputs; for the output-prediction scenarios, the predicted output
- **mechanism**: execution
- **mechanismDetail**: Generated programs run against the test cases; any wrong output, error, or timeout fails the sample. The code-execution and test-output-prediction scenarios compare a predicted output with the reference by exact match.
- **signal**: Binary pass/fail per sample
- **resolution**: Final program (or predicted output) per problem
- **aggregation**: pass@1 and pass@5 (at least one of k samples passes), averaged over the problems in a release-date window
- **metrics**: pass@1; pass@5
- **size**: Continuously growing (400+ problems at launch)
- **methodology**: Continuously scrapes new problems from LeetCode, Codeforces, and AtCoder after a defined cutoff date. Problems include input/output specifications and test cases. Models generate solutions that are tested against the full test suite. Evaluates four scenarios: direct code generation, self-repair after seeing failed test cases, code execution (predicting a program's output), and test output prediction. Temporal filtering keeps only problems published after a model's training cutoff, so they cannot have appeared in its training data, provided the stated cutoff is accurate.

## Regime

- **typicalUses**: measure
- **exposure**: Every problem is public from the day it is published, so a time window is clean only for models whose training cutoff precedes it, and only until later training runs absorb it. It is a headline coding number in reasoning-model releases, and the same platforms are a major source of reinforcement-learning data, so windows age quickly.

## Validation

- **summary**: LiveCodeBench's evidence supports contamination-resistant claims about algorithmic code generation: release-date filtering against model training cutoffs is the control that static benchmarks lack. Competitive programming is a narrow lens on coding ability — a model that excels here may still struggle with real software engineering — so broader coding claims pair it with SWE-bench.
- **evidence**: Scoring only problems released after a model's training cutoff gives contamination-resistant results, provided the cutoff is accurately known; Indicates real-world software engineering ability; Scores from different time windows are comparable
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: low
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/livecodebench