LiveCodeBench

CConstruct

Algorithmic problem solving in code, with related skills: self-repair and predicting program behavior

PPopulation

Competitive-programming problems from LeetCode, AtCoder, and Codeforces, filtered by release date relative to each model's training cutoff

Measures

• Algorithmic problem-solving on genuinely unseen competitive programming problems
• Code generation ability with minimal contamination risk due to continuous temporal updates
• Self-repair capability — whether models can fix their own incorrect solutions given test feedback
• Test output prediction — understanding program behavior without executing code
• Performance trends over time as new problems are continuously added

Does not measure

• Real-world software engineering — competitive programming is a specific and narrow skill
• Ability to work with large codebases, APIs, or existing software systems
• Code quality beyond functional correctness — competitive code is notoriously unreadable
• Collaborative development, code review, or software design capabilities

MMechanism

Execution / effect checker

Generated programs run against the test cases; any wrong output, error, or timeout fails the sample. The code-execution and test-output-prediction scenarios compare a predicted output with the reference by exact match.

Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format

Code

ΨSpecification

A generated program passes when it produces the expected output on every test case for the problem within the time limit.

AAnchor

Test inputs with expected outputs for each problem, collected from the platforms or generated

VEvidence view

The program's outputs on the test inputs; for the output-prediction scenarios, the predicted output

ZSignal

Binary pass/fail per sample

RResolution

Final program (or predicted output) per problem

ΓAggregation

pass@1 and pass@5 (at least one of k samples passes), averaged over the problems in a release-date window

Metrics

pass@1, pass@5

Size

Continuously growing (400+ problems at launch)

Methodology

Continuously scrapes new problems from LeetCode, Codeforces, and AtCoder after a defined cutoff date. Problems include input/output specifications and test cases. Models generate solutions that are tested against the full test suite. Evaluates four scenarios: direct code generation, self-repair after seeing failed test cases, code execution (predicting a program's output), and test output prediction. Temporal filtering keeps only problems published after a model's training cutoff, so they cannot have appeared in its training data, provided the stated cutoff is accurate.

Typical uses

• Measure · supports inference and decision

Exposure

Every problem is public from the day it is published, so a time window is clean only for models whose training cutoff precedes it, and only until later training runs absorb it. It is a headline coding number in reasoning-model releases, and the same platforms are a major source of reinforcement-learning data, so windows age quickly.

LiveCodeBench's evidence supports contamination-resistant claims about algorithmic code generation: release-date filtering against model training cutoffs is the control that static benchmarks lack. Competitive programming is a narrow lens on coding ability — a model that excels here may still struggle with real software engineering — so broader coding claims pair it with SWE-bench.

Evidence notes1 supported · 1 partially supported · 1 not supported

1. Supported

   Scoring only problems released after a model's training cutoff gives contamination-resistant results, provided the cutoff is accurately known

   The authors observed some models' accuracy dropping sharply on problems released after their training cutoffs, consistent with contamination of earlier problems that date filtering removes.

   Source: Jain et al., 2024

2. Not supported

   Indicates real-world software engineering ability

   Competitive programming emphasizes algorithmic tricks; codebases, APIs, design, and code quality are outside the item pool.

3. Partially supported

   Scores from different time windows are comparable

   Problems carry platform difficulty labels, but the mix depends on what platforms publish rather than deliberate calibration; compare models on the same window.

Saturation

Active

Whether top systems still differ, or crowd the ceiling.

Contamination

Low risk

How likely test items, or close variants, have leaked into training data.

Successors

None listed

Last reviewed

Not yet reviewed · card updated October 12, 2026

• HumanEvalCoding · 2021A set of 164 hand-written Python programming problems with function signatures, docstrings, and unit tests. It was the first widely-adopted code generation benchmark but is now far too easy and small to be informative.
• MBPPCoding · 2021Mostly Basic Python Programming contains 974 crowd-sourced Python programming tasks designed to be solvable by entry-level programmers. It complements HumanEval with more problems but at a similar or lower difficulty level.
• SWE-benchCoding · 2023A benchmark of real-world software engineering tasks drawn from actual GitHub issues and pull requests in popular Python repositories. Models must generate patches that resolve genuine issues and pass the corresponding test suites.

• B: Coding agentTests, static analysis, sandbox invariants, trajectory checks and a calibrated readability judge: an agent eval as a portfolio, not a score.