ScienceAgentBench

CConstruct

Data-driven scientific computing: turning a task description and dataset into a program that produces the published result

PPopulation

Tasks extracted from peer-reviewed publications in bioinformatics, computational chemistry, geographical information science, and psychology and cognitive neuroscience

Measures

• Ability to perform data-driven scientific tasks end-to-end, from data processing to result generation
• Code generation for scientific computing including data analysis, visualization, and statistical modeling
• Understanding of scientific methodologies well enough to reproduce published experimental results
• Integration of domain knowledge with programming ability in realistic research workflows

Does not measure

• Scientific creativity, hypothesis generation, or the ability to design novel experiments
• Literature review, paper writing, or scientific communication skills
• Wet-lab experimental ability or physical scientific tasks
• Collaboration with human scientists or interactive research workflows

MMechanism

Hybrid portfolio

The program is executed and task-specific checks compare its outputs with the gold results (execution); CodeBERTScore compares the program with the gold program as a secondary, embedding-based similarity (a weak comparator).

Hierarchy prior: mixed. Each channel carries the prior of its own mechanism. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format

Code

ΨSpecification

A task succeeds when the agent's self-contained Python program runs and its outputs (predictions, tables, or figures) meet the task's success criteria against the gold results.

AAnchor

Gold programs and outputs annotated from the source papers and validated by subject-matter experts

VEvidence view

The generated program and the outputs it produces when executed

ZSignal

Binary success per task, plus a similarity score

RResolution

Final program and its outputs per task

ΓAggregation

Success rate over tasks; mean CodeBERTScore

Metrics

success rate, CodeBERTScore

Size

102 tasks from 44 peer-reviewed publications

Methodology

Draws 102 tasks from 44 peer-reviewed publications in four disciplines: bioinformatics, computational chemistry, geographical information science, and psychology and cognitive neuroscience; subject-matter experts validated the tasks. Each task provides a dataset, task description, and expected output (figures, tables, or numerical results). Agents must write Python code to process the data and produce outputs that match the paper's results. Evaluation compares generated outputs against gold references using both automated metrics and programmatic checks.

Typical uses

• Measure · supports inference and decision

Exposure

Public since 2024. Source papers, datasets, and often their original code are public, so reference solutions may appear in training data; the authors added two strategies to mitigate contamination.

ScienceAgentBench's evidence supports claims about agents assisting with data-driven research computations: grounding tasks in real publications, with expert validation, gives an ecological validity that synthetic tasks lack. The small task count is a legitimate concern for statistical power, though each task is rich, and reproducing published results covers only part of scientific work.

Evidence notes1 supported · 1 partially supported · 2 untested

1. Supported

   Tasks reflect authentic research computations

   Tasks were extracted from peer-reviewed publications and validated over multiple rounds by annotators and subject-matter experts.

   Source: Chen et al., 2024

2. Partially supported

   Success rates distinguish agents reliably

   Binomial arithmetic: with 102 tasks, a 95% interval around 30% success spans roughly 22–39%, so only large gaps are reliable.

3. Untested

   Reproducing a published analysis indicates scientific ability

   Hypothesis generation, experiment design, and interpretation are outside the task pool; reproduction is an imperfect proxy.

4. Untested

   Program similarity (CodeBERTScore) indicates correctness

   Similarity to one reference program is a weak comparator: a correct program written differently can score low, so it is secondary to success rate.

Saturation

Emerging

Whether top systems still differ, or crowd the ceiling.

Contamination

Medium risk

How likely test items, or close variants, have leaked into training data.

Successors

None listed

Last reviewed

Not yet reviewed · card updated October 12, 2026

• GAIAAgents · 2023General AI Assistants benchmark evaluates AI systems on real-world tasks requiring multi-step reasoning, web browsing, tool use, and file manipulation. Questions have unambiguous, verifiable answers but require complex workflows to solve.
• SWE-benchCoding · 2023A benchmark of real-world software engineering tasks drawn from actual GitHub issues and pull requests in popular Python repositories. Models must generate patches that resolve genuine issues and pass the corresponding test suites.
• AssistantBenchAgents · 2024A benchmark of challenging, realistic web tasks that require an assistant to navigate the open web to find precise, verifiable answers. Tasks span planning, fact-checking, data collection, and complex information retrieval.

• D: Autonomous research agent"Research quality" is not one unverifiable blob. Route provenance, reproducibility, entailment, novelty and direction-setting to different channels.