# ScienceAgentBench

> A benchmark for evaluating AI agents on data-driven scientific discovery tasks drawn from real published research papers. Agents must write code to process datasets, run experiments, and produce results that match the original paper's findings.

## Claim

- **headline**: ScienceAgentBench success rates support comparing agents on writing Python programs that reproduce data-driven analyses from peer-reviewed papers in four disciplines; the small task set limits precision, and they say nothing about hypothesis generation or the wider research process.
- **construct**: Data-driven scientific computing: turning a task description and dataset into a program that produces the published result
- **population**: Tasks extracted from peer-reviewed publications in bioinformatics, computational chemistry, geographical information science, and psychology and cognitive neuroscience
- **measures**: Ability to perform data-driven scientific tasks end-to-end, from data processing to result generation; Code generation for scientific computing including data analysis, visualization, and statistical modeling; Understanding of scientific methodologies well enough to reproduce published experimental results; Integration of domain knowledge with programming ability in realistic research workflows
- **doesNotMeasure**: Scientific creativity, hypothesis generation, or the ability to design novel experiments; Literature review, paper writing, or scientific communication skills; Wet-lab experimental ability or physical scientific tasks; Collaboration with human scientists or interactive research workflows

## Instrument

- **itemFormat**: code
- **specification**: A task succeeds when the agent's self-contained Python program runs and its outputs (predictions, tables, or figures) meet the task's success criteria against the gold results.
- **anchor**: Gold programs and outputs annotated from the source papers and validated by subject-matter experts
- **evidenceView**: The generated program and the outputs it produces when executed
- **mechanism**: hybrid
- **mechanismDetail**: The program is executed and task-specific checks compare its outputs with the gold results (execution); CodeBERTScore compares the program with the gold program as a secondary, embedding-based similarity (a weak comparator).
- **signal**: Binary success per task, plus a similarity score
- **resolution**: Final program and its outputs per task
- **aggregation**: Success rate over tasks; mean CodeBERTScore
- **metrics**: success rate; CodeBERTScore
- **size**: 102 tasks from 44 peer-reviewed publications
- **methodology**: Draws 102 tasks from 44 peer-reviewed publications in four disciplines: bioinformatics, computational chemistry, geographical information science, and psychology and cognitive neuroscience; subject-matter experts validated the tasks. Each task provides a dataset, task description, and expected output (figures, tables, or numerical results). Agents must write Python code to process the data and produce outputs that match the paper's results. Evaluation compares generated outputs against gold references using both automated metrics and programmatic checks.

## Regime

- **typicalUses**: measure
- **exposure**: Public since 2024. Source papers, datasets, and often their original code are public, so reference solutions may appear in training data; the authors added two strategies to mitigate contamination.

## Validation

- **summary**: ScienceAgentBench's evidence supports claims about agents assisting with data-driven research computations: grounding tasks in real publications, with expert validation, gives an ecological validity that synthetic tasks lack. The small task count is a legitimate concern for statistical power, though each task is rich, and reproducing published results covers only part of scientific work.
- **evidence**: Tasks reflect authentic research computations; Success rates distinguish agents reliably; Reproducing a published analysis indicates scientific ability; Program similarity (CodeBERTScore) indicates correctness
- **status**: draft

## Lifecycle

- **saturation**: emerging
- **contamination**: medium
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/scienceagentbench