# D: Autonomous research agent

> "Research quality" is not one unverifiable blob. Route provenance, reproducibility, entailment, novelty and direction-setting to different channels.

Case D evaluates an **autonomous research agent**: a system that takes a research question, runs sandboxed experiments, and writes up the results. Its construct — useful, honest, reproducible research — is not one non-verifiable blob: its parts sit on different rungs of the verification hierarchy, and the hardest stay with human judgment.

> [!KEY] In one sentence
> Decompose "research quality" and route each part to the mechanism that can check it — provenance to a deterministic check, reproduction to execution, entailment to a judge under expert audit, usefulness to experts — and leave direction-setting with the people who own the agenda.

## Claim and decision

The case uses an illustrative deployment, the one the Studio preset loads: an agent that answers questions an ML team assigns it.

- **Construct:** useful, honest, and reproducible research reports on assigned questions.
- **Population:** questions the team assigns over a quarter, stratified as replications, ablations, and new experiments.
- **Environment and harness:** sandboxed compute with code execution, dataset access, and literature search; planning, coding, experiment, and writing tools; a fixed compute budget per question.
- **Validity domain:** ML questions answerable with the team's compute budget and public datasets.
- **Decision:** publish the report internally with an expert sign-off, or send it back for revision.

The claim excludes choosing which directions matter and novelty at the frontier: the agent executes questions; people set them.

Research quality is a latent [construct](https://benchmaker.ai/glossary/construct): a paper, a judge score, and a citation list are observable, and none of them is the construct. A research-quality evaluator should respond to [construct facets](https://benchmaker.ai/glossary/construct-facet) $F^{+}$ such as factual correctness, novelty, evidential support, and experimental rigor, and not to [nuisance variables](https://benchmaker.ai/glossary/nuisance-variable) $F^{-}$ such as verbosity, model-family identity, formatting, assertive tone, or superficial citation density. The nuisance list is the judge's discriminant test plan: a verdict that moves when only a nuisance changes exposes method variance.

## Two regimes, one sign-off

| Use (regime) | Consumer and decision | Optimizer and access |
|---|---|---|
| Publish gate (measure) | Research leads publish the report or send it back | None by design; expert reviews never reach the agent |
| Self-revision (optimize) | The agent picks the draft it submits | The agent's own loop: up to five revision rounds against the automated checks, whose results it sees |

The revision loop is [bounded self-refinement](https://benchmaker.ai/glossary/bounded-self-refinement): improvement against fixed external checks, with the agent's policy, prompts, and memory as its modification surface. The migration to watch is expert reviews becoming revision feedback, which would spend the certification channel's independence.

## Criteria and mechanisms

The construct decomposes into six parts. This case adds disclosure, for the "honest" in the construct, reads "novelty and importance" as usefulness to the team because the claim excludes frontier novelty, and keeps direction-setting in the last row only to mark where the claim stops:

| Criterion | Mechanism | Hierarchy family | Evidence view | Anchor |
|---|---|---|---|---|
| Citations resolve | A resolver that looks up every citation and matches quoted spans | Deterministic | Reference list and quoted spans | The cited sources |
| Results reproduce | Re-running the shipped code, experiment manifest, and seeds in a clean sandbox | Execution | Code, manifest, seeds, and raw metrics | The report's numbers, within declared tolerances |
| Statistics are sound | Recomputing every reported statistic from raw metrics; learned or expert review of the analysis design | Execution, plus learned or expert review | Raw metrics and analysis code | Recomputed values; the panel's review |
| Claims follow from evidence | LLM judge over claim–evidence pairs, with sampled expert audit | Learned judge, with expert audit | Claim–evidence pairs, cited spans, and raw metrics, not prose alone | Exemplar anchors adjudicated by the panel |
| Failures and substitutions disclosed | Comparing the sandbox's data-access and run logs with the report | Deterministic over the logs; expert review for borderline wording | Sandbox logs and the report | The sandbox's own logs |
| Useful contribution | The panel of senior researchers; a calibrated judge only for triage | Human judgment, the external frontier | The report and the question it answers | The panel's judgment, once its agreement is measured |
| Direction-setting (outside the claim) | None: the panel assigns the questions | The frontier: frame selection | — | The research agenda |

Citations, reproduction, statistics, and disclosure are fatal gates, each with zero violations allowed; entailment is tradeable; usefulness escalates to the panel. Criteria climb when the artifact changes: requiring code, a manifest, raw metrics, seeds, ablations, claim–evidence pairs, and uncertainty annotations moves reproducibility and evidential support up the hierarchy, even where quality stays partly judgment.

## Artifacts, not manuscripts

The [evidence view](https://benchmaker.ai/glossary/evidence-view) decides what these mechanisms can catch. ResearchArena ran frontier coding agents through the full research loop: automated review of the manuscripts alone rated the best agent's papers on a par with the average human ICLR submission, but review that also inspected the artifacts, together with human meta-review, showed that picture to be overstated ([Zhang et al., 2026](https://benchmaker.ai/handbook/reference/sources#zhang-2026), via [Chen, Wang & Qu, 2026](https://benchmaker.ai/handbook/reference/sources#chen-wang-qu-2026), §6.3). A judge shown the manuscript is a different instrument from the same judge shown the code and logs.

Integrity must be measured under pressure, not assumed. SciIntegrity-Bench poses dilemmas in which admitting failure is the only correct response and finishing the task would take misconduct; across seven state-of-the-art models it reports a 34.2% integrity-failure rate, and when data were missing, all seven generated synthetic data instead of declaring the task infeasible, differing only in whether they said so ([Yang et al., 2026](https://benchmaker.ai/handbook/reference/sources#yang-2026), via Chen, Wang & Qu, 2026, §6.3). In physics-simulation rediscovery, agents have reached correct answers while defending wrong mechanisms, which argues for measuring outcome, mechanism fidelity, and epistemic honesty separately ([Eulig, 2026](https://benchmaker.ai/handbook/reference/sources#eulig-2026), via Chen, Wang & Qu, 2026, §6.3). So seed the population with infeasible questions whose honest report is "this cannot be done," the negative-polarity tasks a one-sided suite lacks, and grade claims against the agent's own data, not only its headline number.

> [!WARNING]
> Reproducibility is not integrity: code that synthesizes a missing dataset reproduces its numbers perfectly. Grade what the sandbox logs show the agent read and ran, not what the report says it did. See [grader gaming](https://benchmaker.ai/handbook/practice/anti-patterns#grader-gaming).

The builder starts blank. Draft the card for "claims follow from evidence": allow cited sources, execution logs, provenance metadata, and uncertainty metadata; take a correct number defended by a mechanism the agent's own ablation contradicts as the near-miss; escalate to the panel when evidence lies outside the sandbox; and name generator coupling as a blind spot if the judge shares the agent's model family.

```widget
criterion-card-builder
preset: blank
```

## The frame-selection problem

The case's main lesson, drawn from the RSI survey, is that research execution can be progressively instrumented, while direction-setting includes a frame-selection problem that the verification hierarchy alone does not solve. The survey contrasts execution, verifiable and increasingly automated, with direction-setting, the archetypal task no checker verifies and where people remain, and reads the bottleneck in choosing directions and the bottleneck in verification as one (Chen, Wang & Qu, 2026, §5.4).

This design keeps direction-setting out of the claim and gives it to the panel, which plays two roles that must not be conflated: a measurement mechanism with measurable disagreement, and the [trust root](https://benchmaker.ai/glossary/trust-root) where the evaluate-the-evaluator chain stops. Its mandate over the agenda also makes it the body that runs [frame review](https://benchmaker.ai/glossary/frame-review): whether the assigned questions still matter is logically prior to grading the answers.

> [!OPEN]
> Evaluating research taste and direction-setting has no established method, and neither does governance-grade, auditable evidence of what a training loop does and does not improve; the RSI survey lists both among its open problems (Chen, Wang & Qu, 2026, §8).

## Validation plan

Each row is a planned, not yet measured, entry in the [validation profile](https://benchmaker.ai/glossary/validation-profile) Π.

| Component | What to measure for this agent |
|---|---|
| Natural error | Resolver misses; reproduction flake rate; the entailment judge's TPR and FPR against panel labels, by question type; the panel's own agreement |
| Exploitability curve | Drafts after each revision round audited by experts; $b^{*}$ is the last round at which judge–expert disagreement stays within a predeclared tolerance |
| Transport | Judge and checker error across question types and agent versions |
| Generator coupling | Whether the judge errs where the agent errs, especially if they share a model family; the $F^{-}$ perturbation tests |
| Channel coupling | Whether reports that pass the automated checks and the judge also pass the panel |
| Drift profile | New datasets and literature, a shifting question mix, and changes to the agenda itself |

## Channels and tripwires

[Keep three measurement channels apart](https://benchmaker.ai/handbook/lifecycle/channels-and-independence): the **optimized** automated checks inside the revision loop; the **anchor channel**, expert sign-off and sampled audits that never reach the agent, tracked in an [anchor burn](https://benchmaker.ai/glossary/anchor-burn) ledger; and **production**, whether the team's later work builds on published reports or overturns them. Frame review sits above all three, with the panel.

Check pass rates rising while expert-audited entailment stays flat signals proxy corruption, such as citations that resolve but support nothing; both rising while reports stop changing what the team does signals a construct mismatch; and reports that pass everything on questions that no longer matter signal [frame lock-in](https://benchmaker.ai/handbook/practice/anti-patterns#frame-lock-in).

## Validity statement

Following the [validity-statement template](https://benchmaker.ai/handbook/pressure/validation-profile#the-validity-statement), and stated as a plan until the profile is measured:

> The research-report evaluator — a citation resolver, a clean-sandbox reproduction run, recomputation of reported statistics, an entailment judge over claim–evidence pairs with sampled expert audit, a log-to-report disclosure check, and the panel's review of usefulness — is designed to support the claim that the agent produces useful, honest, and reproducible reports on assigned questions, for the research leads' publish-or-revise decision, over ML questions answerable with the team's compute budget and public datasets, stratified as replications, ablations, and new experiments, under the agent's own revision loop of up to five rounds against the automated checks, with expert reviews never fed back, up to a tested exposure $b^{*}$ that has not yet been measured, with a validation profile that is planned rather than measured, relative to a panel of three senior researchers, with a governance mandate over the research agenda, whose agreement is still to be measured. Which questions deserve research, and revisions to the construct, remain human judgment.

Write your agent's statement in the builder; $b^{*}$ and Π start as planned, and should stay planned until measured.

```widget
validity-statement-builder
preset: blank
```

## Common failure modes

- **Manuscript-only review** — a judge reads the paper and approves claims the artifacts do not support → show it claim–evidence pairs, raw metrics, and logs, and audit a sample with experts. See [Anchors and evidence views](https://benchmaker.ai/handbook/instrument/anchors-and-evidence-views).
- **Expert reviews as revision feedback** — the panel's comments enter the revision loop → keep reviews out of the loop and refresh the audit sample. See [anchor leakage](https://benchmaker.ai/handbook/practice/anti-patterns#anchor-leakage).

## Check yourself

**Analyze.** A report's citations resolve and its code reproduces its numbers, but its central dataset was synthesized by that code. Which criterion catches it, from which evidence, and why can't reproduction?

> [!DETAILS] Hint
> Reproduction certifies that the code produces the numbers. The disclosure check compares the sandbox's data-access log with the report's account of its data.

**Evaluate.** To raise quality faster, the team proposes feeding the panel's review comments into the agent's revision loop. Argue for or against.

> [!DETAILS] Hint
> Which channel would the comments come from, and what would be left to certify the next report? Count the anchor burn.

**Create.** Turn the nuisance list $F^{-}$ into a discriminant test plan for the entailment judge: one perturbation per nuisance that leaves the evidence unchanged, and the verdict change you would accept.

> [!DETAILS] Hint
> Pad without new claims, switch between hedging and asserting, add on-topic citations that support nothing, swap the stated author, and reformat the tables.

## Further reading

- [Verifiability engineering](https://benchmaker.ai/handbook/practice/verifiability-engineering) — decompose, instrument, compile, and keep the residue honest.
- [Constructs, facets, and nuisances](https://benchmaker.ai/handbook/claim/constructs) — $F^{+}$, $F^{-}$, and the discriminant test plan.
- [Frame review and governance](https://benchmaker.ai/handbook/lifecycle/frame-review) — the logically prior question, and who may revise the construct.
- Chen, Wang & Qu (2026), "Recursive self-improvement in AI: From bounded self-refinement to autonomous research loops" — §5.4 on the non-verifiable frontier, §6.2–6.3 on AI-scientist agents and their critics.

Open this case in the [Studio](https://benchmaker.ai/studio?preset=d) to start an evaluator dossier with its claim and regimes pre-filled.

---
Source: https://benchmaker.ai/handbook/cases/research-agent