Harness architecture and reproducibility

An eval harness is the machinery that administers an evaluation: it provisions environments, executes trials, records them, grades, and aggregates. It belongs to the instrument, so a benchmark is not a dataset plus a metric but an executable protocol, and a reported number is only as reproducible as the harness that produced it.

A benchmark is an executable protocol#

Agent capability is observed only through an interface. Token limits, tool schemas, retry policies, filesystem permissions, browser state, simulator behavior, timeouts, and error handling can all change the measured result. The same tasks and metric, run through a different harness, are a different measurement.

Marr's implementation level names the machinery: judge model version, prompt, sampling settings, parser, test runner, sandbox, API state, database snapshot, permission system, retry policy, and logging. A valid idea can fail here through a parser bug or environment leak. The typical mistakes at this level are flaky tests, judge-version drift, leakage, and non-reproducible state, and none of them shows up in the claim or the rubric (Marr's levels for evaluation).

The framework draws the same line between mechanism and system: a checker is a mechanism, while a release gate that depends on it under a particular environment, version, and threshold is an evaluator system. A pass for SQLAgent, the text-to-SQL agent of Case C, comes not from an executor alone but from the executor, a pinned database snapshot, a result-set comparator, and the permission and retry constraints around them; change any of them and a pass means something else.

Before reading a low score as a capability fact, check that the harness lets the intended capability be expressed: a strong model scoring near 0% is overwhelmingly a task or harness bug (Anthropic, 2026). Population, environment, harness, and validity domain covers this compatibility check.

Two harnesses#

Two harnesses must be kept apart:

• the eval harness, which provisions environments, executes trials, records, grades, and aggregates, and is part of the instrument;
• the agent harness, the scaffold that lets the model act, made of prompts, tools, memory, retrieval, skills, orchestration, permissions, and stopping logic, which is part of the system under test.

Collapsing them invites a classic confound: a "model regression" that is actually the scaffold's timeout bug. The per-trial record below is what exposes it. When a new version's failures concentrate in trials that ended in timeouts or tool errors rather than in wrong answers, the harness is the first suspect, not the model.

The boundary matters most when the agent can edit its own harness. Harnesses are inspectable and editable, including potentially by the agent itself, which makes harness self-modification a concrete form of self-modification even when the model's weights stay frozen. The agent harness can then sit inside the modification surface; the eval harness, its tests, and its reference anchors must not (Exposure, modification, and access). The coding-agent case enforces this by copying the grading tests in from outside the agent's workspace after the agent finishes (Case B).

Isolated trials and the per-trial record#

Every trial should start from clean state. Shared state produces correlated failures, and correlated trials silently break the independence that every interval assumes; when correlation cannot be removed, resample at the cluster level instead of pretending it is absent. The tell is failures that cluster in time or infrastructure rather than in task space, as when a shared flaky dependency or a rate-limit storm makes outcomes co-move. Estimands, pass rates, and intervals covers resampling by task.

At minimum, every trial should persist the fields below, shown here with what they hold for SQLAgent:

Without this record, a surprising number is not an experimental result; it is an unreproducible event. The record is also what makes the earlier diagnosis possible: a regression that lives in the timeout column is a harness finding.

Pinned environments and simulated users#

Stateful evaluation needs realism without uncontrolled mutability. A website can be realistic because its behavior is modeled faithfully, without being the live public site; containerized services, database snapshots, recorded API fixtures, simulators, and versioned policy documents make stateful evaluation reproducible. When a live dependency is unavoidable, report it as a validity limitation, record enough state to identify drift, and do not rank results from materially different environment states on one leaderboard as if they were exchangeable trials. This is the run-level half of versioned dynamism: each version of a suite refreshes its corpus, and each run freezes its world (Living eval suites).

DocsAgent, the documentation agent of Case G, shows why. Its answers depend on the documentation version and the retrieval index, so a run that silently picked up last night's documents is a different measurement; freezing the corpus and index for each run, and making a new benchmark version when the documents drift, keeps old results reproducible.

Multi-turn evaluation needs someone to play the user. Scripted turns are cheap and brittle; simulated users, built by prompting an LLM with personas mined from real conversation logs, suit capability evals better; recorded-replay hybrids sit in between. Two disciplines keep a simulator honest. Audit its fidelity against real logs on a schedule, because a simulator drifts exactly like a judge does, and remember that the simulator is environment, not system, so its quirks can manufacture both fake failures and fake passes. Version it with the environment; Evaluating agents covers simulated users and dual-control environments.

The harness also decides which adversaries you can run: tree search over a conversation needs snapshot and restore, since you cannot branch an episode you cannot fork, and regression on discovered failures needs deterministic replay. Falsification and failure regions lists what each search method needs; decide which adversaries the risk profile requires before you build the harness.

The evaluator as an agent#

Some agent-evaluation platforms put the evaluator itself inside an agent. The Berkeley lecture calls it a green agent: a hosting agent that defines the assessment and its tasks, while the agents under test are participant, or white, agents (Song, 2025, slides 65–66). The green agent prepares the environment, distributes tasks, collects results, verifies the environment, and reports back to the platform (slide 67), and it bundles a dataset of tasks, a predefined testing process, and the environment the tasks run in (slide 70). In this handbook's terms, an evaluator agent is an eval harness, a specification, and a mechanism packaged together, and the lecture's own summary says why that matters: "The green agent dictates what metrics get measured" (slide 69). Version it as one measurement.

When the lecture wraps an existing benchmark in a green agent, its first step is to sort out the interface, on two principles: a human should be able to solve the task if presented with the same interface, and the solving procedure should be as agent-friendly as possible (slide 91). Its worked example, τ-bench, shows the two hard parts (slides 93–94). The original benchmark handed tools directly to a completion interface, and under a standardized agent interface every replacement has a cost: a special participant agent with tool access is less standardized; explaining the tool request to the participant and asking for tool names and arguments bypasses the agent's own tool-calling machinery; and an MCP server requires dynamic discovery. Once evaluation moves into the green agent, the tool trace is no longer directly visible to it.

Each of those choices is a different measurement. An interface that requires dynamic discovery tests discovery, which belongs in the construct only if the deployment interface requires it. An invisible tool trace is an evidence view problem: a grader that cannot see the trace cannot check what the trace would show, so the interface must deliver the evidence the specification needs (Anchors and evidence views). The lecture's last step is to make the green agent reproducible and openly accessible, which means settling how participants are reached, how the LLM API is accessed, how results and traces are reported, and how the repository is packaged for hosting (slide 99).

One measurement version#

Version the judgment, not only the judge. Rubric text, examples, judge model, tool checker, thresholds, aggregation logic, and expert policy together form one measurement version, and changing any load-bearing component requires revalidation or an explicit bridging study. A changed rubric, judge model, tool checker, or threshold creates a new measurement form even when the task corpus is unchanged.

Without that discipline you get judge rot: unversioned judge prompts, no recalibration after a model swap, a well-meaning edit that silently repoints the judge, and every downstream number inheriting the rot. If DocsAgent's grounding judge gets a clarified prompt, that is a new measurement version: recalibrate its error rates against the double-labeled SME set, and before comparing across the change, score a bridging sample under both versions (illustrative).

In practice#

Build a reproducibility record for one evaluator.

1. Draw the boundary. List every component and mark it agent harness (system under test) or eval harness (instrument). Keep grading code, tests, and reference anchors outside anything the agent can write.
2. Isolate trials. Start every trial from clean state, and check whether failures cluster in time or infrastructure rather than in task space.
3. Persist the per-trial record. Store the fields in the table above for every trial, within your privacy policy.
4. Pin the environment. Use snapshots, fixtures, and simulators, and record each unavoidable live dependency as a validity limitation, with enough state to detect drift.
5. Audit the simulators. Compare simulated users with real logs on a schedule, and version them with the environment.
6. Version the measurement. Issue a new measurement version for any change to rubric, examples, judge, checker, thresholds, or aggregation, and run a bridging study before comparing across it.
7. Reproduce before you report. Rerun a sample of trials from their records and confirm that the reported number comes back.

The artifact is a reproducibility record, which fills the reproducibility entry of the evaluator dossier: harness and configuration, environment snapshots, raw traces or privacy-compatible summaries, and exact model identifiers (see Templates).

Common failure modes#

• Scaffold bug read as model regression — a score drops after an upgrade and the model is blamed → read the per-trial record for timeouts, retries, and tool errors, and diff the harness versions before concluding anything. See Task validity and outcome validity.
• Correlated trials — failures cluster on one host or in one hour while the interval assumes independence → isolate trials and resample at the task or cluster level. See correlated trials.
• Judge rot — a well-meaning prompt edit changes every downstream number → version the whole measurement and recalibrate on every change. See judge rot.
• Live state on one leaderboard — results from different environment states are ranked together → pin or record the state, and keep different states in different versions. See Living eval suites.

Check yourself#

Analyze. After a model upgrade, an agent benchmark's pass rate drops sharply. The per-trial records show that the new failures end mostly in timeouts and tool errors, while failures with a wrong final answer are as frequent as before. What does the drop support, and what would you read next?

Hint

Failures that live in the timeout and error columns implicate the harness before the model: the classic confound is a "model regression" that is really the scaffold's timeout bug. Diff the agent-harness and eval-harness versions, compare retries and latency per trial, and rerun a sample of trials from their records.

Evaluate. A team replaces its scripted user with an LLM-simulated user, and the pass rate jumps. Is that a capability gain?

Hint

The simulator is environment, not system, and its quirks can manufacture fake passes. Treat the switch as a new measurement version, audit the simulator against real logs, and compare the agent under both simulators on a bridging sample.

Create. Wrap an existing benchmark in an evaluator agent. Specify the interface, what the grader must see, the per-trial record it writes, and what triggers a new measurement version.

Hint

Start from the two interface principles (Song, 2025, slide 91), then check whether your interface hides the tool trace from the grader (slides 93–94) or tests a skill, such as dynamic tool discovery, that the claim does not include.

Further reading#

• Population, environment, harness, and validity domain — the harness in the claim, and capability–harness compatibility.
• Reliability and generalizability — the replication facets that isolation and versioning keep measurable.
• Living eval suites — versioned dynamism across the versions of a suite.
• Evaluating agents — trajectories, simulated users, and dual-control environments.
• Song (2025), "Agent evaluation & project overview" — slides 64–71 on green agents and slides 90–99 on wrapping τ-bench.