Population, environment, harness, and validity domain

The validity domain is the declared set of conditions — model families, languages, task classes, environment versions, tool availability, prompt families, user populations, and risk conditions — over which an evaluation's interpretation is claimed to hold. Together with the population, environment, and harness, it says which world a score was earned in and where the claim stops.

Population and claim scope#

Four coordinates of the claim $\mathcal{C} = (C, P, E, H, D, \delta)$ describe the world an evaluation ran in and the world it speaks for: the population $P$, the environment $E$, the harness $H$, and the validity domain $D$. They matter because success depends on more than capability: task success is $\Pr[T = 1 \mid C, E, H]$, conditional on the environment and the harness too (see Evaluation is measurement). For SQLAgent, the text-to-SQL agent of Case C, they are analysts' questions, a pinned enterprise database snapshot, and a harness of a policy prompt, a schema-inspection tool, and a read-only query executor, bounded by a declared domain.

The population $P$ is the task, user, or input distribution the claim is about. How a finite eval set models it is the subject of Task distributions and dataset engineering; the question here is how broad a population the claim takes on. A benchmark's claim scope, a distinction drawn in the Berkeley agent-evaluation course, is whether it evaluates a specific capability, a specific application, or a general set of applications. The examples below come from the course's taxonomy (Song, 2025, slides 25–38):

State the scope explicitly, because the same score with the same tasks licenses different sentences under each reading. A suite that isolates plan generation supports a claim about one facet; it does not show that an agent plans well inside a customer-service workflow, which is an application claim with its own population.

Environment#

The environment $E$ is the external state and transition process the system interacts with, and it may be simulated, sandboxed, pinned, or live. It includes world state, tool backends, other agents or simulated users, stochastic disturbances, and transition dynamics. The line between environment and system under test is control: the system is what you ship and can change, from the model and prompts to memory, tools as exposed through the scaffold, and orchestration logic; the environment is what it acts on. Unlike a static text-to-text model, an agent acts in an environment whose state changes as it acts (Song, 2025, slide 8).

DocsAgent, the documentation agent of Case G, shows why the environment belongs in the claim. Its world includes document versions and a retrieval index. Its criterion card for grounded answers lists a blind spot: the documentation itself may be stale, and that belongs to the environment and validity domain rather than the judge. When the documents change, the world the score was earned in has changed, even if the model has not. An evaluation that faithfully measures performance in a world other than the deployment world is wrong in the way this handbook calls "wrong world".

Harness#

The harness is everything around the model that turns it into an agent: system instructions, tool definitions, memory stores, skill libraries, retrieval indices, orchestration logic, and stopping rules. In the claim, $H$ is the administration context, the harness under which the capability is claimed. It matters because agent capability is observed only through an interface: token limits, tool schemas, retry policies, filesystem permissions, browser state, simulator behavior, timeouts, and error handling can all change the measured result. So a benchmark is a protocol to be run, not only a set of tasks and a metric.

Keep two harnesses apart. The agent harness is part of the system under test; the eval harness, which provisions environments, runs and records trials, and grades them, is part of the instrument. Collapsing them invites a "model regression" that is actually the scaffold's timeout bug; Harness architecture and reproducibility covers the eval harness.

A claim about an agent is therefore a claim about a model in a harness. The coding agent in Case B is evaluated in a network-off container with file, shell, and test tools and 30 minutes per issue; the same model with a different budget or tool set is a different system under test.

Harnesses can also change after the evaluation: they can be inspected and edited from outside, including by the agent itself, which makes harness self-modification a concrete form of self-modification even when model weights stay frozen (Chen, Wang & Qu, 2026, §2.1). If developers or the agent can change the harness between evaluation and use, the claim must name the harness version it covers, and the harness belongs to the regime's modification surface (Exposure, modification, and access).

Capability–harness compatibility#

Before interpreting a low score, check that the harness makes the intended capability expressible. Does the model receive the tool descriptions it needs, and can it discover resources dynamically if deployment requires that skill? Is the context budget artificially below what the task requires? Do rate limits or browser quirks introduce failures that are not part of the construct? If SQLAgent's harness truncated schema listings below what multi-table questions need, a low score on joins would measure the context budget, not join reasoning (illustrative). This is a discriminant-validity issue as much as an engineering one. And as Anthropic (2026) warns, a strong model scoring near 0% on a task usually points to a task or harness bug, not a capability fact.

The validity domain#

Every evaluator should declare the model families, languages, task classes, environment versions, tool availability, prompt families, user populations, and risk conditions where the validation evidence is claimed to transport. Products, interaction lengths, permission levels, tool behavior, tool contracts, and the retrieval corpus can belong on the list too. The crosswalk maps the older term operational design domain onto the same object.

The population and the domain answer different questions. $P$ names the situations the estimate is about; $D$ names the conditions under which the evidence is claimed to carry over. $D$ reaches across every other coordinate, the evaluator included: a judge whose error rates were measured on one model family's outputs has not been validated on another's (The judge as a noisy sensor). For SQLAgent's release evaluation, a declaration might read as follows (illustrative, matching the Studio's Case C):

Outside the validity domain, the benchmark has not been shown wrong; it has simply made no justified claim. A SQLAgent pass rate on schema v12 says nothing, good or bad, about schema v13 until evidence covers it.

Declaring a dimension is not the same as showing that the measurement holds across it. A domain that spans English and Spanish, or two user groups, implicitly claims measurement invariance: that the measurement relation stays stable across groups, model families, languages, or conditions.

Validity is a claim, not a sticker#

The framework turns the domain into a sentence pattern. The statement "Benchmark B is valid" is incomplete. A defensible statement says that the evidence supports interpretation $C$ for population $P$, decision $\delta$, and regime $\mathcal{R}$ under validity domain $D$; change any of them, and the validation argument may need to be rebuilt. In one line: "72%" is incomplete; "72% for this population, environment version, policy, and time window" is a scientific statement.

For SQLAgent, "the SQL benchmark is valid" becomes (illustrative): the execution evidence supports reading the certification pass rate as correct, permission-respecting answers to English analytics questions, for the rollout decision, under developer iteration that never sees the certification set, within schema v12 on the pinned snapshot. The remaining slots of a full validity statement — consumer, tested exposure, validation profile, and trust root — are developed in The validation profile and validity statement.

The builder below loads an illustrative SQL-agent statement whose domain reads "English analytics questions over the finance warehouse, schema v12, on the pinned Q3 snapshot"; widen that field to every warehouse, and ask which other fields would then need new evidence.

Declaring $D$ offline is half the job. At runtime, a detector decides whether each live request falls inside $D$, under an explicit out-of-domain policy and with its own measured error rates; that belongs to Monitoring and control in production. Because documents, schemas, and tools change, the domain also needs a date and a revalidation trigger (Living eval suites).

In practice#

Write a validity-domain declaration for one evaluation you run.

1. State the population and scope. Name the task, user, or input distribution, and say whether the claim is about a capability, an application, or a general set of applications.
2. Describe the environment. List the state, tool backends, other actors, and disturbances, each marked simulated, sandboxed, pinned, or live, with its version.
3. Pin the harness. Record the agent harness the claim covers, apart from the eval harness that ran the trials, and note who can edit each, the agent included.
4. Check capability–harness compatibility. Confirm the harness makes the capability expressible before you read a low score.
5. Declare the domain. For each dimension in the domain table above, write what the evidence covers.
6. Write the out-of-domain policy. Decide whether an out-of-domain request is refused, escalated, run with reduced permissions, confirmed with the user, or routed to a safer process, and hand runtime detection to production monitoring.
7. Write the sentence. State what the evidence supports, for which population, decision, and regime, under which domain.

The artifact is a validity-domain declaration: the population and scope, the environment and harness versions, the domain table with the evidence behind each entry, and the out-of-domain policy. It fills the claim and validity-domain fields of the evaluator dossier and the domain slot of the validity statement.

Common failure modes#

• Headline extrapolation — a score earned on one schema, language, or tool version is quoted for all of them → declare $D$, and treat use outside it as unsupported. See one-number reporting.
• A harness bug read as a model regression — a scaffold timeout or a missing tool description shows up as lost capability → keep the agent and eval harnesses apart, and check compatibility before blaming the model. See Harness architecture and reproducibility.
• Silent environment drift — documents, schemas, or live dependencies change while the claim keeps its old domain → version the environment inside $D$ and revalidate on change. See Living eval suites.
• An unpinned harness — developers or the agent edit prompts, memory, or skills after the evaluation → name the harness version the claim covers, and treat harness edits as part of the modification surface. See Exposure, modification, and access.

Check yourself#

Analyze. Pick a published agent benchmark result. Write down its population, claim scope, environment, and harness as the paper states them, then list the validity-domain dimensions it leaves undeclared.

Hint

Check environment versions, tool availability, interaction lengths, and the scaffold. Was the harness held fixed across the systems compared, or was it part of what was compared?

Evaluate. DocsAgent passed its release evaluation on one version of the documentation. A month later, many pages have changed. Is the evaluation now wrong?

Hint

Neither. The documentation version is part of the environment and the validity domain, so the evaluation has made no justified claim about the new version. Decide whether to re-run it or re-scope the claim.

Create. Write the validity domain for an agent you operate, dimension by dimension, and the out-of-domain policy for a live request that falls outside it.

Hint

Use the dimensions in the domain table and say which evidence covers each; step 6 of the In practice procedure lists the policy options.

Further reading#

• Harness architecture and reproducibility — the eval harness, isolation, and the per-trial record.
• Living eval suites — keeping the domain current as environments drift.
• Monitoring and control in production — detecting out-of-domain requests at runtime.
• The validation profile and validity statement — the full statement in which the domain is one slot.
• Song (2025), slides 8 and 25–38 — the Berkeley course's scope taxonomy for agent evaluations.