# Population, environment, harness, and validity domain

> Every score was earned in some world. Declare the population, environment and harness, and say where the claim stops.

The **validity domain** is the declared set of conditions — model families, languages, task classes, environment versions, tool availability, prompt families, user populations, and risk conditions — over which an evaluation's interpretation is claimed to hold. Together with the population, environment, and harness, it says which world a score was earned in and where the claim stops.

> [!KEY] In one sentence
> Every score was earned in some world: declare the population, environment, and harness behind it, and state the validity domain outside which the evaluation has made no justified claim.

## Population and claim scope

Four coordinates of the [claim](https://benchmaker.ai/glossary/claim-tuple) $\mathcal{C} = (C, P, E, H, D, \delta)$ describe the world an evaluation ran in and the world it speaks for: the population $P$, the environment $E$, the harness $H$, and the validity domain $D$. They matter because success depends on more than capability: task success is $\Pr[T = 1 \mid C, E, H]$, conditional on the environment and the harness too (see [Evaluation is measurement](https://benchmaker.ai/handbook/foundations/evaluation-is-measurement#probabilistic-links-not-equivalences)). For SQLAgent, the text-to-SQL agent of [Case C](https://benchmaker.ai/handbook/cases/sql-data-agent), they are analysts' questions, a pinned enterprise database snapshot, and a harness of a policy prompt, a schema-inspection tool, and a read-only query executor, bounded by a declared domain.

The **population** $P$ is the task, user, or input distribution the claim is about. How a finite eval set models it is the subject of [Task distributions and dataset engineering](https://benchmaker.ai/handbook/claim/task-distributions); the question here is how broad a population the claim takes on. A benchmark's **claim scope**, a distinction drawn in the Berkeley agent-evaluation course, is whether it evaluates a specific capability, a specific application, or a general set of applications. The examples below come from the course's taxonomy ([Song, 2025](https://benchmaker.ai/handbook/reference/sources#song-2025), slides 25–38):

| Claim scope | What a score can support | Examples in the course's taxonomy |
|---|---|---|
| A specific capability | An inference about one facet of a construct, such as planning or tool selection in isolation | Planning and multi-step reasoning suites such as PlanBench |
| A specific application | An inference about a deployment domain | Web agents ([WebArena](https://benchmaker.ai/benchmarks/webarena)), cybersecurity (CyberGym), legal workflows (LegalAgentBench) |
| A general set of applications | The broadest claim, and the most fragile: representation must be defended across many domains at once | Generalist suites such as [GAIA](https://benchmaker.ai/benchmarks/gaia) and OSWorld |

State the scope explicitly, because the same score with the same tasks licenses different sentences under each reading. A suite that isolates plan generation supports a claim about one facet; it does not show that an agent plans well inside a customer-service workflow, which is an application claim with its own population.

## Environment

The **environment** $E$ is the external state and transition process the system interacts with, and it may be simulated, sandboxed, pinned, or live. It includes world state, tool backends, other agents or simulated users, stochastic disturbances, and transition dynamics. The line between environment and [system under test](https://benchmaker.ai/glossary/system-under-test) is control: the system is what you ship and can change, from the model and prompts to memory, tools as exposed through the scaffold, and orchestration logic; the environment is what it acts on. Unlike a static text-to-text model, an agent acts in an environment whose state changes as it acts (Song, 2025, slide 8).

DocsAgent, the documentation agent of [Case G](https://benchmaker.ai/handbook/cases/docs-agent), shows why the environment belongs in the claim. Its world includes document versions and a retrieval index. Its [criterion card](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards) for grounded answers lists a blind spot: the documentation itself may be stale, and that belongs to the environment and validity domain rather than the judge. When the documents change, the world the score was earned in has changed, even if the model has not. An evaluation that faithfully measures performance in a world other than the deployment world is wrong in the way this handbook calls "wrong world".

> [!CONVENTION]
> This handbook does not fold the task distribution into the environment. The claim tuple keeps them apart, with $P$ for which situations the claim covers and $E$ for the world they play out in, so a claim can state a change in one without the other: new users on the same documentation, or the same users on a new documentation version.

## Harness

The **harness** is everything around the model that turns it into an [agent](https://benchmaker.ai/glossary/agent): system instructions, tool definitions, memory stores, skill libraries, retrieval indices, orchestration logic, and stopping rules. In the claim, $H$ is the administration context, the harness under which the capability is claimed. It matters because agent capability is observed only through an interface: token limits, tool schemas, retry policies, filesystem permissions, browser state, simulator behavior, timeouts, and error handling can all change the measured result. So a benchmark is a protocol to be run, not only a set of tasks and a metric.

Keep two harnesses apart. The agent harness is part of the system under test; the [eval harness](https://benchmaker.ai/glossary/eval-harness), which provisions environments, runs and records trials, and grades them, is part of the instrument. Collapsing them invites a "model regression" that is actually the scaffold's timeout bug; [Harness architecture and reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility) covers the eval harness.

A claim about an agent is therefore a claim about a model in a harness. The [coding agent in Case B](https://benchmaker.ai/handbook/cases/coding-agent) is evaluated in a network-off container with file, shell, and test tools and 30 minutes per issue; the same model with a different budget or tool set is a different system under test.

Harnesses can also change after the evaluation: they can be inspected and edited from outside, including by the agent itself, which makes harness self-modification a concrete form of self-modification even when model weights stay frozen ([Chen, Wang & Qu, 2026](https://benchmaker.ai/handbook/reference/sources#chen-wang-qu-2026), §2.1). If developers or the agent can change the harness between evaluation and use, the claim must name the harness version it covers, and the harness belongs to the regime's modification surface ([Exposure, modification, and access](https://benchmaker.ai/handbook/regime/exposure-and-surfaces)).

### Capability–harness compatibility

Before interpreting a low score, check that the harness makes the intended capability expressible. Does the model receive the tool descriptions it needs, and can it discover resources dynamically if deployment requires that skill? Is the context budget artificially below what the task requires? Do rate limits or browser quirks introduce failures that are not part of the construct? If SQLAgent's harness truncated schema listings below what multi-table questions need, a low score on joins would measure the context budget, not join reasoning (illustrative). This is a discriminant-validity issue as much as an engineering one. And as [Anthropic (2026)](https://benchmaker.ai/handbook/reference/sources#anthropic-2026) warns, a strong model scoring near 0% on a task usually points to a task or harness bug, not a capability fact.

## The validity domain

Every evaluator should declare the model families, languages, task classes, environment versions, tool availability, prompt families, user populations, and risk conditions where the validation evidence is claimed to transport. Products, interaction lengths, permission levels, tool behavior, tool contracts, and the retrieval corpus can belong on the list too. The [crosswalk](https://benchmaker.ai/handbook/reference/crosswalk) maps the older term operational design domain onto the same object.

The population and the domain answer different questions. $P$ names the situations the estimate is about; $D$ names the conditions under which the evidence is claimed to carry over. $D$ reaches across every other coordinate, the evaluator included: a judge whose error rates were measured on one model family's outputs has not been validated on another's ([The judge as a noisy sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor)). For SQLAgent's release evaluation, a declaration might read as follows (illustrative, matching the Studio's Case C):

| Dimension | What the evidence covers |
|---|---|
| Task classes | Analytics questions over the finance warehouse |
| Languages | English |
| Environment versions | Schema v12, on a pinned snapshot |
| Tool availability | Schema inspection and read-only query execution |
| Prompt families | The release prompt and its tool schemas |
| User populations | Finance analysts at analyst permission level |
| Model families | Outputs of the agent's current model family, on which the judge was calibrated |
| Risk conditions | Read-only credentials and row-level permissions in force |

Outside the validity domain, the benchmark has not been shown wrong; it has simply made no justified claim. A SQLAgent pass rate on schema v12 says nothing, good or bad, about schema v13 until evidence covers it.

Declaring a dimension is not the same as showing that the measurement holds across it. A domain that spans English and Spanish, or two user groups, implicitly claims [measurement invariance](https://benchmaker.ai/glossary/measurement-invariance): that the measurement relation stays stable across groups, model families, languages, or conditions.

> [!OPEN]
> Procedures for testing measurement invariance across languages and user groups are a known gap in this handbook. List each language or group in $D$ only as far as evidence covers it, and say which entries rest on measured transport and which on assumption.

## Validity is a claim, not a sticker

The framework turns the domain into a sentence pattern. The statement "Benchmark B is valid" is incomplete. A defensible statement says that the evidence supports interpretation $C$ for population $P$, decision $\delta$, and regime $\mathcal{R}$ under validity domain $D$; change any of them, and the validation argument may need to be rebuilt. In one line: "72%" is incomplete; "72% for this population, environment version, policy, and time window" is a scientific statement.

For SQLAgent, "the SQL benchmark is valid" becomes (illustrative): the execution evidence supports reading the certification pass rate as correct, permission-respecting answers to English analytics questions, for the rollout decision, under developer iteration that never sees the certification set, within schema v12 on the pinned snapshot. The remaining slots of a full [validity statement](https://benchmaker.ai/glossary/validity-statement) — consumer, tested exposure, validation profile, and trust root — are developed in [The validation profile and validity statement](https://benchmaker.ai/handbook/pressure/validation-profile).

The builder below loads an illustrative SQL-agent statement whose domain reads "English analytics questions over the finance warehouse, schema v12, on the pinned Q3 snapshot"; widen that field to every warehouse, and ask which other fields would then need new evidence.

```widget
validity-statement-builder
preset: sql-agent
```

Declaring $D$ offline is half the job. At runtime, a detector decides whether each live request falls inside $D$, under an explicit out-of-domain policy and with its own measured error rates; that belongs to [Monitoring and control in production](https://benchmaker.ai/handbook/regime/production-monitoring). Because documents, schemas, and tools change, the domain also needs a date and a revalidation trigger ([Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites)).

## In practice

Write a validity-domain declaration for one evaluation you run.

1. **State the population and scope.** Name the task, user, or input distribution, and say whether the claim is about a capability, an application, or a general set of applications.
2. **Describe the environment.** List the state, tool backends, other actors, and disturbances, each marked simulated, sandboxed, pinned, or live, with its version.
3. **Pin the harness.** Record the agent harness the claim covers, apart from the eval harness that ran the trials, and note who can edit each, the agent included.
4. **Check capability–harness compatibility.** Confirm the harness makes the capability expressible before you read a low score.
5. **Declare the domain.** For each dimension in the domain table above, write what the evidence covers.
6. **Write the out-of-domain policy.** Decide whether an out-of-domain request is refused, escalated, run with reduced permissions, confirmed with the user, or routed to a safer process, and hand runtime detection to production monitoring.
7. **Write the sentence.** State what the evidence supports, for which population, decision, and regime, under which domain.

The artifact is a validity-domain declaration: the population and scope, the environment and harness versions, the domain table with the evidence behind each entry, and the out-of-domain policy. It fills the claim and validity-domain fields of the evaluator dossier and the domain slot of the validity statement.

## Common failure modes

- **Headline extrapolation** — a score earned on one schema, language, or tool version is quoted for all of them → declare $D$, and treat use outside it as unsupported. See [one-number reporting](https://benchmaker.ai/handbook/practice/anti-patterns#one-number-reporting).
- **A harness bug read as a model regression** — a scaffold timeout or a missing tool description shows up as lost capability → keep the agent and eval harnesses apart, and check compatibility before blaming the model. See [Harness architecture and reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility).
- **Silent environment drift** — documents, schemas, or live dependencies change while the claim keeps its old domain → version the environment inside $D$ and revalidate on change. See [Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites).
- **An unpinned harness** — developers or the agent edit prompts, memory, or skills after the evaluation → name the harness version the claim covers, and treat harness edits as part of the modification surface. See [Exposure, modification, and access](https://benchmaker.ai/handbook/regime/exposure-and-surfaces).

## Check yourself

**Analyze.** Pick a published agent benchmark result. Write down its population, claim scope, environment, and harness as the paper states them, then list the validity-domain dimensions it leaves undeclared.

> [!DETAILS] Hint
> Check environment versions, tool availability, interaction lengths, and the scaffold. Was the harness held fixed across the systems compared, or was it part of what was compared?

**Evaluate.** DocsAgent passed its release evaluation on one version of the documentation. A month later, many pages have changed. Is the evaluation now wrong?

> [!DETAILS] Hint
> Neither. The documentation version is part of the environment and the validity domain, so the evaluation has made no justified claim about the new version. Decide whether to re-run it or re-scope the claim.

**Create.** Write the validity domain for an agent you operate, dimension by dimension, and the out-of-domain policy for a live request that falls outside it.

> [!DETAILS] Hint
> Use the dimensions in the domain table and say which evidence covers each; step 6 of the In practice procedure lists the policy options.

## Further reading

- [Harness architecture and reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility) — the eval harness, isolation, and the per-trial record.
- [Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites) — keeping the domain current as environments drift.
- [Monitoring and control in production](https://benchmaker.ai/handbook/regime/production-monitoring) — detecting out-of-domain requests at runtime.
- [The validation profile and validity statement](https://benchmaker.ai/handbook/pressure/validation-profile) — the full statement in which the domain is one slot.
- Song (2025), slides 8 and 25–38 — the Berkeley course's scope taxonomy for agent evaluations.

---
Source: https://benchmaker.ai/handbook/claim/validity-domain