The validity argument

A validity argument is the chain of claims that must hold for a score to support an intended interpretation and use: construct, representation, elicitation, scoring, generalization, and decision. Validation gathers evidence for each link. A benchmark has no validity of its own: evidence supports a particular claim, and any link can break.

Validity belongs to an interpretation and a use#

Validation asks whether evidence justifies the intended interpretation and use of a score. Validity is the strength of that justification, not an intrinsic, permanent property of a benchmark. A good evaluator therefore does more than repeat itself: it supports a defensible chain from construct through task, behavior, score, and statistical estimate to decision.

The difference shows in a single sentence. "Model A scored 72 on Benchmark B" is descriptive. A validity claim starts when you infer something from it: that Model A can reliably resolve a target class of customer-service requests under the deployment conditions the benchmark represents, and is suitable for release under a policy threshold. That inference introduces a construct, a population, an interpretation, and a use. Each can fail even when the score was computed perfectly.

SQLAgent, the text-to-SQL agent of Case C, which queries a pinned snapshot of a finance warehouse, makes the same point. An illustrative reading, "SQLAgent passes 72% of the suite (95% Wilson CI 63–80%, n = 100)", is a measurement. "SQLAgent can answer enterprise analyst questions correctly and safely over the target schema" is a claim, and in a release review it carries a decision: ship or hold. The interval makes the reading precise; it says nothing about whether the claim holds.

The framework turns this into a sentence pattern. Never write "the benchmark is valid"; write that the evidence supports interpretation $C$ for population $P$, decision $\delta$, and regime $\mathcal{R}$, under validity domain $D$. Change any of those, and the argument may need to be rebuilt. Outside its declared validity domain, a benchmark has not been shown wrong; it has made no justified claim. Population, environment, harness, and validity domain covers how to declare one.

Six claims from construct to decision#

The argument has six claims:

1. Construct claim. You have defined the capability, behavior, risk, or quality you actually care about.
2. Representation claim. The tasks and environments adequately sample the relevant domain of that construct.
3. Elicitation claim. The setup causes the system to reveal the target capability rather than a shortcut or an unrelated bottleneck.
4. Scoring claim. The evaluator maps observed outcomes or trajectories to the specification with known error.
5. Generalization claim. The estimate transfers across legitimate task, seed, judge, environment, generator, and time variations to the target population.
6. Decision claim. The score and its uncertainty are adequate for the action taken from it.

The claims are chained, so strength in one cannot buy back another. SQLAgent's validation plan supplies evidence for every link; the last column maps each claim to the part of this handbook that tests it.

Task validity and outcome validity are two of the links#

Agentic benchmark work emphasizes two local links: task validity joins the target capability to task success, and outcome validity joins task success to a positive result (Zhu et al., 2025; Song, 2025, slides 40–41). In the six-claim argument, task validity primarily interrogates the representation and elicitation claims, and outcome validity the scoring claim; Zhu et al.'s reporting checklist begins to address generalization and decision. Task validity and outcome validity gives the checks for each.

Zhu et al. state both links as equivalences: the task is solvable if and only if the agent has the capability, and the evaluator returns a pass exactly when the task was completed. As an engineering ideal this is clarifying; as a general theory of measurement it is too strong. A capable agent can fail through stochastic decoding or a transient tool error, a less capable one can succeed by chance, and every grader has an error process, so each link is a conditional probability that good design makes strong, not an identity; Evaluation is measurement writes both out. For SQLAgent, the first link weakens when a capable agent loses its database connection mid-task, and the second when a wrong query happens to return the reference rows on the pinned snapshot. Case C's remedy for the second is metamorphic tests, or several database instances that distinguish accidental equivalence.

One argument, many sources of evidence#

Older teaching presents face, content, criterion, and construct validity as separate types. Contemporary standards are better read as one validity argument supported by different sources of evidence, and the distinction matters for AI evaluation because it prevents checklist thinking: passing one form of evidence does not certify a benchmark. For a benchmark meant to measure autonomous debugging, the claim is not validated by one strong correlation, one plausible task set, or one reliable grader. The case is cumulative.

In this reading, construct validity is not one type among several. It is the overall evidential argument that the measure behaves as expected for the target construct and not for competing explanations. The sources it draws on — face validity, content, response-process, internal-structure, convergent, discriminant, criterion, and consequential evidence — each test a different way a score could mislead. Sources of validity evidence covers them, and Reliability and generalizability covers stability across legitimate replications.

A live theoretical debate asks whether validity should instead be defined causally: a test is valid for an attribute when the attribute exists and causally produces the measurement outcomes (Borsboom et al., 2004). The argument-based view is the operational mainstream and the one this handbook uses; the causal view remains a useful discipline when you choose which evidence to gather. It is close in spirit to the response-process question of which causal route from capability to outcome produced the score.

What the psychometric view changes#

The change is a shift from benchmark-centric to claim-centric thinking, in six moves:

1. A score is an observation, not a capability. The latent construct is inferred through a task and a scoring process; see Evaluation is measurement.
2. Validity is not solved by a better grader. A perfect grader of task success cannot repair a task that measures the wrong capability.
3. Reliability is not solved by deterministic code. The system, tasks, environment, and judge introduce other legitimate replication variance.
4. Convergent and discriminant evidence belong above individual task checks. They test whether the benchmark, as an instrument, sits in the expected network of relationships with other measures.
5. Generalization requires a declared domain. A bare "72%" is incomplete; "72% for this population, environment version, policy, and time window" is a scientific statement.
6. Optimization changes the instrument. Leaderboard and release-gate pressure make validity a lifecycle property rather than a one-time certification.

The sixth move goes past the six claims, which are stated for a target distribution. Adaptive evaluation asks the harder question of what happens when the measured system searches for cases that maximize the instrument, the subject of Validity under pressure.

From argument to statement#

The argument's conclusion travels as a validity statement: an evaluator system supports claim $C$ for consumer $\kappa$ over validity domain $D$, under optimizer and access regime $\mathcal{R}$, up to tested exposure $b^{*}$, with measured validation profile $\Pi$, relative to a reference authority or trust root. The claim and consumer carry the construct and decision claims, the validity domain bounds generalization, and $\Pi$ holds the measured evidence; the regime and $b^{*}$ add what the six claims leave out, the system's exposure to optimization. The validation profile and validity statement explains each slot.

The builder opens in Audit mode on an illustrative statement for a SQL agent, so each slot appears as the question its evidence must answer. Its tested exposure and validation profile are marked planned, so the statement reads "not yet tested" and "not yet measured". Match each question to the claims it tests, then switch to Build and start the DocsAgent statement from the Create exercise below.

In practice#

Write the validity argument for one score you report or rely on.

1. Write the inference. Turn the reading into the sentence you want a reader to believe and the action it should change. Name the construct, the population, and the decision.
2. Expand it into six claims. Write one sentence for each claim, specific to your eval, and list every variation the generalization claim must survive: tasks, seeds, judges, environments, generators, and time.
3. Name the strongest threat to each claim. A missing facet, a shortcut, a grader that passes wrong answers, rankings that move with the environment snapshot, an interval that straddles the threshold.
4. Attach evidence. For each claim, record the evidence you have, with n and an interval, or mark it "planned".
5. Declare the domain and the regime. Say where the claim stops, and who optimizes against the score with what access.
6. Draft the validity statement. Fill every slot, and say which parts are measured and which are planned. Every claim without evidence is a work item, and so is every change of regime.

The artifact is a one-page validity argument: six claims, each with its strongest threat, its evidence, and its status, closed by a draft validity statement. It feeds the natural-validity and validity-domain fields of the evaluator dossier (see Templates).

Common failure modes#

• Validity as a sticker — a report says "the benchmark is valid" and stops → rewrite it as the claim the evidence supports, for a population, decision, regime, and validity domain. See the validity domain.
• Two links taken for the argument — task and outcome checks pass, so the benchmark is declared sound → a benchmark can have strong task and outcome validity while lacking representative sampling, transport evidence, or decision relevance; write the other claims. See task validity and outcome validity.
• Agreement offered as validity — high inter-judge agreement or a deterministic grader is cited as proof → reliability is necessary, not sufficient; ask what the score is stable about. See reliability.
• Validity frozen at launch — the evidence predates the score becoming a leaderboard target or a release gate → treat validity as a lifecycle property and gather evidence again when the regime changes. See Goodharting.

Check yourself#

Analyze. Take the inference "Model A scored 72 on Benchmark B, so it can be released to handle customer-service requests." Split it into the six claims, and mark which ones the benchmark's documentation supplies evidence for.

Hint

The score supplies none of the claims by itself. The construct, the population of deployment requests, and the release decision all enter with the inference. Look for representation and generalization evidence first, then for anything that relates the score to real customer outcomes.

Evaluate. A benchmark paper reports that every task passed a shortcut review and every grader passed the outcome-validity checks, and concludes that the benchmark measures agent capability. What does the evidence support, and what is missing?

Hint

It addresses the elicitation and scoring claims. The construct and representation claims, generalization across seeds, judges, environments, generators, and time, and the decision claim are still open.

Create. Write the six-claim argument for the claim of DocsAgent, the documentation agent of Case G — procedurally correct, grounded answers to help requests within the supported product corpus — with one piece of planned evidence per claim. Then compress it into a validity statement in the builder.

Hint

Case G's plan includes tasks built from production query clusters, learned judges validated against double-labeled SME examples, variance from query, model, judge, and document version, and periodic comparison with fresh user-resolution outcomes. Mark both $b^{*}$ and $\Pi$ as planned.

Further reading#

• Task validity and outcome validity — the two agent-era links, with checks by output type.
• Sources of validity evidence — what each source of evidence tests, and the multitrait–multimethod design.
• The validation profile and validity statement — every slot of the statement, and why statements expire.
• Constructs, facets, and nuisances — the first claim in depth.
• Zhu et al. (2025), "Establishing best practices for building rigorous agentic benchmarks" — task validity and outcome validity as named links.
• Messick (1995), "Validity of psychological assessment," American Psychologist — validation as scientific inquiry into score meaning.
• Borsboom, Mellenbergh, and van Heerden (2004), "The concept of validity," Psychological Review — the causal alternative to the argument-based view.