Sources of validity evidence

Validity evidence is any observation that bears on whether a score supports its intended interpretation and use. It comes from several sources — content, response processes, score structure, relations to other measures, deployment criteria, and consequences — and every source feeds the same argument.

Sources, not types#

Validity is one argument supported by several sources of evidence, not a set of separate validities (The validity argument). Passing one source certifies nothing on its own, because each catches a different failure.

Agent-evaluation concepts map onto the closest psychometric evidence as follows. The mapping is many-to-many, not one-to-one:

Face validity and content evidence#

Face validity asks whether an evaluation looks, to informed observers, like it measures the intended construct. It catches absurd mismatches early: a "web navigation" benchmark made entirely of multiple-choice questions has poor face validity. But a benchmark can look realistic and still be solvable through leaked identifiers, an exploitable grader, or a single superficial cue, while a synthetic task can look artificial and isolate a capability well. Treat face validity as a smoke test, never a stopping rule.

Content evidence asks whether the task blueprint adequately spans the construct's facets; it is the bridge from a capability definition to a sampling frame. "Text-to-SQL capability" is too vague to sample, so the blueprint for SQLAgent, the text-to-SQL agent of Case C, names facets — multi-table joins, temporal logic, aggregations with filters, ambiguity resolution, unanswerable requests, permissions, schema discovery, dialect-specific behavior — and justifies their proportions with domain experts and production traces. A text-to-SQL benchmark of simple filters cannot support a claim about enterprise SQL competence. Content evidence guards against construct underrepresentation and construct-irrelevant variance; Constructs, facets, and nuisances treats both.

Response-process and internal-structure evidence#

Response-process evidence asks whether the system succeeded through construct-relevant processes rather than artifacts or shortcuts: not only whether it got the answer, but what causal route from capability to outcome produced the score. If SQLAgent returns correct result sets because it recognizes table names memorized from a public schema, the output is right and the interpretation is wrong; adversarially similar schemas defeat that route. Evidence comes from reading trajectories, from perturbations that remove a suspected shortcut, and from listing what the agent could reach through its tools; Task validity and outcome validity covers the agent-specific checks.

Internal-structure evidence asks whether score dimensionality, item relations, and aggregation behave as the construct theory predicts. One number over heterogeneous skills makes a structural claim: that the components combine that way into a coherent total. If an agent benchmark has slices for planning, tool selection, memory, policy compliance, and recovery, check whether their correlations and model rankings match the conceptual structure; a model can dominate on tool use and fail on policy compliance, and a mean hides it. Factor analysis, item-response models, hierarchical models, or careful slice correlation matrices can show whether a headline score is defensible; for engineering decisions, a vector of scores with explicit gates or a Pareto analysis is often better (Aggregation geometry).

Convergent and discriminant evidence#

Convergent evidence asks whether different methods intended to measure the same construct agree as theory predicts. The key word is independent. Two metrics built from the same gold labels and the same grader share method variance, so their correlation is weak evidence. Stronger examples include a repository-level coding benchmark that correlates with a separately designed, differently scored one, and an LLM judge's quality score that correlates with blinded domain-expert ratings on a held-out sample. For SQLAgent, the independent method is a separately assembled, SME-reviewed sample of analyst workflows.

Convergence should be predicted from a theory of the construct, not demanded indiscriminately. Two benchmarks with good evidence for their own claims may correlate only moderately if they sample different facets.

Discriminant evidence asks whether the score avoids tracking what it should not. If a "reasoning quality" judge strongly tracks answer length or rhetorical confidence, or a "planning" score is almost entirely predicted by reading speed or compliance with a peculiar JSON format, the instrument may be measuring the wrong thing. The test can be simple: measure the candidate nuisance abilities independently, and check whether they explain most of the score's variance. Controlled perturbations, regression residualization, latent-variable models, and multitrait–multimethod designs are more formal options.

The construct's list of nuisance variables, $F^{-}$, is the discriminant test plan: a judge that moves substantially under a perturbation that preserves the construct has exposed method variance. For SQLAgent, the check is that the score is not mostly a proxy for memorized schema names or output formatting. Naming the three strongest alternative constructs that could explain a high score, then designing evidence against each, turns the list into a plan.

Separating trait from method#

A multitrait–multimethod design measures several constructs, or traits, through several methods, so that trait variance can be separated from method variance (Campbell & Fiske, 1959). Measure several candidate capabilities with independent methods — static questions, interactive environments, execution graders, human panels, and learned judges — then ask which correlations follow the capability and which follow the method. A putative "reasoning" factor that appears only under one benchmark format is suspect; it may be benchmark-format familiarity.

Take two capabilities, multi-step reasoning and tool use, each measured with static questions and in an interactive environment: four scores, three kinds of pair. The table gives each kind's correlation in two illustrative scenarios, computed from a factor model in which every standardized score loads on its capability and on its method, and the two capabilities correlate 0.3. When the capability drives the score, the loadings are 0.8 on capability and 0.4 on method; when the format drives it, they are reversed.

In the first scenario, the same capability measured two ways correlates more strongly than any other pair: the pattern follows the capability. In the second, two different capabilities measured the same way correlate more strongly than one capability measured two ways: the pattern follows the format, and a "reasoning" score from static questions mostly measures familiarity with static questions.

The design is a test only if the pattern is stated before the data arrive. Write down the predicted ordering for every kind of pair, and where theory expects two facets to correlate only moderately, say how moderately; then collect the data and compare.

Criterion and consequential evidence#

Criterion evidence asks whether scores relate to meaningful outcomes outside the benchmark: production task success, incident rate, expert acceptance, time saved, defect density, or later replication. Measured at the same time, it is concurrent evidence; used to forecast later behavior, it is predictive evidence. Predictive or ecological evidence asks the deployment question directly: does the score predict the outcome users actually care about? For SQLAgent, the plan relates eval performance to analyst acceptance and correction rate in shadow deployment.

This is where benchmark validity meets deployment. If offline score improvements for DocsAgent, the documentation agent of Case G, do not predict better outcomes on fresh, production-like queries, the benchmark may still measure something coherent, but it cannot justify the deployment interpretation being made from it.

Consequential evidence asks what happens because the score is used. An eval changes the system it measures: once a score becomes a release gate, leaderboard target, RL reward, or training objective, developers and models search for ways to increase it, and the measurement procedure becomes part of the causal environment. Does the benchmark then reward a proxy that can rise while quality falls? Does repeated tuning saturate the set, or evaluator feedback leak into training? Does the judge share blind spots with the model family it grades? The answer is a portfolio of partially independent instruments, holdouts, audits, and production feedback rather than one immortal score; Goodhart pressure and self-confirming loops and Adaptive validity take the questions further.

In practice#

Build an evidence plan for one score interpretation before you collect the data.

1. Write the interpretation. State exactly what you believe the score means and the use it supports.
2. Check face and content. Ask experts whether the eval looks right and record the mismatches; then write the facet blueprint and compare task counts per facet with the target proportions.
3. Trace response processes. Read a sample of trajectories, list the routes to success, and design a perturbation for each suspected shortcut.
4. Check internal structure. Compute slice correlations and per-slice rankings, and decide whether one number should exist.
5. Predict the correlations. List independent methods for the same construct, the nuisance variables and rival constructs, and the multitrait–multimethod pattern you expect, before you see data.
6. Choose a criterion. Name the external outcome the score should predict, how you will observe it, and how strong a relation you expect.
7. Plan for consequences. Write down what changes once the score is a gate or a target, and which evidence must then be gathered again.

The artifact is an evidence plan: one row per source, with the method, the prediction stated in advance, n, and the status, planned or measured. It becomes the natural-validity field of the evaluator dossier.

Common failure modes#

• Face validity as a stopping rule — "it looks like real work" closes the review → treat plausibility as a smoke test, and gather content and response-process evidence.
• Convergence by shared method — two scores built from the same labels or grader agree, and the agreement is reported as convergence → use independent methods, and report what they share. See common-mode assurance.
• Correlations without a theory — every pair is held to one cutoff, or the pattern is read after the fact → predict the convergent and discriminant pattern from the construct before seeing data.
• One number for a multidimensional construct — the mean hides contradictory slices → check internal structure, and report a vector or explicit gates. See one-number reporting.
• Offline gains without transfer — benchmark scores rise while production outcomes do not → collect criterion evidence on fresh, production-like cases, and watch for divergence. See Goodharting.

Check yourself#

Analyze. A "reasoning quality" judge's scores correlate strongly with answer length. List the rival explanations, and the evidence that would tell them apart.

Hint

Length may be construct-relevant, because better answers can contain more steps, or a nuisance. A perturbation that pads or trims an answer without changing its content separates the two: a judge that moves under it has exposed method variance.

Evaluate. A team reports that its LLM judge correlates strongly with its exact-match metric on the same gold answers, and calls this convergent evidence for the judge. Evaluate the claim.

Hint

The two scores share their reference anchor, so part of the correlation is shared method variance. Look for an independent method, such as blinded expert ratings on a held-out sample.

Create. Design a multitrait–multimethod study for three capabilities of interest and three evaluation methods. State the convergent and discriminant pattern your theory predicts before seeing any data, and say which result would make you doubt the scores.

Hint

Nine scores give 36 pairs: 9 share a capability across methods, 9 share a method across capabilities, and 18 share neither. Predict an ordering of the three kinds, and say which capabilities your theory expects to correlate, and how strongly.

Further reading#

• Constructs, facets, and nuisances — the facet blueprint and nuisance list that content and discriminant evidence test.
• Reliability and generalizability — the stability evidence the generalization claim needs.
• Task validity and outcome validity — response-process and scoring evidence for agent benchmarks.
• Goodhart pressure and self-confirming loops — consequential evidence once a score becomes a target.
• Campbell and Fiske (1959), "Convergent and discriminant validation by the multitrait-multimethod matrix," Psychological Bulletin — the original trait-and-method design.
• Cronbach and Meehl (1955), "Construct validity in psychological tests," Psychological Bulletin — construct validity and the nomological network.