Reliability and generalizability

Reliability is the stability of a measurement across legitimate replications. It is necessary for validity and never sufficient: a perfectly repeatable evaluator can reliably measure the wrong construct. Measuring it starts with a declaration of what may legitimately vary, because the claim, not the data, decides which variation is error.

Reliability is not validity#

Reliability asks whether measurements are stable across legitimate replications; validity asks whether they support the interpretation you draw. The two come apart in both directions. A deterministic string matcher is perfectly repeatable and can still be an invalid measure of semantic correctness. An evaluator can be unbiased on average and still noisy enough that model rankings flip across runs, failing the practical requirement of a stable decision instrument.

Measurement theory states the asymmetry plainly: reliability is a necessary condition for validity, never a sufficient one, and optimizing for agreement alone can reduce validity when a measurement scheme is simplified until annotators or scores agree (Krippendorff, 2004; Reidsma & Carletta, 2008). Riezler and Hagmann (2024) carry the separation into machine learning: agreement coefficients are descriptive statistics about consistency, validity is a distinct question, and a model can be reliably predictive of an invalid construct.

Four words are easy to collapse, and none is a synonym for another:

A confidence interval measures precision. It is not evidence of validity, or of reliability in the broad sense.

Specify the replication universe#

Reliability is meaningless until you say what is allowed to vary; psychometric standards require identifying the domain of legitimate replications (AERA, APA & NCME, 2014). For an agent evaluation, the replication universe can include task instances or alternate task forms, rollout seeds, the judge model and judge seed, human raters, tool latency and transient errors, simulated-user realizations, environment snapshots, retrieval index builds, harness versions, generator families, and time.

The intended claim decides which of these count as measurement error and which are part of what is being measured. If the product will face random tool failures in deployment, tool-failure variation may belong to the target environment rather than being noise to eliminate; if the benchmark compares model policies independent of infrastructure, the same variation may need to be controlled. One way to make the decision explicit is to classify every variable that can change without a model change as a controlled condition, legitimate target-environment variation, or measurement error.

For the claim of DocsAgent, the documentation agent of Case G — procedurally correct, grounded answers to help requests within the supported product corpus — one illustrative classification looks like this:

Four reliability studies#

Four practical studies probe the main facets:

The validation plans of Cases C and G use three of them. SQLAgent, the text-to-SQL agent of Case C, runs multiple rollouts for ambiguous and multi-step tasks, with tasks treated as clusters, and checks whether rankings hold across alternate database snapshots that preserve schema semantics; DocsAgent validates its learned judges against double-labeled SME examples.

Inter-rater agreement answers one narrow question: do qualified graders assign similar scores? It is not the reliability of the whole eval. A benchmark can have perfect inter-rater agreement and unstable model scores, because tasks are heterogeneous or agent rollouts are flaky. Where a chance-corrected coefficient is wanted, Cohen's κ is standard but depends on prevalence, so report the confusion matrix and TPR and TNR with it, and state the prevalence; The judge as a noisy sensor shows the size of the effect.

The environment-version study guards against a quieter failure. If rankings swing because a website changed, the benchmark is measuring infrastructure drift. Snapshots and recorded fixtures make the study possible; Harness architecture and reproducibility covers them.

Variance components and generalizability#

Generalizability theory replaces a single agreement coefficient with a decomposition of variance (Brennan, 2001). Write the score of one trial as

$$\begin{aligned} Y_{mtrje} = {} & \mu + \alpha_m + u_t + v_r + w_j + q_e \\ & + \text{interactions} + \varepsilon \end{aligned}$$

where:

• $Y_{mtrje}$ is the score of system $m$ on task $t$, in rollout $r$, graded by judge $j$, in environment snapshot $e$;
• $\mu$ is the grand mean, and $\alpha_m$ is the system's own effect, the object of measurement;
• $u_t$, $v_r$, $w_j$, and $q_e$ are the task, rollout, judge, and environment-snapshot effects;
• "interactions" collects effects such as system by task, and $\varepsilon$ is the residual.

In words: every trial's score is the system's effect plus a contribution from each facet that varied, and a mixed-effects model can estimate how much of the observed variability each facet contributes. Riezler and Hagmann apply this analysis to machine-learning evaluation, fitting linear mixed-effects models and computing reliability as intraclass correlations, with meta-parameter settings, random seeds, and test items as competing sources of variance.

Generalizability theory's absolute coefficient condenses the idea into one ratio:

$$\Phi = \frac{\sigma^2_{\text{object}}}{\sigma^2_{\text{object}} + \sigma^2_{\Delta}}$$

where:

• $\sigma^2_{\text{object}}$ is the variance attributable to the object of measurement, such as the systems being compared;
• $\sigma^2_{\Delta}$ collects the variance from sources that should not change the interpretation, which shrinks as the design averages over more conditions of each source.

In words: Φ is the share of the variance in reported scores that comes from real differences between systems rather than from the conditions they happened to be measured under. The formula matters less than the design question it forces: which sources of variation are legitimate, and how many replications of each facet make the decision stable?

Publish a variance budget#

Instead of publishing one standard error, this handbook proposes, as a research direction, that benchmark developers publish a variance budget: how much score or ranking variance comes from tasks, model rollouts, judge choice, environment snapshots, tool outages, and their interactions. The question it answers is economic: where does the next evaluation dollar buy the largest reduction in uncertainty — more tasks, more rollouts, better adjudication, or a more stable environment?

For a pass rate, the budget follows from how often the design samples each source. Suppose a design runs $n_q$ queries on each of $n_d$ documentation versions, with $n_r$ rollouts per query and version and $n_j$ judge calls per rollout. Ignoring any query-by-version interaction, the variance of the reported pass rate is

$$\begin{aligned} \operatorname{Var}(\hat p) = {} & \frac{\sigma^2_q}{n_q} + \frac{\sigma^2_d}{n_d} \\ & + \frac{\sigma^2_r}{n_q n_d n_r} + \frac{\sigma^2_j}{n_q n_d n_r n_j} \end{aligned}$$

where:

• $\hat p$ is the reported pass rate;
• $\sigma^2_q$, $\sigma^2_d$, $\sigma^2_r$, and $\sigma^2_j$ are the variances contributed by one query, one documentation version, one rollout, and one judge call;
• $n_q$, $n_d$, $n_r$, and $n_j$ are how many of each the design samples.

In words: each source's variance is divided by the number of independent draws of that source the design averages over, so a source you sample once never shrinks.

An illustrative DocsAgent budget shows what that means. Suppose DocsAgent passes 70% of 200 queries on the current documentation, with one rollout and one judge call each: 70% (95% Wilson CI 63–76%, n = 200). Suppose also that the query, rollout, and judge-call variances are 0.12, 0.06, and 0.04, together close to the 0.21 that a pass-or-fail score near 70% must have, and that documentation versions shift the whole suite's pass rate with a standard deviation of 4 points, so $\sigma^2_d = 0.0016$. All of these values are illustrative.

The contributions sum to $\operatorname{Var}(\hat p) = 0.0027$, a 95% half-width of ±10.2 points. The version term is three-fifths of the budget, and no number of queries can shrink it: with every query on one version, even unlimited queries leave a 95% half-width of ±7.8 points. The next 200 trials buy different amounts depending on where they go:

The second version buys the most, which is the variance budget's point: spend the next dollar on the dominant component, and increase replication where the decision is unstable rather than everywhere uniformly. The last row is a different kind of move. Pinning the documentation takes the version term out of $\sigma^2_{\Delta}$ by narrowing the claim to one release, and the validity domain shrinks to match.

The last row also shows why a narrow interval is not a reliable instrument. Its ±6.5 points is close to the Wilson interval on the original run, because the binomial interval quietly assumes the same thing: that the 200 trials are the only randomness. With enough tasks, a benchmark can estimate its own biased scoring rule very precisely, and a high reliability coefficient does not show that a two-point difference between models is statistically or practically meaningful. Keep three questions apart for every reported claim: is the measurement stable across legitimate replications, is the estimate precise enough for the comparison or threshold, and is the interpretation valid for the intended use? Estimands, pass rates, and intervals and Comparing systems take up the second question.

In practice#

Run a generalizability study for one reported score.

1. List what can vary. Write down every variable that can change without a model change: tasks, rollouts, judge calls, judge versions, raters, environment snapshots, index builds, tool errors, harness versions, and time.
2. Classify each against the claim. Mark it a controlled condition (pin and record it), legitimate variation (sample it), or measurement error (estimate and reduce it).
3. Sample every legitimate source more than once. One version, one judge call, or one rater gives no estimate of that source's variance.
4. Run the studies that apply. Repeated rollouts, alternate forms, inter-scorer overlap reported as confusion matrices, and equivalent environment snapshots.
5. Estimate the components. Fit a mixed-effects model with tasks as clusters, and keep the per-trial records so that the analysis can be rerun.
6. Build the variance budget. For the reported score and the decision it serves, compute each source's contribution and the interval it implies, and set it beside the naive binomial interval.
7. Spend or narrow. Put the next run where the dominant component is, or narrow the claim and record the narrower validity domain.

The artifact is a variance budget: the replication universe with each variable's classification, the estimated components, their contributions to the reported score's uncertainty, and the design change they justify. It fills the reliability field of the evaluator dossier — replication facets, variance components, inter-rater agreement, and repeatability.

Common failure modes#

• Agreement reported as reliability — one inter-rater coefficient stands in for the reliability of the whole eval → measure the facets that move the decision: tasks, rollouts, judges, and environments. See one-number reporting.
• An unstated replication universe — tool errors are noise in one report and target behavior in the next → classify every variable against the claim before measuring.
• Precision mistaken for reliability — many tasks on one snapshot with one judge produce a narrow interval → budget every legitimate source, and report the interval it implies.
• Correlated trials — shared state or a flaky dependency makes trial outcomes co-move → isolate trials and resample at the task level. See correlated trials.

Check yourself#

Analyze. For an interactive agent benchmark you know, list every variable that can change without a model change, and classify each as a controlled condition, legitimate target-environment variation, or measurement error.

Hint

Start from the facets of the replication universe above — task form, rollout, judge, rater, tool errors, simulated user, environment snapshot, index build, harness version, and time — and let the claim decide each classification.

Evaluate. A benchmark report cites high inter-rater agreement and thousands of tasks as evidence that its leaderboard is reliable. Evaluate the claim.

Hint

Agreement covers one facet, and the task count buys precision. Rollouts, environment versions, and the choice of judge are unmeasured, and any of them can reorder the leaderboard.

Create. Construct two evals with identical reliability but radically different validity. Then construct two with similar validity targets but radically different reliability. For each pair, name the engineering intervention that fixes the weaker one.

Hint

For the first pair, grade the same SQL tasks by exact string match and by execution with result-set comparison: both are deterministic, and only one tracks semantic correctness. For the second, apply the same rubric with one judge call or with several calls and adjudication.

Further reading#

• The judge as a noisy sensor — inter-scorer evidence as error rates, and why Cohen's κ depends on prevalence.
• Estimands, pass rates, and intervals — the precision question: Wilson intervals and clustered trials.
• Harness architecture and reproducibility — isolation, pinned environments, and the per-trial record a variance study needs.
• Sources of validity evidence — what a stable score still has to show.
• Brennan (2001), Generalizability Theory — variance components and the absolute coefficient Φ.
• Riezler and Hagmann (2024), Validity, Reliability, and Significance: Empirical Methods for NLP and Data Science — mixed-effects reliability analysis for machine-learning evaluation.