# Reliability and generalizability

> Specify what may legitimately vary, then measure how much tasks, seeds, judges and environments move the score.

**Reliability** is the stability of a measurement across legitimate replications. It is necessary for validity and never sufficient: a perfectly repeatable evaluator can reliably measure the wrong construct. Measuring it starts with a declaration of what may legitimately vary, because the claim, not the data, decides which variation is error.

> [!KEY] In one sentence
> Declare what may legitimately vary, measure how much tasks, seeds, judges, and environments each move the score, and spend the next evaluation dollar on the component that dominates.

## Reliability is not validity

Reliability asks whether measurements are stable across legitimate replications; validity asks whether they support the interpretation you draw. The two come apart in both directions. A deterministic string matcher is perfectly repeatable and can still be an invalid measure of semantic correctness. An evaluator can be unbiased on average and still noisy enough that model rankings flip across runs, failing the practical requirement of a stable decision instrument.

Measurement theory states the asymmetry plainly: reliability is a necessary condition for validity, never a sufficient one, and optimizing for agreement alone can reduce validity when a measurement scheme is simplified until annotators or scores agree ([Krippendorff, 2004](https://benchmaker.ai/handbook/reference/sources#krippendorff-2004); [Reidsma & Carletta, 2008](https://benchmaker.ai/handbook/reference/sources#reidsma-carletta-2008)). [Riezler and Hagmann (2024)](https://benchmaker.ai/handbook/reference/sources#riezler-hagmann-2024) carry the separation into machine learning: agreement coefficients are descriptive statistics about consistency, validity is a distinct question, and a model can be reliably predictive of an invalid construct.

Four words are easy to collapse, and none is a synonym for another:

| Word | The question it answers |
|---|---|
| Validity | Does the score support the intended interpretation and use? |
| Reliability | Are scores stable across legitimate replications? |
| Precision | How uncertain is this particular estimate? |
| [Calibration](https://benchmaker.ai/glossary/calibration) | Do probabilistic scores correspond to observed frequencies? |

A confidence interval measures precision. It is not evidence of validity, or of reliability in the broad sense.

## Specify the replication universe

Reliability is meaningless until you say what is allowed to vary; psychometric standards require identifying the domain of legitimate replications ([AERA, APA & NCME, 2014](https://benchmaker.ai/handbook/reference/sources#aera-2014)). For an agent evaluation, the [replication universe](https://benchmaker.ai/glossary/replication-universe) can include task instances or alternate task forms, rollout seeds, the judge model and judge seed, human raters, tool latency and transient errors, simulated-user realizations, environment snapshots, retrieval index builds, harness versions, generator families, and time.

The intended claim decides which of these count as measurement error and which are part of what is being measured. If the product will face random tool failures in deployment, tool-failure variation may belong to the target environment rather than being noise to eliminate; if the benchmark compares model policies independent of infrastructure, the same variation may need to be controlled. One way to make the decision explicit is to classify every variable that can change without a model change as a controlled condition, legitimate target-environment variation, or measurement error.

For the claim of DocsAgent, the documentation agent of [Case G](https://benchmaker.ai/handbook/cases/docs-agent) — procedurally correct, grounded answers to help requests within the supported product corpus — one illustrative classification looks like this:

| Variable | Classification | What to do |
|---|---|---|
| Which help request is asked | Legitimate variation: the population the claim covers | Sample it from production query clusters |
| Rollout seed | Legitimate variation: deployment is stochastic too | Repeat rollouts where the decision is unstable |
| Judge call on the same answer | Measurement error | Estimate it; reduce it with repeated calls or adjudication |
| Judge model, prompt, and settings | Controlled condition | Freeze and version them; recalibrate on change |
| Documentation version | Legitimate variation if the claim covers the corpus as it changes; controlled if it covers one release | Sample versions, or pin one and narrow the validity domain |
| Retrieval index build | Controlled condition | Pin and record it |
| Transient tool errors | Legitimate variation if deployment sees them; controlled for a model comparison | Decide, and record the decision |

> [!SOURCE]
> The Berkeley lecture notes that human evaluations are "regarded as gold standards", then lists their issues: they are slow, expensive, and not reproducible, with disagreement between annotators and within one annotator over time ([Song, 2025](https://benchmaker.ai/handbook/reference/sources#song-2025), slide 20). This handbook does not adopt the label. A [human panel](https://benchmaker.ai/glossary/human-rater) is a measurement mechanism with ordinary error and, separately, the trust root where the evaluate-the-evaluator chain stops; raters belong to the replication universe, and their disagreement is measured like any other facet.

## Four reliability studies

Four practical studies probe the main facets:

| Study | What varies | What to report |
|---|---|---|
| Test-retest, or repeated rollouts | Rollouts of the same system on the same tasks, under legitimate stochasticity | Score variance and decision consistency |
| Alternate form | Task forms sampled or generated independently from the same blueprint | Whether rankings stay reasonably stable, if the claim is about a broad construct |
| Inter-scorer | Human raters, LLM judges, or formal and effect-based graders, where they overlap | Confusion matrices or agreement, not one correlation |
| Environment version | Environment snapshots that should be equivalent | Whether rankings swing |

The validation plans of Cases C and G use three of them. SQLAgent, the text-to-SQL agent of [Case C](https://benchmaker.ai/handbook/cases/sql-data-agent), runs multiple rollouts for ambiguous and multi-step tasks, with tasks treated as clusters, and checks whether rankings hold across alternate database snapshots that preserve schema semantics; DocsAgent validates its learned judges against double-labeled SME examples.

Inter-rater agreement answers one narrow question: do qualified graders assign similar scores? It is not the reliability of the whole eval. A benchmark can have perfect inter-rater agreement and unstable model scores, because tasks are heterogeneous or agent rollouts are flaky. Where a chance-corrected coefficient is wanted, [Cohen's κ](https://benchmaker.ai/glossary/cohens-kappa) is standard but depends on prevalence, so report the confusion matrix and TPR and TNR with it, and state the prevalence; [The judge as a noisy sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor) shows the size of the effect.

The environment-version study guards against a quieter failure. If rankings swing because a website changed, the benchmark is measuring infrastructure drift. Snapshots and recorded fixtures make the study possible; [Harness architecture and reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility) covers them.

> [!OPEN]
> Inter-scorer studies with human raters depend on annotation operations — recruiting, training, and monitoring raters, and running adjudication at scale — that this handbook does not cover. Characterize the human mechanism's error as you would a judge's, and report it next to the score.

## Variance components and generalizability

[Generalizability theory](https://benchmaker.ai/glossary/generalizability-theory) replaces a single agreement coefficient with a decomposition of variance ([Brennan, 2001](https://benchmaker.ai/handbook/reference/sources#brennan-2001)). Write the score of one trial as

$$
\begin{aligned}
Y_{mtrje} = {} & \mu + \alpha_m + u_t + v_r + w_j + q_e \\
& + \text{interactions} + \varepsilon
\end{aligned}
$$

where:

- $Y_{mtrje}$ is the score of system $m$ on task $t$, in rollout $r$, graded by judge $j$, in environment snapshot $e$;
- $\mu$ is the grand mean, and $\alpha_m$ is the system's own effect, the object of measurement;
- $u_t$, $v_r$, $w_j$, and $q_e$ are the task, rollout, judge, and environment-snapshot effects;
- "interactions" collects effects such as system by task, and $\varepsilon$ is the residual.

In words: every trial's score is the system's effect plus a contribution from each facet that varied, and a mixed-effects model can estimate how much of the observed variability each facet contributes. Riezler and Hagmann apply this analysis to machine-learning evaluation, fitting linear mixed-effects models and computing reliability as intraclass correlations, with meta-parameter settings, random seeds, and test items as competing sources of variance.

Generalizability theory's [absolute coefficient](https://benchmaker.ai/glossary/generalizability-coefficient) condenses the idea into one ratio:

$$
\Phi = \frac{\sigma^2_{\text{object}}}{\sigma^2_{\text{object}} + \sigma^2_{\Delta}}
$$

where:

- $\sigma^2_{\text{object}}$ is the variance attributable to the object of measurement, such as the systems being compared;
- $\sigma^2_{\Delta}$ collects the variance from sources that should not change the interpretation, which shrinks as the design averages over more conditions of each source.

In words: Φ is the share of the variance in reported scores that comes from real differences between systems rather than from the conditions they happened to be measured under. The formula matters less than the design question it forces: which sources of variation are legitimate, and how many replications of each facet make the decision stable?

## Publish a variance budget

Instead of publishing one standard error, this handbook proposes, as a research direction, that benchmark developers publish a [variance budget](https://benchmaker.ai/glossary/variance-components): how much score or ranking variance comes from tasks, model rollouts, judge choice, environment snapshots, tool outages, and their interactions. The question it answers is economic: where does the next evaluation dollar buy the largest reduction in uncertainty — more tasks, more rollouts, better adjudication, or a more stable environment?

For a pass rate, the budget follows from how often the design samples each source. Suppose a design runs $n_q$ queries on each of $n_d$ documentation versions, with $n_r$ rollouts per query and version and $n_j$ judge calls per rollout. Ignoring any query-by-version interaction, the variance of the reported pass rate is

$$
\begin{aligned}
\operatorname{Var}(\hat p) = {} & \frac{\sigma^2_q}{n_q} + \frac{\sigma^2_d}{n_d} \\
& + \frac{\sigma^2_r}{n_q n_d n_r} + \frac{\sigma^2_j}{n_q n_d n_r n_j}
\end{aligned}
$$

where:

- $\hat p$ is the reported pass rate;
- $\sigma^2_q$, $\sigma^2_d$, $\sigma^2_r$, and $\sigma^2_j$ are the variances contributed by one query, one documentation version, one rollout, and one judge call;
- $n_q$, $n_d$, $n_r$, and $n_j$ are how many of each the design samples.

In words: each source's variance is divided by the number of independent draws of that source the design averages over, so a source you sample once never shrinks.

An illustrative DocsAgent budget shows what that means. Suppose DocsAgent passes 70% of 200 queries on the current documentation, with one rollout and one judge call each: 70% (95% Wilson CI 63–76%, n = 200). Suppose also that the query, rollout, and judge-call variances are 0.12, 0.06, and 0.04, together close to the 0.21 that a pass-or-fail score near 70% must have, and that documentation versions shift the whole suite's pass rate with a standard deviation of 4 points, so $\sigma^2_d = 0.0016$. All of these values are illustrative.

| Source | Variance per draw | Draws | Share of $\operatorname{Var}(\hat p)$ |
|---|---|---|---|
| Query | 0.12 | 200 | 22.2% |
| Documentation version | 0.0016 | 1 | 59.3% |
| Rollout | 0.06 | 200 | 11.1% |
| Judge call | 0.04 | 200 | 7.4% |

The contributions sum to $\operatorname{Var}(\hat p) = 0.0027$, a 95% half-width of ±10.2 points. The version term is three-fifths of the budget, and no number of queries can shrink it: with every query on one version, even unlimited queries leave a 95% half-width of ±7.8 points. The next 200 trials buy different amounts depending on where they go:

| Next step | Added cost | 95% half-width |
|---|---|---|
| None | — | ±10.2 points |
| 200 more queries on the same version | 200 trials | ±9.1 points |
| A second rollout for every query | 200 trials | ±9.7 points |
| Three judge calls per answer instead of one | 400 judge calls | ±9.9 points |
| The same 200 queries on a second version | 200 trials | ±8.0 points |
| Pin one version and narrow the claim to it | None | ±6.5 points |

The second version buys the most, which is the variance budget's point: spend the next dollar on the dominant component, and increase replication where the decision is unstable rather than everywhere uniformly. The last row is a different kind of move. Pinning the documentation takes the version term out of $\sigma^2_{\Delta}$ by narrowing the claim to one release, and the [validity domain](https://benchmaker.ai/glossary/validity-domain) shrinks to match.

The last row also shows why a narrow interval is not a reliable instrument. Its ±6.5 points is close to the [Wilson interval](https://benchmaker.ai/glossary/wilson-interval) on the original run, because the binomial interval quietly assumes the same thing: that the 200 trials are the only randomness. With enough tasks, a benchmark can estimate its own biased scoring rule very precisely, and a high reliability coefficient does not show that a two-point difference between models is statistically or practically meaningful. Keep three questions apart for every reported claim: is the measurement stable across legitimate replications, is the estimate precise enough for the comparison or threshold, and is the interpretation valid for the intended use? [Estimands, pass rates, and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals) and [Comparing systems](https://benchmaker.ai/handbook/validation/comparing-systems) take up the second question.

## In practice

Run a generalizability study for one reported score.

1. **List what can vary.** Write down every variable that can change without a model change: tasks, rollouts, judge calls, judge versions, raters, environment snapshots, index builds, tool errors, harness versions, and time.
2. **Classify each against the claim.** Mark it a controlled condition (pin and record it), legitimate variation (sample it), or measurement error (estimate and reduce it).
3. **Sample every legitimate source more than once.** One version, one judge call, or one rater gives no estimate of that source's variance.
4. **Run the studies that apply.** Repeated rollouts, alternate forms, inter-scorer overlap reported as confusion matrices, and equivalent environment snapshots.
5. **Estimate the components.** Fit a mixed-effects model with tasks as clusters, and keep the per-trial records so that the analysis can be rerun.
6. **Build the variance budget.** For the reported score and the decision it serves, compute each source's contribution and the interval it implies, and set it beside the naive binomial interval.
7. **Spend or narrow.** Put the next run where the dominant component is, or narrow the claim and record the narrower validity domain.

The artifact is a variance budget: the replication universe with each variable's classification, the estimated components, their contributions to the reported score's uncertainty, and the design change they justify. It fills the reliability field of the evaluator dossier — replication facets, variance components, inter-rater agreement, and repeatability.

## Common failure modes

- **Agreement reported as reliability** — one inter-rater coefficient stands in for the reliability of the whole eval → measure the facets that move the decision: tasks, rollouts, judges, and environments. See [one-number reporting](https://benchmaker.ai/handbook/practice/anti-patterns#one-number-reporting).
- **An unstated replication universe** — tool errors are noise in one report and target behavior in the next → classify every variable against the claim before measuring.
- **Precision mistaken for reliability** — many tasks on one snapshot with one judge produce a narrow interval → budget every legitimate source, and report the interval it implies.
- **Correlated trials** — shared state or a flaky dependency makes trial outcomes co-move → isolate trials and resample at the task level. See [correlated trials](https://benchmaker.ai/handbook/practice/anti-patterns#correlated-trials).

## Check yourself

**Analyze.** For an interactive agent benchmark you know, list every variable that can change without a model change, and classify each as a controlled condition, legitimate target-environment variation, or measurement error.

> [!DETAILS] Hint
> Start from the facets of the replication universe above — task form, rollout, judge, rater, tool errors, simulated user, environment snapshot, index build, harness version, and time — and let the claim decide each classification.

**Evaluate.** A benchmark report cites high inter-rater agreement and thousands of tasks as evidence that its leaderboard is reliable. Evaluate the claim.

> [!DETAILS] Hint
> Agreement covers one facet, and the task count buys precision. Rollouts, environment versions, and the choice of judge are unmeasured, and any of them can reorder the leaderboard.

**Create.** Construct two evals with identical reliability but radically different validity. Then construct two with similar validity targets but radically different reliability. For each pair, name the engineering intervention that fixes the weaker one.

> [!DETAILS] Hint
> For the first pair, grade the same SQL tasks by exact string match and by execution with result-set comparison: both are deterministic, and only one tracks semantic correctness. For the second, apply the same rubric with one judge call or with several calls and adjudication.

## Further reading

- [The judge as a noisy sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor) — inter-scorer evidence as error rates, and why Cohen's κ depends on prevalence.
- [Estimands, pass rates, and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals) — the precision question: Wilson intervals and clustered trials.
- [Harness architecture and reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility) — isolation, pinned environments, and the per-trial record a variance study needs.
- [Sources of validity evidence](https://benchmaker.ai/handbook/validation/validity-evidence) — what a stable score still has to show.
- Brennan (2001), *Generalizability Theory* — variance components and the absolute coefficient Φ.
- Riezler and Hagmann (2024), *Validity, Reliability, and Significance: Empirical Methods for NLP and Data Science* — mixed-effects reliability analysis for machine-learning evaluation.

---
Source: https://benchmaker.ai/handbook/validation/reliability