Comparing systems

Comparing systems means estimating how much better one system is than another for the same claim, with its uncertainty, and deciding whether the difference is worth acting on. A paired design runs both on the same tasks, so task difficulty drops out and only the tasks where they disagree carry information.

Pair the tasks#

Fair comparison across models and agents is one of the reasons evaluation exists (Song, 2025, slide 6). It starts with the design: run the variants on the same tasks whenever possible, because pairing takes task difficulty out of the comparison between models. Comparisons should exploit task-level pairing, for example McNemar-style analysis for binary outcomes, rather than treat the systems as independent samples.

With binary outcomes, every task falls into one cell of a two-by-two table. In an illustrative comparison, a release candidate (A) and the current release (B) of SQLAgent, the text-to-SQL agent of Case C, run on the same 100 tasks:

A passes 75 tasks and B passes 65. Read as independent samples, the two rates are hard to tell apart: 75% (95% Wilson CI 66–82%, n = 100) against 65% (95% Wilson CI 55–74%, n = 100), with intervals that overlap widely. But 80 of the 100 tasks say nothing about which system is better, because both systems pass or both fail. The information sits in the 20 discordant pairs: the 15 tasks that only A passes and the 5 that only B passes.

Discordant pairs and McNemar's test#

McNemar's test uses only the discordant counts:

$$\chi^2 = \frac{(b - c)^2}{b + c}$$

where:

• $b$ is the number of tasks that A passes and B fails;
• $c$ is the number of tasks that B passes and A fails.

These counts are unrelated to the exposure budget $b$ of the regime tuple. In words: if the systems were equally good, each discordant task would be equally likely to favor either one, so the statistic measures how lopsided the disagreements are relative to how many there are. Under that hypothesis it approximately follows a chi-square distribution with one degree of freedom; when $b + c$ is small, use the exact binomial test on the discordant pairs instead.

For the table above, $\chi^2 = (15 - 5)^2/20 = 5.0$, with p = 0.025, and the exact test on the 20 discordant pairs gives p = 0.041. An unpaired test on the same pass counts, which ignores which tasks the passes came from, gives p = 0.12. Pairing changed nothing about the data; it stopped the 80 agreeing tasks from diluting the comparison.

A p-value measures only how surprising the data would be if the systems were equally good, so report the effect size and its uncertainty, not only a p-value. Here the candidate gains $(b - c)/N = 10$ points on $N = 100$ paired tasks, and a 95% interval for the paired difference runs from about +1 to +18 points. A paired bootstrap, which resamples whole tasks with both systems' outcomes attached, gives nearly the same range. The interval is compatible with a negligible gain and with a large one, which the p-value alone hides. The opposite error matters too: a gain of 0.2 points can be statistically detectable and still irrelevant to the product decision.

The calculator starts at this table. Move the slider for agreeing tasks that both pass: McNemar's test stays put while the unpaired p-value moves. Then trade b against c with their sum held at 20, and watch the exact test cross p = .05 between 14–6 and 15–5.

Size for the difference that matters#

A comparison that finds nothing has not shown that nothing is there: an underpowered comparison that comes back "not significant" is weak evidence of absence, not evidence of no effect. Statistical power is the probability that a test detects a real difference of a stated size. A low-power design misses a real effect of the size that matters most of the time, so size the suite for the smallest difference that would change the decision before reading anything into a null.

Because McNemar's statistic depends only on the discordant pairs, a rough sizing rule follows from its normal approximation:

$$N \approx \frac{\left(z_{1-\alpha/2} + z_{1-\beta}\right)^2 d}{\delta^2}$$

where:

• $N$ is the number of paired tasks;
• $\delta$ is the smallest true difference in pass rates that would change the decision;
• $d$ is the expected share of tasks on which the two systems disagree;
• $z_{1-\alpha/2}$ and $z_{1-\beta}$ are standard normal quantiles for the significance level $\alpha$ and the power $1 - \beta$: 1.96 and 0.84 for $\alpha = .05$ and 80% power.

In words: the suite grows with how often the systems disagree and with the inverse square of the difference you need to see, so at a fixed disagreement rate, halving the difference that matters quadruples the suite.

In an illustrative release, a 5-point gain would change the decision, and past releases disagreed on 15% of tasks. The rule gives $N \approx 7.85 \times 0.15 / 0.05^2 \approx 470$ tasks. It runs slightly optimistic: with 10% of tasks favoring the candidate and 5% the current release, exact calculation puts 80% power for McNemar's exact test at about 500 tasks. At the 100 tasks of the example above, the same test detects that real 5-point gain only about 18% of the time, so a "not significant" result from 100 tasks would say almost nothing about whether the gain exists.

Many gates, many looks#

Two habits inflate false alarms without changing a single task.

The first is testing many gates. With $m$ independent gates, each tested at level $\alpha$, the chance that a change which does nothing trips at least one of them is

$$\Pr[\text{at least one false alarm}] = 1 - (1 - \alpha)^m$$

where:

• $m$ is the number of gates;
• $\alpha$ is the significance level of each gate.

In words: each gate has a small chance of firing by accident, and the chances accumulate across gates. Six gates at α = .05 give about a 26% chance that a release which changed nothing fails at least one of them. Pre-register the gates or correct for the family-wise false-alarm rate, for example by testing each gate at α/m, which is 0.0083 for six gates. The lower panel of the calculator above starts at six gates and shows 26.5%. Positively correlated gates tend to fire together, which lowers the uncorrected rate, but $m\alpha$ bounds it under any dependence.

The second is optional stopping: checking a comparison as data arrive and stopping as soon as it turns significant, which inflates false positives. Each look is another chance for noise to cross the line. In an illustrative check, two SQLAgent versions with identical true pass rates, disagreeing on 15% of tasks, are compared with McNemar's exact test after every 10 tasks up to 200. Stopping at the first p < .05 declares a winner about 11% of the time; a single test at 200 tasks does so about 3% of the time. Use a fixed sample size, set in advance from the power calculation, or an explicitly sequential design.

When significance is not enough#

Variance beyond the task sample#

McNemar's test treats the sample of tasks as the only source of chance. Agent evaluations have more: rollouts, judges, and environment versions vary too. Riezler and Hagmann treat significance as the third empirical question, asked once validity and reliability are in hand (Riezler & Hagmann, 2024). In an agent setting the analogue is mixed-effects significance: model system identity as the effect of interest, treat tasks, rollouts, judges, and environment versions as variance components, and use a likelihood-ratio test, bootstrap, or posterior contrast to ask whether the difference between systems outlasts that variation.

The score of DocsAgent, the documentation agent of Case G, varies with query choice, model stochasticity, judge stochasticity, and document version, so a claimed gain for a new DocsAgent should survive all four, not only the sampling of queries. Estimating those components is the business of reliability and generalizability.

Error rates that differ by system#

When a judge grades both systems, its error rates can differ between them, which is differential misclassification, and a system optimized against the judge can turn an exploit into an apparent gain: in the illustrative case in the judge as a noisy sensor, two systems that both truly pass 60% of tasks end up 7.5 points apart after a correction with one system's error rates. Audit judge error per compared system, or at least per generator family and optimization regime, before testing the difference.

Validity comes first#

The ordering matters: statistical significance cannot rescue invalid measurement. A highly significant difference on a benchmark dominated by leakage or a construct-irrelevant shortcut is precise evidence about the wrong thing. If the candidate's extra passes came from schema artifacts that leak the table choice, a smaller p-value would only describe the leak more precisely. Test the difference after the validity argument supports the measurement, not instead of it.

In practice#

Write a comparison plan before the run, and complete it as a record after.

1. Fix the comparison. Name the two systems, the estimand (the paired difference on the claim's population), the smallest difference that would change the decision, and the gates.
2. Size the suite. Estimate the disagreement rate from a pilot or past releases, then size N for 80% power at that difference, per gate.
3. Pair everything. Run both systems on the same tasks, environment versions, and judge configuration, and keep per-task outcomes.
4. Audit the grader per system. If a judge grades, estimate its error rates on adjudicated samples of each system's outputs.
5. Test and estimate. Count b and c; run McNemar's test, exact when b + c is small, or a paired bootstrap; report the difference with its interval. Where rollouts, judges, or environments vary, fit a mixed-effects model.
6. Hold the line. Apply the planned correction for multiple gates and stop at the planned N.

The artifact is a comparison record: the pre-registered estimand, minimum difference, N, and gates; the two-by-two table; the test, the effect size, and its interval; and the per-system judge audit. It answers the audit questions on whether comparisons are paired and whether the evidence supports the size of the claimed difference rather than only its existence.

Common failure modes#

• Overlapping intervals read as no difference — two independent Wilson intervals overlap, so the change is dismissed → pair the tasks and test the discordant pairs.
• A p-value without an effect size — "significant, p = 0.03" with no difference or interval → report the paired difference and its interval. See one-number reporting.
• A null read as no effect — "not significant on 100 tasks, so v2 is no worse" → compute power for the smallest difference that matters before the run.
• Uncorrected gates and peeking — six gates at .05, or a comparison re-run until it turns significant → pre-register the gates, correct for the family, and fix N or use a sequential design.
• Significance on a broken measure — a small p-value on a leaked or shortcut-ridden benchmark → settle validity first. See contamination.

Check yourself#

Analyze. Candidate A passes 75 of 100 tasks and the current release B passes 65. A reviewer says the Wilson intervals overlap, so there is no evidence of improvement. What do you need to know, and how could it change the conclusion?

Hint

The paired table. With 15 tasks that only A passes and 5 that only B passes, the exact McNemar test gives p = 0.041; with 25 and 15, the same pass counts give p = 0.15.

Evaluate. "On 100 paired tasks, v2 was not significantly worse than v1 (p = 0.4), so it is safe to ship." Evaluate the argument for a product where a 5-point regression would be unacceptable.

Hint

If the versions disagree on about 15% of tasks, 100 tasks detect a real 5-point difference only about 18% of the time. Size the suite for the regression you must rule out, or report the interval and ask whether it excludes −5 points.

Create. Plan the release comparison for a coding agent whose candidate must pass six gates, three of them graded by an LLM judge, with three rollouts per task.

Hint

Fix the smallest relevant difference per gate and size N for it; test each gate at α/6 or pre-register another family-wise correction; fix N in advance; audit the judge's TPR and FPR on both systems' outputs; and treat tasks as clusters, with rollouts and judge calls as variance components.

Further reading#

• Estimands, pass rates, and intervals — intervals for a single system, clustered trials, and reweighting.
• Reliability and generalizability — the variance components a mixed-effects comparison needs.
• The judge as a noisy sensor — differential misclassification in full.
• Decision claims — where the smallest difference that matters comes from.
• Riezler and Hagmann (2024), Validity, Reliability, and Significance: Empirical Methods for NLP and Data Science — significance as the third question, after validity and reliability.