Calculator
McNemar test calculator for comparing two systems
Is system A really better than system B on the same tasks? McNemar's test on the discordant pairs, plus the false-alarm rate across several gates.
| B passes | B fails | |
|---|---|---|
| A passes | 60 | b = 15 |
| A fails | c = 5 | 20 |
A passes 75 of 100 tasks, B passes 65. Only the highlighted discordant cells enter McNemar’s test.
Pass-rate difference, A − B
+10.0 pts
(b − c)/N. 95% CI +1.2 to +18.4 pts (Agresti–Min).
McNemar χ²
5.00
Asymptotic p = 0.025, 1 df. Prefer the exact p when b + c is small.
Exact McNemar p
0.041
Two-sided binomial test on the 20 discordant pairs.
Unpaired test p, same data
0.123
Treats A and B as independent samples, discarding the pairing.
Many gates
A release must pass m checks, each tested at level α, on a change that does nothing.
At least one false alarm
26.5%
6 independent gates at α = 0.05: 1 − (1 − α)^m.
Per-gate α for 0.05 overall
0.0083
Bonferroni α/m. Šidák 1 − (1 − α)^(1/m) gives 0.0085.
- Uncorrected
- Bonferroni, each at α/m
Assumes independent gates and no real change. Positively correlated gates tend to fire together, so the rate is lower; mα bounds it under any dependence.
Pair the tasks. The 80 tasks where A and B agree say nothing about which is better: move the both-pass slider and McNemar’s test does not change, while the unpaired test does. The 20 discordant pairs carry the information. Report the difference with its interval, not just a p-value, and remember that significance cannot rescue invalid measurement: if the tasks or the grader are wrong, a small p is a precise answer to the wrong question. Fix n in advance, too; peeking and stopping once p < .05 (optional stopping) inflates false positives.
When two systems run on the same tasks, the tasks they agree on say nothing about which is better. McNemar's test uses only the discordant pairs: the tasks A passes and B fails, and the tasks B passes and A fails. Enter those counts to see the difference, its interval and the test, then check the false-alarm rate across several release gates.
What it computes#
With tasks that A passes and B fails, and tasks that B passes and A fails,
is approximately chi-square with one degree of freedom when the two systems perform equally. When is small, use the exact binomial test on the discordant pairs, which the calculator also shows.
Report the effect, on paired tasks, with its interval, not only a p-value. For comparison, the calculator also shows an unpaired test on the same data, which treats A and B as independent samples and discards the pairing.
Several gates at once#
A release that must pass checks, each tested at level , raises a false alarm on a change that does nothing with probability
for independent gates, and at most under any dependence. Pre-register the gates or correct for the family-wise false-alarm rate, for example by testing each gate at .
The chapter Comparing systems derives these and sizes a comparison for the smallest difference that would change the decision.