The judge as a noisy sensor

Treating a judge as a noisy sensor means modeling its binary verdicts the way epidemiology models a screening test: by a true-positive rate and a false-positive rate, measured against adjudicated labels. The model shows how far the judge's raw pass rate sits from the true one, how to correct it, and when no correction can be trusted.

The sensor model#

A binary judge emits pass or fail for each item. Score it against a reference — independently adjudicated labels under a specification that has already been justified — and it has two error rates:

• its true-positive rate (TPR), or sensitivity: the probability that it passes an item that truly passes;
• its false-positive rate (FPR): the probability that it passes an item that truly fails, which is one minus its specificity.

If the system under evaluation truly passes a fraction $p$ of the population, the judge reports

$$p_{\text{obs}} = p\,\mathrm{TPR} + (1 - p)\,\mathrm{FPR}$$

where:

• $p$ is the system's true pass rate, relative to the specification;
• $\mathrm{TPR}$ and $\mathrm{FPR}$ are the judge's error rates on this system's outputs;
• $p_{\text{obs}}$ is the pass rate the judge reports.

In words: the reading blends the true passes the judge catches with the true failures it lets through. This is the same mathematics as estimating the prevalence of a disease from an imperfect screening test (Rogan & Gladen, 1978).

Two consequences can be read straight off the model. Take an illustrative judge with TPR 0.90 and FPR 0.10. On a system that truly passes 75% of tasks it reports $0.75 \times 0.90 + 0.25 \times 0.10 = 70\%$; on a system that truly passes 20% it reports 26%. First, the bias depends on the system: every reading lies between FPR and TPR, so this judge under-reports strong systems and over-reports weak ones. Second, differences shrink by the factor TPR − FPR: if the judge errs the same way on two systems, a true 10-point gap between them reads as an 8-point gap.

Nothing in the model is specific to LLMs. A human rater is also a measurement mechanism with ordinary error, and the adjudicated labels that define "truly passes" come from a trust root whose own disagreement should be measured. What the sensor view will not accept is agreement in place of error rates: a judge can agree with itself and with a homogeneous human panel while all of them apply the wrong criterion.

Familiar evaluation terms map onto the model like this:

Inverting the model#

If the error rates transport to the eval set and TPR ≠ FPR, the sensor model can be inverted:

$$\hat p = \frac{p_{\text{obs}} - \mathrm{FPR}}{\mathrm{TPR} - \mathrm{FPR}}$$

where $\hat p$ is the corrected estimate of the true pass rate and the other symbols are as above. In words: subtract the false passes you expect, then rescale by how well the judge separates true passes from true failures. When TPR equals FPR, the verdict is independent of the truth and there is nothing to invert.

This inversion is the Rogan–Gladen correction. For the judge above, a 70% reading corrects to $(0.70 - 0.10)/(0.90 - 0.10) = 75\%$. It needs binary verdicts and measured error rates, one more reason to decompose a vague rubric into binary criteria that experts can adjudicate. Where a judge's error model has been measured, report the corrected rate, not the raw reading.

The calculator starts at this example. Try three things:

• Drag the observed pass rate down to 6%: the estimate clips at 0%, the first caution below.
• Set FPR to 40% and the observed rate to 50%, then move TPR between 60% and 65%: five points of TPR move the estimate ten points.
• With the same settings, move TPR from 90% to 95%: the estimate moves less than two points.

Before you trust the correction#

Three cautions make the correction decision-grade rather than decorative. A fourth limits what any correction can do.

Clip, then ask why#

The corrected estimate can leave the interval from 0 to 1. With the judge above, a 6% reading gives $\hat p = -5\%$. Clip it to zero and treat the clipping as a finding: the observed rate is impossible under the calibration error rates, so those rates do not transport to this data.

Carry the calibration noise#

TPR and FPR are themselves estimates from a finite calibration set, and their sampling noise propagates into the corrected rate. To first order:

$$\begin{aligned} \operatorname{Var}(\hat p) \approx \frac{1}{J^{2}} \Big[\, & \frac{p_{\text{obs}}(1 - p_{\text{obs}})}{n} \\ &+ \hat p^{2} \operatorname{Var}(\mathrm{TPR}) \\ &+ (1 - \hat p)^{2} \operatorname{Var}(\mathrm{FPR}) \,\Big] \end{aligned}$$

where:

• $J = \mathrm{TPR} - \mathrm{FPR}$ is the judge's separation between true passes and true failures;
• $n$ is the number of eval items the judge graded;
• $\operatorname{Var}(\mathrm{TPR})$ and $\operatorname{Var}(\mathrm{FPR})$ are the sampling variances of the calibration estimates — for binomial estimates, about $\mathrm{TPR}(1-\mathrm{TPR})/n_{+}$ and $\mathrm{FPR}(1-\mathrm{FPR})/n_{-}$, with $n_{+}$ adjudicated passes and $n_{-}$ adjudicated failures.

In words: the eval sample's noise and the calibration set's noise both pass through the correction, and both are divided by $J^2$.

An illustrative calculation shows the size of the effect. The judge above grades n = 400 items and reports 70% (95% Wilson CI 65–74%, n = 400), and its error rates come from 100 adjudicated passes and 100 adjudicated failures. Treating TPR and FPR as exact gives $\hat p = 75\%$ with a 95% half-width of about ±5.6 points; carrying their noise widens it to about ±8.1 points. A weaker judge on the same system, with TPR 0.70 and FPR 0.30 so that $J = 0.4$, reads 60%, still corrects to 75%, and has a half-width of about ±21 points: a weak judge makes correction less stable, not more.

These delta-method figures size the problem; they are not a release-grade interval. With a hundred or so expert labels the calibration terms are not negligible, and decision-grade intervals on corrected rates need a construction designed for the purpose, such as the one by Lang and Reiczigel (2014).

Keep the calibration set fresh#

The correction assumes the error rates carry over from the calibration set to the eval set. Slice-dependent judge error breaks that assumption, which is why the calibration set must not go stale. Re-measure TPR and FPR after any change to the judge's model, prompt, examples, or decoding settings, after material shift in the task mix, and when production disagreement alarms fire.

The correction cannot repair the specification#

The correction removes a known measurement error relative to a specification that was already justified. It does not rescue a rubric that operationalizes the wrong construct. A well-corrected pass rate on the wrong criterion is still a measurement of the wrong criterion; that problem belongs to the validity argument.

Differential misclassification#

The correction's most fragile assumption is that one set of error rates fits every system. In AI evaluation, transport can fail by system. If System B was trained directly against a judge and System A was not, nothing guarantees that the judge's TPR and FPR are the same on both, because optimization moves B's outputs toward the regions the judge selects. Error rates that differ between the systems being compared are differential misclassification.

An illustrative case makes the bias concrete. Both systems truly pass 60% of tasks. On A's outputs, the judge has TPR 0.90 and FPR 0.10. B was tuned against the judge, and its failures now fool the judge a quarter of the time: FPR 0.25 on B's outputs.

The correction is not at fault. Given the wrong error rates, it faithfully converts an exploit into a 7.5-point improvement. Audit judge error per compared system, or at least per generator family and optimization regime.

Differential misclassification is where measurement meets optimization. As optimization pressure grows, the judge's error profile moves with it (adaptive validity), and comparisons between systems inherit the bias (comparing systems).

Agreement is not an error rate#

Judge studies often report a single agreement coefficient. The standard chance-corrected choice is Cohen's κ, computed as $(p_o - p_e)/(1 - p_e)$ from observed agreement $p_o$ and chance agreement $p_e$. It depends on prevalence: the same scorer, with identical sensitivity and specificity, posts very different values at different base rates. This handbook always writes the name in full, because a bare κ denotes the consumer in the regime tuple.

Two illustrative calibration sets, scored against the same judge, show the size of the effect:

Nothing about the judge changed, yet Cohen's κ nearly halved. Report the confusion matrix and TPR and TNR alongside Cohen's κ, never the coefficient alone, and state the calibration set's prevalence with the number.

The same discipline applies to panels of judges. Majority vote can reduce independent noise; it does not cure correlated bias, and judges built from similar training data, model families, or prompts can form a more confident shared blind spot. A voting panel emits one verdict per item, so it is one sensor: measure its error rates as a unit.

In practice#

Build a judge error record for each binary criterion a judge grades. The procedure follows the judge calibration protocol in LLM judges in practice, narrowed to what the sensor model needs.

1. Fix the criterion and freeze the judge. Write the criterion card, then freeze the judge's model, prompt, examples, decoding settings, parser, and the evidence it is shown.
2. Build an adjudicated calibration set. Stratify it around hard boundaries and known failure slices, with enough true passes and true failures to estimate TPR and FPR separately. Double-label a subset with qualified experts and adjudicate the disagreements; when experts disagree systematically, revise the specification before blaming the judge.
3. Sample every system you will compare. Draw calibration items from each system's outputs, or at least from each generator family and optimization regime.
4. Estimate the error rates. Report TPR and FPR with intervals, by slice, together with the calibration set's prevalence, and check that they hold across generator families and over time.
5. Correct and propagate. Compute $\hat p$, treat any clipping as a transport failure, carry the calibration variance, and use a purpose-built interval for release decisions.
6. Report and schedule revalidation. Publish the corrected rate and its interval next to the raw reading, n, TPR, FPR, and prevalence, and list the events that trigger recalibration.

The artifact is a judge error record per criterion: the frozen configuration, the calibration set's composition and prevalence, error rates by slice and by system, the corrected estimate with its interval, and the revalidation triggers. It is the natural-error entry of the validation profile.

Common failure modes#

• Raw readings reported as results — the dashboard shows the judge's pass rate as if it were the system's → report the corrected rate, its interval, and the error model behind it. See one-number reporting.
• Stale calibration — the judge's prompt or model changed, or the task mix drifted, and nobody re-measured TPR and FPR → version the judge configuration and recalibrate on every change. See judge rot.
• One calibration for every system — error rates measured on one system's outputs are used to correct another's → audit per system before correcting or comparing. See reward hacking.
• Agreement in place of error rates — "Cohen's κ = 0.6 against humans", with no confusion matrix or prevalence → report TPR, TNR, the confusion matrix, and the calibration set's prevalence.

Check yourself#

Analyze. A vendor reports that its judge "agrees with human raters at Cohen's κ = 0.6" and that your system scores 82% under that judge. List what you need before you can state your system's true pass rate, and mark which items the vendor's report supplies.

Hint

You need TPR and FPR on adjudicated samples of your system's outputs, the calibration set's size and prevalence, the eval n, and whether your system was tuned against this judge. Cohen's κ supplies none of these on its own.

Evaluate. System B was trained with a judge as its reward; System A was not. Under that judge, B leads by 6 points raw and by 7.5 points after correction with error rates estimated on A's outputs. Which number belongs in the release notes?

Hint

Neither, yet. Both assume the judge errs the same way on B's outputs. Estimate B's own error rates on adjudicated samples, then compare the two systems on paired tasks, with intervals.

Create. Plan the calibration set for a binary "grounded answer" criterion that will gate a weekly release. Decide how many adjudicated passes and failures you need so that calibration noise adds no more than about one point to the corrected estimate's half-width, which slices to stratify, which systems to sample, and what triggers recalibration.

Hint

Hold the eval n fixed and grow $n_{+}$ and $n_{-}$ until the TPR and FPR terms are small beside the eval-sample term. In this chapter's example (n = 400, TPR 0.90, FPR 0.10), about 300 adjudicated items per class keeps the added half-width under one point; the requirement grows quickly as $J$ shrinks.

Further reading#

• LLM judges in practice — building and characterizing the judge whose error rates this chapter uses.
• Adaptive validity — how evaluator error changes as optimization pressure grows.
• Estimands, pass rates, and intervals — the eval-sample term: Wilson intervals and clustered trials.
• Comparing systems — paired comparisons, and how differential misclassification biases them.
• Rogan and Gladen (1978), "Estimating prevalence from the results of a screening test," American Journal of Epidemiology — the original correction.
• Lang and Reiczigel (2014), "Confidence limits for prevalence of disease adjusted for estimated sensitivity and specificity," Preventive Veterinary Medicine — intervals that carry the calibration uncertainty.