Skip to content
Benchmaker

Calculator

LLM judge noise correction calculator (Rogan–Gladen)

Recover a system's true pass rate from the rate an imperfect judge reports, given the judge's true- and false-positive rates.

InteractiveCorrecting a noisy judge
90%
10%
70%

Corrected pass rate p̂

75.0%

Rogan–Gladen estimate, assuming the judge's error rates transfer to this system.

Raw reading minus corrected

−5.0 pts

How far the uncorrected judge pass rate is off.

Signal spread TPR − FPR

80%

As it shrinks, calibration noise in TPR and FPR is amplified.

hat p = fracp_obs - FPRTPR - FPR
0%25%50%75%100%0%25%50%75%100%True pass rate pJudge pass rateobserved 70%p̂ 75%
  • What the judge reports
  • Perfect judge

A judge with 90% sensitivity and a 10% false-positive rate reports 70% passing. If those error rates hold for this system, the true pass rate is about 75.0%. If the system was optimized against this judge, they probably don't — audit the judge per system.

An LLM judge that sometimes passes bad outputs and fails good ones reports a pass rate that is off in a predictable direction. If you know the judge's true-positive and false-positive rates on this system's outputs, the Rogan–Gladen correction recovers an estimate of the true pass rate. Enter the judge's rates and its reported pass rate to see the corrected value.

What it computes#

The judge's reported rate mixes its two error rates:

pobs=p TPR+(1−p) FPRp_{\text{obs}} = p\,\mathrm{TPR} + (1 - p)\,\mathrm{FPR}

where pp is the true pass rate relative to the specification, and TPR and FPR are measured against adjudicated labels on this system's outputs. Solving for pp gives the correction (Rogan & Gladen, 1978):

p^=pobs−FPRTPR−FPR\hat p = \frac{p_{\text{obs}} - \mathrm{FPR}}{\mathrm{TPR} - \mathrm{FPR}}

Cautions#

  • The judge's error rates belong to the outputs they were measured on. A system optimized against the judge can have different ones, which is differential misclassification.
  • As TPR − FPR shrinks, small errors in the two rates are amplified in the corrected value.
  • A result outside 0 to 1 should be clipped, and treated as evidence that the error rates do not transport to this eval set.
  • No correction rescues a rubric that operationalizes the wrong construct.

The chapter The judge as a noisy sensor develops the model and the variance of the corrected rate.

Terms on this page

The chapter behind it

Handbook · ValidationThe judge as a noisy sensorModel judge error like a diagnostic test: correct pass rates, carry calibration noise into the result, and audit error per system.

More calculators