LLM judges in practice

An LLM judge is a general-purpose language model prompted to evaluate another system's output, often with a rubric, references, or a pairwise comparison. It is a learned measurement instrument, not a verifier: its errors are empirical, depend on everything it is shown, and must be measured before its verdicts are trusted.

Every input is part of the instrument#

The right mental model is not that the LLM is a verifier; it is that the LLM is a learned measurement instrument. A rubric-conditioned judge computes

$$z = g_\theta(x, y, \Psi, e)$$

where:

• $x$ is the task and $y$ the candidate output;
• $\Psi$ is the specification in rubric form, the criteria the judge is told to apply;
• $e$ is the evidence the judge is shown, fixed by the instrument's evidence view;
• $\theta$ is the configuration: model and version, prompt, examples, decoding settings, and the parser that extracts the verdict;
• $z$ is the signal the judge emits.

In words: the verdict depends on everything the judge sees and how it was set up, not on the output alone.

Change any argument and the instrument changes. An LLM judge shown only the final answer is not the same instrument as the same judge shown cited passages, tool traces, hidden state, or database snapshots (see Anchors and evidence views). Take DocsAgent, the documentation agent of Case G, whose answers must be grounded in the product documentation. A grounding judge that reads only the answer has nothing to check grounding against; given the extracted claims and the spans they cite, the same model can check each claim.

The LLM judge belongs to the learned judge family, with reward models, classifiers, and learned process reward models. All of them have an empirical validity domain, and their limitations are empirical: competence, calibration, bias, distribution shift, self-preference, and optimization vulnerability. In this handbook a judge is always learned or prompted; people who score are raters, panels, or adjudicators. Mechanism families compares them.

Agreement justifies a judge but does not characterize it#

Zheng et al. (2023) made LLM-as-a-judge a measurable method. Their MT-Bench and Chatbot Arena studies reported about 80% agreement between strong judges and humans, which justifies the approach, and documented position, verbosity, and self-enhancement biases and limited reasoning ability, which qualify it (via Chen, Wang & Qu, 2026, §5.1).

Agreement is useful evidence, but it does not fully characterize an instrument. One agreement figure cannot say whether the misses are false passes or false failures, whether they cluster on one generator family, or whether a verdict survives swapping two candidates' order. A judge can agree with itself and with a homogeneous human panel while all of them apply the wrong criterion. The outcome-validity checklist for agentic benchmarks therefore asks for documented or experimental evidence of a judge's accuracy, self-consistency, and agreement with humans (O.c.1), and a design that resists adversarial inputs and reward hacking (O.c.2) (Zhu et al., 2025, via Song, 2025, slide 42).

The judge datasheet#

Usami et al. (2026) replace the single agreement number with a Judge Datasheet that probes, among other properties, the judge's baseline preference when there is no real quality signal, which they call dark current, its positional false preference, and the decision criterion its prompt induces. The lesson: measure the measuring device before trusting the measurements.

Keep a judge datasheet for every learned judge, with at least five classes of evidence. The last column is an illustrative plan for the DocsAgent grounding judge.

Reference agreement is where the sensor model lives: scored against adjudicated labels, a binary judge has a true-positive and a false-positive rate, and these correct its raw pass rate through the Rogan–Gladen correction only if they transport to the outputs being graded, which cannot be assumed for a system optimized against the judge (differential misclassification). The judge as a noisy sensor derives the correction and its cautions, and explains why the datasheet reports confusion matrices and prevalence rather than Cohen's κ alone.

Threshold behavior needs its own entry because a new prompt often moves a judge's operating point without improving its underlying resolution (Usami et al., 2026). Take an illustrative pair: a lenient prompt gives a judge TPR 0.90 and FPR 0.30, and a stricter one TPR 0.70 and FPR 0.10. Both versions agree with adjudicated labels on 80% of a balanced calibration set, with the same separation, TPR − FPR = 0.60. On a system that truly passes 50% of tasks, the lenient version reports 60% and the strict one 40%, so an unversioned prompt edit moves every downstream number by 20 points while the system stands still. Corrected with each version's own error rates, both readings return 50%.

The calculator starts at the lenient version; set TPR to 70%, FPR to 10%, and the observed rate to 40%, and the raw reading falls 20 points while the corrected rate stays at 50%.

When raters disagree#

Reference agreement is only as good as its labels. They come from qualified experts who double-label part of a calibration set concentrated around hard boundaries, not random easy examples, and from an adjudicating panel that is the trust root: it can define a standard while having measurable disagreement of its own.

Treat disagreement as data, and separate it into at least four causes:

• Random noise: no pattern across items or raters; adjudication and more labels reduce it.
• An underspecified rubric: raters read the criterion differently, and each reading is defensible.
• Legitimate plural values: raters weigh a real trade-off differently. Record the trade-off instead of averaging it away, because simplifying a measure until raters agree can reduce its validity (Reidsma & Carletta, 2008).
• Expertise differences: one rater sees what another cannot, so the criterion needs a stated qualification.

For the second cause the rule is blunt: when experts disagree systematically, revise the specification before blaming the judge, because a better judge would only apply an ambiguous criterion more consistently (see Specification). If two DocsAgent SMEs split on whether a step paraphrased from an older documentation version counts as grounded, the judge is not at fault: the criterion card lacks a boundary rule, and stale documentation is a validity-domain question that belongs to the environment rather than the judge.

Verdicts, panels, and repeated calls#

By default, have the judge emit a binary verdict per atomic criterion, with abstention as a third value and an escalation route for low-confidence and out-of-domain cases. Binary verdicts that experts can adjudicate are easier to calibrate than an unanchored 1–10 score, and they are what the sensor correction needs; Signals and resolution explains why verdicts come first and why abstention is a value, not a gap.

The Berkeley slides list common fixes (Song, 2025, slide 22). Cross-checking with human evaluation supplies reference agreement. Repeating the judgment with the same model measures, and averages away, the judge's randomness but not its bias. Several rubrics become a set of atomic criteria. Detailed prompts, chain-of-thought, and reasoning modes join $\theta$: frozen, versioned, and re-measured on change. Two fixes need conditions.

More dangerous under optimization#

Used once for reporting, a judge is a sensor; used for best-of-N, it becomes a search objective; and used as a policy reward, it becomes a training target. The model is unchanged; its use regime is not.

Selection exposes false positives. If the lenient judge above (TPR 0.9, FPR 0.3) runs best-of-N selection over candidates that are each correct 20% of the time, the selected output is correct only about 43% of the time however many candidates are drawn, under a benign selection model whose assumptions Adaptive validity spells out. Training goes further: error rates measured on ordinary outputs need not hold on outputs selected to maximize the judge, and a learned proxy optimized hard enough can keep rising after a stronger reference measure starts to fall (Gao et al., 2023). When the judge shares the generator's model family, their errors correlate and selection drifts toward shared blind spots, a self-confirming loop that needs no adversary (Chen, Wang & Qu, 2026, §5.3); cross-family judging is the structural defense.

In every regime, the judge must remain the instrument that was calibrated. Judge rot is the failure where it does not: unversioned prompts, no recalibration after a model swap, a well-meaning edit that silently repoints the judge, and every downstream number inherits the rot. Treat rubric text, examples, judge model, tool checkers, thresholds, aggregation logic, and expert policy as one measurement version, and revalidate, or run an explicit bridging study, whenever a load-bearing part changes.

In practice#

Calibrate each new judge one criterion at a time with this twelve-step protocol:

1. Define one criterion and its criterion card, following Rubrics and criterion cards.
2. Collect a stratified calibration set concentrated around hard boundaries and known failure slices.
3. Double-label a subset with qualified experts and adjudicate the disagreements.
4. Separate the disagreement into random noise, an underspecified rubric, legitimate plural values, and expertise differences.
5. Revise the specification before blaming the judge when experts disagree systematically.
6. Freeze the judge: model, prompt, examples, decoding settings, parser, and the evidence it is shown.
7. Measure sensitivity, specificity, the confusion matrix, calibration, and decision agreement by criterion and slice.
8. Run nuisance-invariance tests and adversarial probes, including order swaps and irrelevant verbosity.
9. Compare against an independent measurement channel when one is available, as convergent evidence.
10. Document discriminant tests showing that the judge is not mainly reacting to nuisance properties.
11. Define an abstention and escalation region for low-confidence or out-of-domain cases.
12. Revalidate after judge or model changes, material distribution shift, or production disagreement alarms.

The artifact is a versioned judge datasheet per criterion, including the calibration set's composition and prevalence and the revalidation triggers; its reference-agreement entry feeds the natural-error entry of the validation profile.

Common failure modes#

• Agreement as characterization — "agrees with humans 80% of the time" is the whole validation → keep a datasheet with all five classes of evidence. See one-number reporting.
• Judge rot — a prompt edit or model swap moves every downstream number with no new version → version the whole configuration and recalibrate on change. See judge rot.
• A same-family panel — judges from one model family vote, and their shared blind spot reads as consensus → measure the panel as one sensor and diversify mechanisms. See common-mode assurance.
• A reporter promoted to reward — a judge validated for reporting becomes a selector or a reward without new evidence → re-measure its error in the new regime and test it against the actual optimizer. See reward hacking.

Check yourself#

Analyze. A team edits its grounding judge's prompt to "be stricter," and the weekly pass rate falls from 60% to 40% while the assistant is unchanged. Use the datasheet to explain what changed and what you need before reporting either number.

Hint

Start with threshold behavior: a stricter prompt can lower TPR and FPR together without improving separation. Measure both versions on the same adjudicated set and compare corrected rates. Either way, the edit is a new measurement version.

Evaluate. To cut SME cost, a team proposes certifying DocsAgent releases by a majority vote of three judges from the model family that generates the answers. Argue for or against the proposal.

Hint

A vote reduces independent noise, not correlated bias, and a judge from the generator's family invites generator coupling. Which independent channel would the release gate keep?

Create. Design a nuisance-invariance test suite for a rubric-guided judge in your own domain.

Hint

List the nuisances first (order, verbosity, formatting, names, dialect, presentation). For each, write a semantics-preserving transformation, the verdict that should not change, and the flip rate you will tolerate. Add a quality ladder to test sensitivity.

Further reading#

• The judge as a noisy sensor — error rates, the Rogan–Gladen correction, and differential misclassification.
• Rubrics and criterion cards — the criterion a judge applies, and the judgment compiler (this handbook's synthesis) that produces it.
• Adaptive validity — a judge's error once something optimizes against it.
• Use regimes: same judge, four jobs — one judge as reporter, selector, reward, and gate.
• Zheng et al. (2023), "Judging LLM-as-a-judge with MT-Bench and Chatbot Arena," NeurIPS Datasets and Benchmarks Track — the agreement figures and the classic biases.
• Usami et al. (2026), "LLM judges have dark current: A psychometric datasheet for LLM-as-a-judge evaluation" — the datasheet and dark current.