Skip to content
Benchmaker

Glossary · Evaluator mechanisms

Judge datasheet

2Instrument

A maintained record that characterizes a learned judge as an instrument: target sensitivity, nuisance invariance, threshold behavior, reference agreement by criterion and slice, and repeatability and transport. Measure the measuring device before trusting it.

Defined in

Handbook · InstrumentLLM judges in practiceAn LLM judge is a learned measurement instrument. Design its inputs, characterize its biases, calibrate it against adjudicated labels, and version it.
LLM-as-a-judge
A prompted general-purpose LLM evaluating another artifact, usually with a rubric, references, or pairwise comparison.
Dark current
A judge's baseline preference when there is no real quality signal to detect: what it favors when the options do not differ in the construct. One of the properties a judge datasheet probes.
Sensitivity and specificity (TPR, TNR, FPR)
A binary evaluator's error rates against a reference: sensitivity (TPR) is the probability it accepts a true positive, specificity (TNR) the probability it rejects a true negative, and the false-positive rate FPR = 1 − TNR.

See it in the full glossary →