Glossary · Evaluator mechanisms
Dark current
2InstrumentA judge's baseline preference when there is no real quality signal to detect: what it favors when the options do not differ in the construct. One of the properties a judge datasheet probes.
Defined in
Handbook · InstrumentLLM judges in practiceAn LLM judge is a learned measurement instrument. Design its inputs, characterize its biases, calibrate it against adjudicated labels, and version it.Related terms
- Judge datasheet
- A maintained record that characterizes a learned judge as an instrument: target sensitivity, nuisance invariance, threshold behavior, reference agreement by criterion and slice, and repeatability and transport. Measure the measuring device before trusting it.