Glossary · Evaluator mechanisms
LLM-as-a-judge
A prompted general-purpose LLM evaluating another artifact, usually with a rubric, references, or pairwise comparison.
Defined in
Handbook · InstrumentLLM judges in practiceAn LLM judge is a learned measurement instrument. Design its inputs, characterize its biases, calibrate it against adjudicated labels, and version it.Where it comes up
- Vocabulary without category errorsHandbook · Foundations
- Anchors and evidence viewsHandbook · Instrument
- Mechanism familiesHandbook · Instrument
- Task validity and outcome validityHandbook · Validation
- CrosswalkHandbook · Reference