Skip to content
Benchmaker

Glossary · Meaning and specification

Judgment compiler

2Instrument

An eleven-step procedure that compiles expert judgment, the source language, through a rubric, the intermediate representation, into executable graders, with calibration and validity studies as its tests.

Defined in

Handbook · InstrumentRubrics and criterion cardsA rubric is an intermediate representation, not a weak verifier. Compile tacit expertise into cards that send each criterion to its strongest mechanism.
Rubric
A structured, human-readable specification with criteria, allowed evidence, exemplar anchors or scales, and gate or aggregation semantics. A rubric is a specification, not a hierarchy rung.
Criterion card
The atomic specification for one criterion: facet, operational definition, allowed evidence, positive and negative exemplars, boundary and abstention rules, severity (fatal, tradeable, advisory, or escalate), mechanism, blind spots, owner, and version.

Where it comes up

See it in the full glossary →