Skip to content
Benchmaker

Glossary · Meaning and specification

Inverse evaluation design

2Instrument

A proposed research program that treats expert preferences, corrections, escalations, rubric edits, and accept/reject decisions as evidence of a latent judgment and infers its criteria, weights, gates, and thresholds.

Defined in

Handbook · PracticeInverse evaluation designTreat expert accept/reject decisions as evidence about a latent judgment, and use the posterior to find criteria, thresholds and the questions worth asking.
Inverse reinforcement learning (IRL)
Infers which reward functions could explain observed behavior, often as a Bayesian posterior under a Boltzmann (noisily rational) choice model. Many rewards can explain the same behavior, so the inferred reward is only partly identifiable.
Rubric
A structured, human-readable specification with criteria, allowed evidence, exemplar anchors or scales, and gate or aggregation semantics. A rubric is a specification, not a hierarchy rung.

Where it comes up

See it in the full glossary →