Glossary · Evaluator mechanisms
Verification hierarchy
Formal verification > execution feedback > learned judges > intrinsic signals: a prior over expected grounding strength, not a verdict on validity. Human judgment sits outside the ladder as the external frontier and trust root.
Defined in
Handbook · FoundationsThe verification hierarchy, demotedFormal > execution > learned judges > intrinsic signals. A useful prior on how grounded a signal is, never a verdict on validity.Related terms
- Reliability–coverage trade-off
- Signal reliability rises toward the top of the verification hierarchy while task coverage widens toward the bottom. The response is to decompose the judgment and push each criterion as high as it can legitimately go.
- Trust root (reference authority)
- Where the evaluate-the-evaluator chain stops: a configured proof kernel, or an expert panel with measured reliability and a governance mandate.
- Validation profile
- The measured profile Π = (N, E, T, G, C, D): natural error, exploitability, transport, generator coupling, channel coupling, drift. It replaces hierarchy labels with measurements.
Where it comes up
- Rubrics and criterion cardsHandbook · Instrument
- The validation profile and validity statementHandbook · Validity under pressure
- Self-improvement and evaluator co-evolutionHandbook · Agents and self-improvement
- Verifiability engineeringHandbook · Practice
- Inverse evaluation designHandbook · Practice