Glossary · Evaluator mechanisms
Structural guarantee
A property enforced by the system's construction (permissions, sandboxes, verified scaffold rules, or an output grammar), so a failure is unreachable rather than rare. It holds only for its configured trusted base.
Defined in
Handbook · PracticeGuarantee before you estimateMake bad states unreachable, verify the scaffold you control, and spend samples only on what construction can't reach.Related terms
- Capability removal
- Making an unsafe state unreachable by construction, with read-only credentials, egress allowlists, spend caps, or sandbox walls, so its failure probability is zero as a property of the configured system rather than an estimate.
- Constrained decoding
- Grammar-constrained generation that guarantees every output belongs to a formal language: every emission parses and every tool call is schema-valid. A genuine guarantee about syntax, never semantics.
- Subtraction principle
- Every property enforced by construction or formal verification is removed from the sampling burden, so scarce empirical evaluation budget is spent only on the properties that cannot be guaranteed.
- Configured trusted base
- Everything a formal guarantee is relative to: permitted axioms, theorem statements, checker version, kernel configuration, environment assumptions, and whether the optimizer can modify any of them. The Goodhart boundary moves here rather than disappearing.
Where it comes up
- Monitoring and control in productionHandbook · Regime
- CrosswalkHandbook · Reference