Skip to content
Benchmaker
Part 3 · The five layers
  1. 1. Claim
  2. 2. Instrument
  3. 3. Regime (current)
  4. 4. Validation
  5. 5. Lifecycle

Regime

The same instrument means different things in different causal loops.

Coordinates

κConsumerOOptimizerbExposure budgetSModification surfaceA_accessInformation accessKCost/latency envelope
  1. 3.1Use regimes: same judge, four jobsInteractiveMeasure, optimize or monitor: an evaluator changes meaning when its signal enters a different loop, and signals migrate between loops.9 min read
  2. 3.2Exposure, modification, and accessWho consumes the signal, who optimizes against it, how hard, what they can change, and what they can see.6 min read
  3. 3.3Evaluators inside training loopsInteractiveRLHF is a pipeline, not an evaluator. Where the learned judge sits in RLHF, RLAIF, RLVR and DPO, and what it is exposed to.10 min read
  4. 3.4Monitoring and control in productionInteractiveMonitoring is not continuous benchmarking: base rates, calibration, conformal caveats, and the moment a monitor becomes a training signal.8 min read
Start with Use regimes