Inverse evaluation design

Inverse evaluation design treats what experts do (verdicts, preferences, corrections, escalations, rubric edits, accept/reject decisions) as evidence about a latent judgment, and infers a posterior over the criteria, weights, gates, and thresholds that could explain it. It is a research direction this handbook proposes, built on inverse reinforcement learning, not an established method.

From inverse RL to inverse evaluation#

Forward reinforcement learning asks which policy maximizes a given reward. Inverse reinforcement learning (IRL) runs the other way: given observed behavior, which reward functions could explain it? Each observed decision then counts as evidence about the preferences behind it.

The evaluator-engineering analogue treats expert behavior as evidence about a latent judgment. For DocsAgent, the documentation agent of Case G whose answers subject-matter experts (SMEs) grade, the evidence is the SMEs' accept and reject decisions, the corrections they make, the cases they escalate, and the edits they make to the grounding rubric. Let $z$ parameterize the judgment behind all of it. Then

$$p(z \mid \mathcal{D}) \propto p(\mathcal{D} \mid z)\, p(z)$$

where:

• $z$ parameterizes the latent judgment: which criteria matter, their weights, which failures are gates, the thresholds $\theta_i$, and how criteria interact;
• $\mathcal{D}$ is the record of expert behavior (not the validity domain $D$);
• $p(\mathcal{D} \mid z)$ is a model of how an expert holding judgment $z$ would decide, given in the next section;
• $p(z)$ is the prior: the judgment structures you are willing to consider, and how plausible each is before you look.

In words: keep every candidate account of the experts' judgment, weighted by how well it explains what they actually did and how plausible it was to begin with.

The output is a distribution, not a single best $\hat z$, and its spread shows where the experts' boundary is unclear and where asking them is worth the most. Used this way, inverse evaluation design is a quantitative form of the judgment compiler's second step, eliciting expert judgment before the rubric is written, and the compiler's own advice agrees: boundary disagreements are more informative than obvious examples. Rubrics and criterion cards covers the full compiler.

A model of the expert#

The likelihood needs a model of how experts decide. A common choice is noisy rationality in the Boltzmann form:

$$\Pr[a \mid s, z, \beta] = \frac{\exp\big(\beta\, Q_z(s, a)\big)}{\sum_{a'} \exp\big(\beta\, Q_z(s, a')\big)}$$

where:

• $s$ is the case in front of the expert: an answer, a trajectory, or a pair of candidates;
• $a$ is the expert's decision, such as accept, reject, escalate, or prefer the first;
• $Q_z(s, a)$ is how good decision $a$ is under judgment $z$ (reinforcement learning usually writes the reward as $R$; this handbook reserves $R$ for resolution);
• $\beta \ge 0$ is the expert's consistency: at $\beta = 0$ decisions are random, and as $\beta$ grows the expert almost always takes the decision that $z$ favors.

In words: experts usually choose what their judgment favors, and they err most often when the options are close.

With two options the model becomes a logistic curve in the value gap: $\Pr[\text{accept} \mid s] = \sigma\big(\beta\, \Delta_z(s)\big)$, where $\Delta_z(s) = Q_z(s, \text{accept}) - Q_z(s, \text{reject})$ and $\sigma(u) = 1/(1 + e^{-u})$. A pairwise preference is the same model again. If $s$ is a pair of candidates $y_w$ and $y_l$ for an input $x$, the value of preferring a candidate is its reward $r(x, y)$, and $\beta = 1$, then the probability of preferring $y_w$ is $\sigma\big(r(x, y_w) - r(x, y_l)\big)$: the Bradley–Terry form used to train reward models from human preferences. This is the sense in which pairwise preference learning is IRL-shaped even when no decision process is written down. The reward model in RLHF is an inferred judgment; Evaluators inside training loops covers what happens when a policy is optimized against it.

The same model prices trade-offs. When $Q_z$ is a weighted sum of criterion scores, every pairwise choice the specification owner makes is evidence about the weights. That is how this handbook wants weights set: from the owner's choices between concrete pairs, not typed into a configuration file. The weights are the product decision; elicitation is how it is written down honestly. Aggregation geometry covers what to do with the result, including the case in which no amount of one criterion can buy back a failure on another.

A worked example: where the expert draws the line#

This example is illustrative. The counts are invented for this chapter, and the posterior was computed from them on a grid, with flat priors over $\theta$ from 0 to 4 and $\beta$ from 0.1 to 10.

DocsAgent's grounding criterion has a written card (the full card is in Rubrics and criterion cards). Its pass line requires every material claim to be entailed by a cited span or labeled as uncertain. Its fail line is "any material unsupported product instruction." Suppose an SME reviews 120 DocsAgent answers, accepts or rejects each, and marks every material claim that the cited spans do not support.

Start with the simplest account of the SME's judgment: count the unsupported claims $k$ in an answer and accept when $k$ is below a tolerance $\theta$, so that $\Delta_z(s) = \theta - k$ and $z = (\theta, \beta)$, with three or more claims counted as three. The posterior puts the tolerance at about one claim, with a median $\theta$ of 1.1 (90% credible interval 0.9–1.3) and a median consistency $\beta$ of 2.6 (90% credible interval 1.9–3.5). Its predictions match the counts closely, as the last column shows.

Two findings follow. First, the SME is not applying the card's pass line: under it, no answer with an unsupported claim passes, yet the SME accepted 17 of 30 such answers (57%, 95% Wilson CI 39–73%, n = 30). Second, the boundary is ambiguous exactly there. The prediction for one unsupported claim is close to a coin flip, while answers with none or with two are predictable. A near-even prediction in a well-populated cell is the signal to look for: either the boundary really is fuzzy at one claim, or a criterion the model leaves out is deciding those cases, and more verdicts of the same kind will not tell the two apart. The next step is a question: why were 17 accepted and 13 rejected?

Suppose the SME's reasons point to where the unsupported claim sits. Split that way, the SME accepted 16 of 18 answers whose unsupported claim was a background statement (89%, 95% Wilson CI 67–97%, n = 18) and 1 of 12 whose unsupported claim was a procedural instruction (8%, 95% Wilson CI 1–35%, n = 12). The SME was applying the card's fail line, which targets instructions, and tolerating unsupported background claims, a case the card's two lines leave undecided. The one-feature model fitted the counts and was still wrong about the mechanism: it predicts 57% acceptance for an answer whose one unsupported claim is an instruction. This is the identifiability problem in miniature: many reward functions can explain the same behavior, and a restrictive hypothesis class can become confidently mistaken.

Even the split leaves an ambiguity. The 16 acceptances look the same whether the SME tolerates unsupported background claims or did not notice them. Only another question separates tolerance from inattention: show the SME those answers with the unsupported claim highlighted, and ask again.

Asking the next question#

Cooperative inverse reinforcement learning (CIRL) makes asking part of the method. It casts a person and a machine as partners pursuing the person's objective, of which the machine is unsure, so that optimal play for the pair can include questions, teaching, and actions chosen for the information they reveal (Hadfield-Menell et al., 2016). Carried over to evaluation, the evaluator stops being a passive scorer of finished artifacts: where its posterior is uncertain, it can ask for an ablation, for provenance, or for a rerun, or put a question to a human. The highlighted-claim question in the worked example is such a query.

Inverse reward design adds a second source of evidence: the rubric itself. It reads a hand-written reward as a clue to what its designer actually wanted, interpreted in light of the situations the designer had in view when writing it (Hadfield-Menell et al., 2017). For evaluation, the lesson is that a rubric, or a benchmark score, does not define the construct; it documents what its author was trying to capture. If DocsAgent's card was written with fully grounded answers and answers with unsupported instructions in view, answers whose only unsupported claim is background fall outside its design environment, where the card's text and its author's intent can come apart.

Fit the model for each expert as well as for the pool. A tolerance that differs systematically between SMEs is a disagreement to adjudicate, not noise to average away. LLM judges in practice separates the four causes of rater disagreement: random noise, an underspecified rubric, legitimate plural values, and differences in expertise.

What inverse methods can and cannot do#

Inverse methods can:

• surface criteria that experts apply but never wrote down;
• estimate trade-offs and thresholds;
• propose the structure of a rubric;
• expose disagreements and plural values;
• rank the clarification questions worth asking;
• point to criteria that could later be checked by execution or formal means.

They cannot:

• convert expert behavior into a formal soundness guarantee;
• show that what experts do is what they should do;
• escape the fact that many judgments can explain the same behavior;
• validate the construct from the preference data alone.

The limit fits in one line: IRL can be a judgment compiler, not a proof compiler. Inferred criteria move up the verification hierarchy only later, when someone instruments them and compiles them into stronger checks; that is the work of Verifiability engineering.

The safe pipeline therefore runs in a fixed order:

1. Descriptive preference inference. What do the experts appear to optimize? The output is a posterior over $z$, pooled and per expert.
2. Normative review. Should they optimize it? The specification owner decides which inferred structure becomes policy: in the example, whether an unsupported background claim may pass.
3. Specification. Write the decision into criterion cards, including the boundary rule the old card lacked.
4. Instrumentation. Compile each criterion to the strongest affordable mechanism. For grounding, citation existence can be checked deterministically, which leaves entailment to a calibrated judge under periodic SME audit.
5. Validation. Validate the new evaluator as an instrument on fresh adjudicated cases, not on the decisions it was inferred from.

In practice#

Infer the judgment behind one criterion before you write its card.

1. Collect decisions near the boundary. Show experts real cases and deliberately contrasted pairs, and for each verdict collect the evidence they used, their reasons, the failures they consider unacceptable, and what would have changed their mind.
2. Define the hypothesis space. Turn candidate criteria into measurable features of each case, decide which structures $z$ may take (weights, gates, thresholds, interactions), and write the prior down.
3. Fit a choice model. Compute the posterior over $z$ under a Boltzmann model, pooled and for each expert.
4. Check the fit by slice. Compare predicted and observed decisions in slices the model does not use; a model can fit the totals with the wrong mechanism.
5. Ask where the posterior is uncertain. Take near-even cells back to the experts with the relevant evidence highlighted, and ask why.
6. Review normatively. Have the specification owner decide what the inferred structure should become, and record the decision and its reasons.
7. Specify, instrument, and validate. Write the criterion card, compile what can be checked, and validate the evaluator on held-out adjudicated cases.

The artifact is an inferred-judgment record: the hypothesis space and prior, the posterior pooled and per expert, the questions asked and their answers, and the normative decisions taken, ready to become criterion cards.

Common failure modes#

• Descriptive taken as normative — an inferred tolerance goes into the rubric because it fits the data → route every inferred structure through normative review with a named owner.
• Validating on the inference data — the new evaluator is scored against the same decisions it was inferred from → hold out adjudicated cases, and keep certification labels out of the inference set. See anchor leakage.
• A confident wrong model — one feature fits the counts and hides the criterion that decides the cases → check predictions by slice, and ask experts about the near-even cells.
• Optimizing the inferred judgment — the inferred evaluator becomes a training reward with no adaptive-validity test → treat it as a learned judge under optimization and measure its exploitability curve. See reward hacking.

Check yourself#

Analyze. A specification owner makes 40 pairwise choices between DocsAgent answers that differ in completeness and concision. Write the choice model and its parameters $z$. What does the posterior on the completeness weight say if the owner never prefers a less complete answer, however concise?

Hint

Each choice is logistic in the weighted difference. If the owner never trades completeness away, the posterior on its weight runs to the edge of the prior: evidence for a gate or a lexicographic rule, not a weight.

Evaluate. When should you trust an explicit rubric over preferences inferred from expert behavior?

Hint

Think of sparse decisions near the boundary, inconsistent or mistaken experts, a hypothesis class too narrow for the real criteria, cases unlike those the evaluator will meet, and a rubric encoding a normative commitment, such as a safety gate, that behavior has drifted from.

Create. Design a small experiment to test whether inferring preferences from expert decisions can recover a criterion the experts use but never wrote down.

Hint

Have experts write their rubric first, then collect decisions on contrasted cases that vary one undeclared feature at a time. Count a success only if the posterior singles out the feature and the experts confirm it with the evidence highlighted, which also controls for inattention.

Further reading#

• Rubrics and criterion cards — the judgment compiler whose elicitation step this chapter makes quantitative.
• Aggregation geometry — turning inferred weights and gates into a decision rule.
• Evaluators inside training loops — Bradley–Terry reward models, the inferred judgments that RLHF optimizes against.
• Verifiability engineering — compiling discovered criteria into stronger mechanisms.
• Hadfield-Menell et al. (2016), "Cooperative inverse reinforcement learning" — the partial-information game behind evaluation as information gathering.
• Hadfield-Menell et al. (2017), "Inverse reward design" — a written objective read as evidence about its designer's intent.