Notation and formula card

This card collects the handbook's notation and formulas in one place. The first half fixes the symbols: the five tuples of the evaluator system, the rules for letters that mean different things in different layers, and the conventions for rates and margins. The second half is a formula card grouped by the job each formula does.

Each formula appears in the same form as in the chapter that develops it, with a link to that chapter, the published source where there is one, and the calculator that computes it, if one exists. Calculators live in their chapters, the most used also have pages of their own, and the glossary gives every term in words.

The five tuples#

The evaluator system#

This handbook writes the evaluator system as five layers:

$$\mathcal{E}_{\text{sys}} = (\mathcal{C}, \mathcal{I}, \mathcal{R}, \mathcal{V}, \mathcal{L})$$

where each coordinate is a layer with its own question:

The evaluator system explains the master object. The lifecycle layer has no tuple: its controls are the O/A/P channels, anchor-burn accounting, tripwires, versioning, and frame review $F$ (Living eval suites, Frame review and governance).

The claim#

$$\mathcal{C} = (C, P, E, H, D, \delta)$$

where:

The instrument#

$$\mathcal{I} = \langle \Psi, A, V, M, Z, R, \Gamma \rangle$$

where:

A rubric is the specification in rubric form, $\Psi = \{(c_i, E_i, A_i, G_i)\}_{i=1}^{k}$: criterion, allowed evidence, exemplar anchors or scale, and gate semantics for each of $k$ criteria (Rubrics and criterion cards).

The regime#

$$\mathcal{R} = (\kappa, O, b, S, A_{\text{access}}, K)$$

where:

The regime's three values are measure, optimize, and monitor-control (Use regimes).

The validation profile#

$$\Pi = (N, E, T, G, C, D)$$

where:

Inside $\Pi$, name the exploitability curve, channel coupling, drift profile, and generator coupling in words; the letters collide with the claim's.

Collision rules#

The same letter often carries different meanings in different layers. The rules below keep one meaning per page; where a formula uses a letter locally, its "where" list fixes the meaning.

Conventions#

• Pass rates. $p$ is the true rate on the claim's population, or a base rate; $\hat p$ is an estimate; $p_{\text{obs}}$ is a judge's raw reading; $p_{\text{fail}}$ is a failure probability. A sampled rate always carries its interval and n, as in the illustrative 80% (95% Wilson CI 71–87%, n = 100).
• Error rates. TPR and FPR are written $\mathrm{TPR}$ and $\mathrm{FPR}$ in the sensor formulas and $t$ and $f$ in the selection and alert formulas. Specificity, or TNR, is $1 - \mathrm{FPR}$ (sensitivity and specificity).
• Trajectories. $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$, with states $s_t$ and actions $a_t$ (Evaluating agents).
• Margins. $\rho(\psi, \tau)$ is a signed robustness margin: positive when $\tau$ satisfies $\psi$, negative when it violates it, and its size says by how much (Specification).
• Exploitability. $\varepsilon(b)$ is the evaluator's disagreement with an independent anchor at exposure $b$, and $b^{*}$ is the largest tested exposure before that divergence exceeds a predeclared tolerance (tested exposure). Report $b^{*}$ with the tested range, never as "robust".
• Typography. Multi-letter names are upright, as in $\mathrm{FPR}$ and $\mathrm{CVaR}$; $\Pr[\cdot]$ is a probability, $\mathbb{E}[\cdot]$ an expectation, and $\mathbb{1}[\cdot]$ is 1 when its condition holds and 0 otherwise.

Formula card#

Each entry gives the formula, a "where" list, when it applies, the published source where there is one, the chapter that develops it, and the calculator that computes it, if one exists.

Estimating a rate and its interval#

Target and observed pass rate#

$$p = \mathbb{E}_{x \sim P}\big[\psi(S(x))\big], \qquad \hat p = \frac{1}{n}\sum_{i=1}^{n} y_i$$

where:

• $S(x)$ is the system's behavior on task $x$, drawn from the claim's population $P$;
• $y_i \in \{0, 1\}$ is the outcome of task $i$ among $n$ independent tasks.

$\hat p$ estimates the never-observed $p$ only when the tasks represent $P$; a suite drawn from another distribution $Q$ estimates $\mathbb{E}_{x \sim Q}[\psi(S(x))]$ instead (Task distributions and dataset engineering).

Chapter: Estimands, pass rates, and intervals

Wilson interval#

$$\begin{aligned} \text{center} &= \frac{\hat p + z^2/(2n)}{1 + z^2/n} \\ \text{half-width} &= \frac{z}{1 + z^2/n}\sqrt{\frac{\hat p(1 - \hat p)}{n} + \frac{z^2}{4n^2}} \end{aligned}$$

where:

• $\hat p$ is the observed pass rate on $n$ independent tasks;
• $z$ is the standard normal quantile, 1.96 for 95%.

The Wilson interval is the default for independent binary outcomes at moderate n, and unlike the Wald form it keeps a nonzero width when every task passes or every task fails; when trials repeat within tasks, resample whole tasks instead.

Chapter: Estimands, pass rates, and intervals · Calculator: Pass-rate intervals and sample size

Beta-binomial posterior#

$$\begin{aligned} p &\sim \mathrm{Beta}(\alpha, \beta) \\ p \mid y &\sim \mathrm{Beta}(\alpha + s,\ \beta + n - s) \end{aligned}$$

where:

• $\alpha$ and $\beta$ set the prior, and $\alpha = \beta = 1$ is uniform;
• $s$ is the number of passes among $n$ tasks.

Use the Beta-binomial posterior, which states its prior openly and assumes independent tasks, when the release question is itself a probability, such as $\Pr[p > 0.95]$.

Chapter: Estimands, pass rates, and intervals

Stratified estimate#

$$\begin{aligned} \hat p_{\text{strat}} &= \sum_h w_h\, \hat p_h \\ \mathrm{SE}(\hat p_{\text{strat}}) &= \sqrt{\sum_h w_h^{2}\, \frac{\hat p_h (1 - \hat p_h)}{n_h}} \end{aligned}$$

where:

• $h$ indexes the strata, and $w_h$ is stratum $h$'s share of the claim's population, with the shares summing to 1;
• $\hat p_h$ and $n_h$ are the stratum's observed pass rate and number of tasks.

The stratified estimate reweights a suite that over-samples some slices on purpose, with weights taken from the claim's population; no weighting reaches a stratum the suite never sampled.

Chapter: Estimands, pass rates, and intervals

Kish effective sample size#

$$n_{\text{eff}} = \frac{\left(\sum_i w_i\right)^2}{\sum_i w_i^2}$$

where $w_i$ is the weight on item $i$.

The Kish effective sample size prices any reweighting as the number of equally weighted items that would carry the same information; when it falls to a small fraction of n, the suite's mix is far from the population's.

Chapter: Estimands, pass rates, and intervals

Covariate-shift weight#

$$w(x) = \frac{P(x)}{Q(x)}$$

where:

• $P(x)$ is the probability of task $x$ under the claim's population;
• $Q(x)$ is its probability under the distribution the suite actually samples.

The covariate-shift weight works only where $Q$ puts probability wherever $P$ does, because weighting cannot repair missing support.

Chapter: Estimands, pass rates, and intervals

Correcting a noisy judge#

Judge sensor model#

$$p_{\text{obs}} = p\,\mathrm{TPR} + (1 - p)\,\mathrm{FPR}$$

where:

• $p$ is the system's true pass rate, relative to the specification;
• $\mathrm{TPR}$ and $\mathrm{FPR}$ are the judge's error rates on this system's outputs, measured against adjudicated labels;
• $p_{\text{obs}}$ is the pass rate the judge reports.

The model applies to binary verdicts, as in estimating prevalence from an imperfect screening test, and its error rates belong to the outputs they were measured on: a system optimized against the judge can have different ones, which is differential misclassification.

Chapter: The judge as a noisy sensor · Calculator: Correcting a noisy judge

Rogan–Gladen correction#

$$\hat p = \frac{p_{\text{obs}} - \mathrm{FPR}}{\mathrm{TPR} - \mathrm{FPR}}$$

where $\hat p$ is the corrected estimate of the true pass rate, and the other symbols are as in the sensor model.

The Rogan–Gladen correction (Rogan & Gladen, 1978) applies when the error rates transport to the eval set and TPR ≠ FPR; clip a result outside 0 to 1 and treat the clipping as evidence that they do not transport, and note that no correction rescues a rubric that operationalizes the wrong construct.

Chapter: The judge as a noisy sensor · Calculator: Correcting a noisy judge

Variance of the corrected rate#

$$\begin{aligned} \operatorname{Var}(\hat p) \approx \frac{1}{J^{2}} \Big[\, & \frac{p_{\text{obs}}(1 - p_{\text{obs}})}{n} \\ &+ \hat p^{2} \operatorname{Var}(\mathrm{TPR}) \\ &+ (1 - \hat p)^{2} \operatorname{Var}(\mathrm{FPR}) \,\Big] \end{aligned}$$

where:

• $J = \mathrm{TPR} - \mathrm{FPR}$ is the judge's separation between true passes and true failures;
• $n$ is the number of eval items the judge graded;
• $\operatorname{Var}(\mathrm{TPR})$ and $\operatorname{Var}(\mathrm{FPR})$ are the sampling variances of the calibration estimates.

This first-order approximation sizes the problem but is not a release-grade interval: it blows up as $J \to 0$, and decision-grade intervals on corrected rates need a construction built for the purpose, such as the one by Lang and Reiczigel (2014).

Chapter: The judge as a noisy sensor

Selection and alert precision#

Benign selection model#

$$\Pr[C \mid +] = \frac{p\,t}{p\,t + (1-p)\,f}$$

where:

• $C$ is the event that a candidate is correct, and $+$ the event that the evaluator flags it;
• $p$ is the share of candidates that are correct;
• $t$ and $f$ are the evaluator's TPR and FPR on these candidates.

The formula gives the precision of best-of-N selection through a binary evaluator as a model, not a theorem, assuming independent candidates, values of $t$ and $f$ that hold for every candidate at every N, and a uniform choice among flagged candidates; N does not appear in it, so more samples cannot raise the ceiling.

Chapter: Adaptive validity · Calculator: Best-of-N with an imperfect verifier

Winner correctness under top-score selection#

$$\Pr[C_{\text{winner}}] = N \int p\, f_1(s)\, G(s)^{N-1}\, ds$$

where:

• $F_1$ and $F_0$ are the score distributions of correct and incorrect candidates, and $f_1$ is the density of correct scores;
• $G(s) = p\,F_1(s) + (1-p)\,F_0(s)$ is the chance that one other candidate scores below $s$;
• $N$ is the number of candidates.

This stylized expression applies when the selector keeps the highest score; the result depends on the extreme tail of the score distributions, so precision can fall, rise, or plateau as N grows, and ordinary calibration does not answer the question.

Chapter: Adaptive validity · Calculator: Best-of-N with an imperfect verifier, with continuous scores

Alert precision#

$$\Pr[\text{danger} \mid \text{alert}] = \frac{p\,t}{p\,t + (1 - p)\,f}$$

where:

• $p$ is the base rate of truly dangerous events;
• $t$ is the monitor's true-positive rate on dangerous events, and $f$ its false-positive rate on harmless ones.

Alert precision is the selection arithmetic at a production base rate, where even a small false-positive rate can dominate the alerts when $p$ is tiny, so validate a monitor at that base rate, not on a balanced test set.

Chapter: Monitoring and control in production · Calculator: Alert precision at production base rates

Conformal flag threshold#

$$\theta = s_{(k)}, \qquad k = \lfloor (n + 1)\,\alpha \rfloor$$

where:

• $s_{(1)} \le \dots \le s_{(n)}$ are the monitor's badness scores on $n$ labeled true failures, sorted;
• $\alpha$ is the target miss rate, and events that score at or above $\theta$ are flagged.

Under exchangeability, at most about $\alpha$ of future true failures score below the threshold, but the guarantee of conformal prediction is marginal, not per slice, breaks under drift, and says nothing about false positives.

Source: Kochenderfer et al. (2026) · Chapter: Monitoring and control in production

pass@k and pass^k#

Independent pass@k and pass^k#

$$\text{pass@}k = 1 - (1 - p)^k, \qquad \text{pass}^k = p^k$$

where:

• $p$ is the per-attempt success probability;
• $k$ is the number of attempts.

pass@k is the probability that at least one of k attempts succeeds and pass^k the probability that all k succeed, both idealized for independent attempts at one rate, so compute them per task and then average when difficulty varies. The Berkeley slides label the all-k quantity "pass@k" (Song, 2025, slides 55, 57, and 61); read it as pass^k.

Chapter: Estimands, pass rates, and intervals · Calculator: pass@k versus pass^k

Unbiased pass@k estimator#

$$\widehat{\text{pass@}k} = 1 - \binom{n - c}{k} \Big/ \binom{n}{k}$$

where:

• $n$ is the number of samples drawn for one task, at least $k$;
• $c$ is the number of those samples that pass.

The unbiased estimator of Chen et al. (2021) is computed per task, averaged across tasks, and given an interval by resampling tasks; the plug-in $1 - (1 - \hat p)^k$ reads low on average.

Source: Chen et al. (2021) · Chapter: Estimands, pass rates, and intervals · Calculator: pass@k versus pass^k

Comparing two systems#

McNemar's test#

$$\chi^2 = \frac{(b - c)^2}{b + c}$$

where:

• $b$ is the number of tasks that system A passes and system B fails;
• $c$ is the number of tasks that B passes and A fails.

McNemar's test compares two systems on the same tasks through the discordant pairs, and under equal performance its statistic is approximately chi-square with one degree of freedom; use the exact binomial test when $b + c$ is small, and report the effect, $(b - c)/N$ on $N$ paired tasks, with its interval, not only a p-value.

Chapter: Comparing systems · Calculator: Comparing two systems on the same tasks

Paired-comparison sample size#

$$N \approx \frac{\left(z_{1-\alpha/2} + z_{1-\beta}\right)^2 d}{\delta^2}$$

where:

• $N$ is the number of paired tasks;
• $\delta$ is the smallest true difference in pass rates that would change the decision;
• $d$ is the expected share of tasks on which the two systems disagree;
• $z_{1-\alpha/2}$ and $z_{1-\beta}$ are standard normal quantiles for the significance level and the power $1 - \beta$.

This is a rough sizing rule from McNemar's normal approximation, slightly optimistic against the exact test. It sizes a comparison for the smallest difference that would change the decision; the chapter derives it.

Chapter: Comparing systems

Family-wise false alarms#

$$\Pr[\text{at least one false alarm}] = 1 - (1 - \alpha)^m$$

where:

• $m$ is the number of gates;
• $\alpha$ is the significance level of each gate.

The formula assumes independent gates, and $m\alpha$ bounds the family-wise false-alarm rate under any dependence; pre-register the gates or correct for the family, for example by testing each gate at $\alpha/m$.

Chapter: Comparing systems · Calculator: Comparing two systems on the same tasks, lower panel

Rare events#

Trials a rare failure needs#

Under naive sampling from the claim's population, seeing even one failure at rate $p$ takes about $1/p$ trials, and estimating $p$ to within about 10% of its value takes about $100/p$ (Kochenderfer et al., 2026).

Chapter: Rare-event assurance

Rule of three#

$$p_{\text{fail}} \lesssim \frac{3}{n}$$

where:

• $p_{\text{fail}}$ is the true probability that one trial fails;
• $n$ is the number of independent, representative trials, none of which failed.

The rule of three is a rough 95% upper bound, exactly $1 - 0.05^{1/n}$; its Bayesian form updates a uniform prior to $\mathrm{Beta}(1, n + 1)$, with 95% upper bound $1 - 0.05^{1/(n+1)} \approx 3/(n+1)$, and extends to $s$ failures as $\mathrm{Beta}(1 + s, 1 + n - s)$. Clustered or correlated trials make every one of these bounds optimistic.

Chapter: Rare-event assurance · Calculator: What zero failures buys you

Importance sampling#

$$\begin{aligned} P_f &= \mathbb{E}_{q}\!\left[\frac{p(x)}{q(x)}\, \mathbb{1}\{\text{fail}(x)\}\right] \\ &\approx \frac{1}{N} \sum_{i=1}^{N} \frac{p(x_i)}{q(x_i)}\, \mathbb{1}\{\text{fail}(x_i)\} \end{aligned}$$

where:

• $P_f$ is the failure probability under the nominal distribution;
• $p(x)$ is the probability of task $x$ under the nominal distribution, not a pass rate;
• $q(x)$ is its probability under the proposal from which the $N$ tasks $x_i$ are drawn.

Importance sampling samples where failures are common and reweights each outcome by its likelihood ratio, which keeps the estimate unbiased for the nominal distribution only if the proposal covers the whole failure region; a poorly matched proposal makes the variance blow up.

Source: Kochenderfer et al. (2026) · Chapter: Rare-event assurance

Multilevel splitting#

$$p_{\text{fail}} = \prod_{\ell=1}^{L} \Pr\big[f \le \gamma_\ell \mid f \le \gamma_{\ell-1}\big]$$

where:

• $f$ is a robustness-style objective, with failure exactly when $f \le 0$;
• $\gamma_1 > \gamma_2 > \dots > \gamma_L = 0$ are the thresholds, with $\gamma_0$ above every outcome.

Multilevel splitting writes a rare probability as a product of common conditional ones, each estimated among the trials that reached the previous level: the honest form of a staged red-team funnel, whose end count is never a rate from uniform sampling.

Source: Kochenderfer et al. (2026) · Chapter: Rare-event assurance

Falsification objective#

$$\min_{x,\,\delta}\ \rho\big(\psi,\ S(x;\delta)\big)$$

where:

• $x$ is an input and $\delta$ a disturbance, such as a paraphrase, a tool failure, or an adversarial suffix;
• $S(x;\delta)$ is the trajectory the system produces, and $\rho$ its robustness margin.

Falsification searches for failures rather than counting them: any negative value is a counterexample, the search needs a slope that a pass/fail grader does not give, and a failure found by search says nothing about how often it occurs.

Source: Kochenderfer et al. (2026) · Chapter: Falsification and failure regions

Most likely failure#

$$\begin{aligned} \tau^{*} = {} & \arg\max_{\tau}\ \log p_{0}(\tau) \\ & \text{subject to}\ \ \rho(\psi, \tau) < 0 \end{aligned}$$

where:

• $p_0$ is the nominal distribution over trajectories, the traffic production actually serves;
• $\tau^{*}$ is the violating trajectory that distribution is most likely to produce.

Weight the search by plausibility when nature is the adversary; drop the weight against a strategic adversary, who chooses the input.

Source: Kochenderfer et al. (2026) · Chapter: Falsification and failure regions

Aggregation#

Five geometries#

For criterion-level signals $z = (z_1, \dots, z_k)$, the common geometries are:

where $w_i$ are weights, $\theta_i$ thresholds, $L$ the per-task loss, and $\lambda$ a tolerance. Each geometry is a normative claim about compensability, not a statistical detail. This handbook writes the aggregate as $\Gamma(z)$, the thresholds as $\theta_i$, and the tolerance as $\lambda$, keeping $S$ and $\tau$ for the modification surface and the trajectory.

Chapter: Aggregation geometry · Calculator: Aggregation geometry

VaR and CVaR#

$$\mathrm{CVaR}_\alpha(L) = \frac{1}{m} \sum_{j=1}^{m} L_{(j)}$$

where:

• $L_{(1)} \ge L_{(2)} \ge \dots \ge L_{(n)}$ are the task losses, sorted from worst to best;
• $m = \lceil (1 - \alpha)\, n \rceil$ is the number of tasks in the tail;
• $\mathrm{VaR}_\alpha(L) = F_L^{-1}(\alpha)$ is the $\alpha$-quantile of the loss: $\mathrm{VaR}_{0.95}$ of latency is its p95.

CVaR, the average severity of the worst $1 - \alpha$ share, sees how bad the tail is and not only where it starts, so gate the CVaR of a harm score rather than its mean, remembering that with few tasks the tail rests on a handful of observations.

Source: Kochenderfer et al. (2026) · Chapter: Aggregation geometry · Calculator: Aggregation geometry

Reliability#

Cohen's κ#

$$\text{Cohen's }\kappa = \frac{p_o - p_e}{1 - p_e}$$

where $p_o$ is the observed agreement between two scorers and $p_e$ the agreement expected by chance.

Cohen's κ depends on prevalence, so the same scorer posts very different values at different base rates; report the confusion matrix, TPR and TNR, and the calibration set's prevalence with it, never the coefficient alone.

Chapter: The judge as a noisy sensor

Variance components#

$$\begin{aligned} Y_{mtrje} = {} & \mu + \alpha_m + u_t + v_r + w_j + q_e \\ & + \text{interactions} + \varepsilon \end{aligned}$$

where:

• $Y_{mtrje}$ is the score of system $m$ on task $t$, in rollout $r$, graded by judge $j$, in environment snapshot $e$;
• $\mu$ is the grand mean, $\alpha_m$ the system's effect, and $u_t$, $v_r$, $w_j$, $q_e$ the task, rollout, judge, and snapshot effects;
• $\varepsilon$ is the residual.

Generalizability theory decomposes a score's variability by facet, with a mixed-effects model estimating each share, and the claim decides which facets count as error.

Source: Brennan (2001) · Chapter: Reliability and generalizability

Generalizability coefficient#

$$\Phi = \frac{\sigma^2_{\text{object}}}{\sigma^2_{\text{object}} + \sigma^2_{\Delta}}$$

where:

• $\sigma^2_{\text{object}}$ is the variance attributable to the object of measurement, such as the systems compared;
• $\sigma^2_{\Delta}$ collects the variance from sources that should not change the interpretation.

The absolute coefficient is the share of reported-score variance that comes from real differences between systems, and the design question it forces matters more than the number: which sources of variation are legitimate, and how many replications of each make the decision stable.

Source: Brennan (2001) · Chapter: Reliability and generalizability

Variance budget for a pass rate#

$$\begin{aligned} \operatorname{Var}(\hat p) = {} & \frac{\sigma^2_q}{n_q} + \frac{\sigma^2_d}{n_d} \\ & + \frac{\sigma^2_r}{n_q n_d n_r} + \frac{\sigma^2_j}{n_q n_d n_r n_j} \end{aligned}$$

where:

• $\sigma^2_q$, $\sigma^2_d$, $\sigma^2_r$, and $\sigma^2_j$ are the variances contributed by one query, one documentation version, one rollout, and one judge call;
• $n_q$, $n_d$, $n_r$, and $n_j$ are how many of each the design samples.

Each source's variance shrinks only with draws of that source, ignoring the query-by-version interaction; the chapter derives this variance budget for its DocsAgent design.

Chapter: Reliability and generalizability

Exploitability and independence#

Exploitability curve and tested exposure#

$$\varepsilon(b) = \Pr\big[\,\text{evaluator} \neq \text{anchor} \mid Q_b\,\big]$$

where:

• $b$ is the exposure budget, in the optimizer's own unit: N for best-of-N, KL for regularized RL, iterations for developers;
• $Q_b$ is the distribution of outputs after optimization at exposure $b$;
• "evaluator ≠ anchor" means disagreement with an independent reference anchor the optimizer can neither see nor modify.

The exploitability curve traces $\varepsilon(b)$ with the actual optimizer, each point an estimate with its own n and interval, and $b^{*}$ is the largest tested exposure before divergence exceeds a predeclared tolerance; record where possible the induced shift $D(Q_b \Vert P_0)$ from the starting distribution $P_0$, and claim nothing beyond the tested range.

Chapter: Adaptive validity · Calculator: Exploitability under optimization

Generator coupling#

$$\Pr[E \text{ errs} \mid G \text{ errs}] \;\;\text{vs.}\;\; \Pr[E \text{ errs}]$$

where:

• $E$ is the evaluator and $G$ the generator whose outputs it judges;
• "errs" is measured against adjudicated labels.

If the first term rises well above the second, the evaluator tends to be wrong where the generator is wrong and is not an independent corrective channel; self-confirmation needs no adversary, only this correlated bias.

Chapter: Goodhart pressure and self-confirming loops

Channel coupling#

$$\Pr[\text{A misses} \mid \text{O misses}] \quad \text{versus} \quad \Pr[\text{A misses}]$$

where a miss is a true failure, confirmed against the trust root, that a channel passes; A is the anchor channel and O the optimized channel.

The framework asks for joint miss rates wherever they can be estimated, and the chapter reads them in the generator-coupling form: the further the first term rises above the second, the more the channels' blind spots line up.

Chapter: Channels, independence, and anchor burn

Training signals#

Preference reward model#

$$\begin{aligned} \Pr[y_w \succ y_l \mid x] &= \sigma(r_\phi(x, y_w) - r_\phi(x, y_l)) \\ \mathcal{L}_{\text{RM}}(\phi) &= -\mathbb{E}\big[\log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)\big] \end{aligned}$$

where:

• $r_\phi(x, y)$ is the reward model's score for response $y$ to prompt $x$;
• $y_w \succ y_l$ means the labeler preferred $y_w$ to $y_l$;
• $\sigma(z) = 1/(1 + e^{-z})$ is the logistic function.

The Bradley–Terry model learns an ordering from score differences, not a proof of quality, so a reward model's absolute values carry no meaning of their own, and its validity domain is empirical.

Chapter: Evaluators inside training loops

KL-regularized objective#

$$\begin{aligned} \max_\theta\; & \mathbb{E}_{x,\, y \sim \pi_\theta}\big[r_\phi(x, y)\big] \\ & - \beta\, \mathbb{E}_x\Big[D_{\mathrm{KL}}\big(\pi_\theta(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x)\big)\Big] \end{aligned}$$

where:

• $\pi_\theta$ is the policy being trained and $\pi_{\text{ref}}$ a fixed reference policy;
• $\beta$ sets the price of divergence from the reference.

The KL-regularized objective is the standard schematic of RLHF's policy step, and its KL term is an exposure control that bounds how far the policy may travel without certifying that the reward stays valid along the way.

Chapter: Evaluators inside training loops

Verifiable reward#

$$r_{\text{RLVR}}(x, y) = \mathbb{1}\big[V_\psi(x, y) = 1\big]$$

where $V_\psi(x, y)$ is the verdict of a verifier for specification $\psi$ on response $y$ to task $x$.

RLVR grounds the reward in an external check, but grounded is not complete: a test suite samples behavior, and a final-answer check can reward a flawed derivation that reaches the right answer.

Chapter: Evaluators inside training loops

Step scores and rollout values#

$$\hat r_t = f_\phi(s_t, a_t, \text{ctx}), \qquad \hat V^{\pi}(s_t) = \frac{1}{K} \sum_{j=1}^{K} y_j$$

where:

• $\hat r_t$ is a learned PRM's score for step $t$, with state $s_t$, action $a_t$, and context ctx;
• $K$ is the number of completions sampled from the current policy $\pi$ starting at $s_t$, and $y_j$ is 1 if completion $j$ succeeds.

A rollout label estimates a policy-conditioned value that changes whenever the policy does, not a judgment of the step: a PRM score, a value estimate, and a robustness margin $\rho(\psi, \tau)$ are three different objects.

Chapter: Signals and resolution

Inverse evaluation design#

Posterior over judgments#

$$p(z \mid \mathcal{D}) \propto p(\mathcal{D} \mid z)\, p(z)$$

where:

• $z$ parameterizes the latent judgment: criteria, weights, gates, thresholds, and interactions;
• $\mathcal{D}$ is the record of expert behavior, not the validity domain $D$.

Inverse evaluation design, this handbook's own synthesis, keeps a distribution over candidate judgments whose spread shows where asking experts is worth most, but it cannot deliver formal soundness or validate the construct from inside the preference data.

Chapter: Inverse evaluation design

Boltzmann choice model#

$$\Pr[a \mid s, z, \beta] = \frac{\exp\big(\beta\, Q_z(s, a)\big)}{\sum_{a'} \exp\big(\beta\, Q_z(s, a')\big)}$$

where:

• $s$ is the case in front of the expert and $a$ the expert's decision;
• $Q_z(s, a)$ is how good decision $a$ is under judgment $z$;
• $\beta \ge 0$ is the expert's consistency.

Experts usually choose what their judgment favors and err most when options are close, and with two options and $\beta = 1$ a pairwise preference reduces to the Bradley–Terry form. The value is written $Q_z$ because $R$ is resolution.

Chapter: Inverse evaluation design

Other expressions#

These expressions name relations rather than compute quantities: