# Notation and formula card

> Every symbol and formula, plus the rules that stop R, V, κ and A from meaning two things on one page.

This card collects the handbook's notation and formulas in one place. The first half fixes the symbols: the five tuples of the evaluator system, the rules for letters that mean different things in different layers, and the conventions for rates and margins. The second half is a formula card grouped by the job each formula does.

Each formula appears in the same form as in the chapter that develops it, with a link to that chapter, the published source where there is one, and the calculator that computes it, if one exists. Calculators live in their chapters, the most used also have [pages of their own](https://benchmaker.ai/tools), and the [glossary](https://benchmaker.ai/handbook/reference/glossary) gives every term in words.

## The five tuples

### The evaluator system

This handbook writes the [evaluator system](https://benchmaker.ai/glossary/evaluator-system) as five layers:

$$
\mathcal{E}_{\text{sys}} = (\mathcal{C}, \mathcal{I}, \mathcal{R}, \mathcal{V}, \mathcal{L})
$$

where each coordinate is a layer with its own question:

| Symbol | Layer | The question it answers | Home |
|---|---|---|---|
| $\mathcal{C}$ | Claim | What property, over what world, for what decision? | [Constructs, facets, and nuisances](https://benchmaker.ai/handbook/claim/constructs) |
| $\mathcal{I}$ | Instrument | How is the judgment produced? | [Specification](https://benchmaker.ai/handbook/instrument/specification) |
| $\mathcal{R}$ | Regime | What causal role does the signal play, and what happens when the system adapts to it? | [Exposure, modification, and access](https://benchmaker.ai/handbook/regime/exposure-and-surfaces) |
| $\mathcal{V}$ | Validation | Why believe the signal means what we claim, before and after adaptation? | [The validation profile](https://benchmaker.ai/handbook/pressure/validation-profile) |
| $\mathcal{L}$ | Lifecycle | Does validity survive over time, and are we still evaluating the right thing? | [Channels and independence](https://benchmaker.ai/handbook/lifecycle/channels-and-independence) |

[The evaluator system](https://benchmaker.ai/handbook/foundations/evaluator-systems) explains the master object. The lifecycle layer has no tuple: its controls are the O/A/P channels, anchor-burn accounting, tripwires, versioning, and frame review $F$ ([Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites), [Frame review and governance](https://benchmaker.ai/handbook/lifecycle/frame-review)).

### The claim

$$
\mathcal{C} = (C, P, E, H, D, \delta)
$$

where:

| Symbol | Name | Meaning | Home |
|---|---|---|---|
| $C$ | [Construct](https://benchmaker.ai/glossary/construct) | The latent property the evaluation intends to measure | [Constructs](https://benchmaker.ai/handbook/claim/constructs) |
| $F^{+}$, $F^{-}$ | [Facets](https://benchmaker.ai/glossary/construct-facet) and [nuisances](https://benchmaker.ai/glossary/nuisance-variable) | What the score should track, and what it must not respond to | [Constructs](https://benchmaker.ai/handbook/claim/constructs) |
| $P$ | [Population](https://benchmaker.ai/glossary/population) | The task, user, or input distribution the claim is about | [Validity domain](https://benchmaker.ai/handbook/claim/validity-domain) |
| $E$ | [Environment](https://benchmaker.ai/glossary/environment) | External state, tools, and other actors | [Validity domain](https://benchmaker.ai/handbook/claim/validity-domain) |
| $H$ | [Harness](https://benchmaker.ai/glossary/harness) | Prompts, tools, memory, orchestration, and stopping logic around the model | [Validity domain](https://benchmaker.ai/handbook/claim/validity-domain) |
| $D$ | [Validity domain](https://benchmaker.ai/glossary/validity-domain) | The declared conditions under which the interpretation is claimed to hold | [Validity domain](https://benchmaker.ai/handbook/claim/validity-domain) |
| $\delta$ | [Decision](https://benchmaker.ai/glossary/decision-claim) | The decision the evidence is meant to change | [Decision claims](https://benchmaker.ai/handbook/claim/decision-claims) |

### The instrument

$$
\mathcal{I} = \langle \Psi, A, V, M, Z, R, \Gamma \rangle
$$

where:

| Symbol | Name | Meaning | Home |
|---|---|---|---|
| $\Psi$ | [Specification](https://benchmaker.ai/glossary/specification) | The operational statement of what counts as success | [Specification](https://benchmaker.ai/handbook/instrument/specification) |
| $\psi$ | Predicate | The specification read as a test: $\psi(y) = 1$ when $y$ satisfies it | [Specification](https://benchmaker.ai/handbook/instrument/specification) |
| $A$ | [Anchor](https://benchmaker.ai/glossary/anchor) | What the comparison is rooted in | [Anchors and evidence views](https://benchmaker.ai/handbook/instrument/anchors-and-evidence-views) |
| $V$ | [Evidence view](https://benchmaker.ai/glossary/evidence-view) | What the evaluator may observe | [Anchors and evidence views](https://benchmaker.ai/handbook/instrument/anchors-and-evidence-views) |
| $M$ | [Mechanism](https://benchmaker.ai/glossary/mechanism) | What computes the judgment | [Mechanism families](https://benchmaker.ai/handbook/instrument/mechanisms) |
| $Z$ | [Signal](https://benchmaker.ai/glossary/signal) | What leaves the evaluator: verdict, score, preference, vector, critique | [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution) |
| $R$ | [Resolution](https://benchmaker.ai/glossary/resolution) | The unit judged: step, action, artifact, trajectory, or rate | [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution) |
| $\Gamma$ | [Aggregation geometry](https://benchmaker.ai/glossary/aggregation-geometry) | How criterion-level judgments become a decision, with thresholds $\theta_i$ | [Aggregation geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry) |

A rubric is the specification in rubric form, $\Psi = \{(c_i, E_i, A_i, G_i)\}_{i=1}^{k}$: criterion, allowed evidence, exemplar anchors or scale, and gate semantics for each of $k$ criteria ([Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards)).

### The regime

$$
\mathcal{R} = (\kappa, O, b, S, A_{\text{access}}, K)
$$

where:

| Symbol | Name | Meaning | Home |
|---|---|---|---|
| $\kappa$ | [Consumer](https://benchmaker.ai/glossary/consumer) | Who or what acts on the signal | [Exposure and surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces) |
| $O$ | [Optimizer](https://benchmaker.ai/glossary/optimizer) | Who adapts against the signal | [Exposure and surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces) |
| $b$ | [Exposure budget](https://benchmaker.ai/glossary/exposure-budget) | How much optimization pressure, in the optimizer's own unit | [Exposure and surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces) |
| $S$ | [Modification surface](https://benchmaker.ai/glossary/modification-surface) | Everything the optimizing loop can change | [Exposure and surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces) |
| $A_{\text{access}}$ | [Information access](https://benchmaker.ai/glossary/information-access) | What flows from the evaluator or anchor back to the optimizer | [Exposure and surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces) |
| $K$ | [Cost and latency envelope](https://benchmaker.ai/glossary/cost-latency-envelope) | What each evaluator call may cost, and how fast it must answer | [Exposure and surfaces](https://benchmaker.ai/handbook/regime/exposure-and-surfaces) |

The regime's three values are measure, optimize, and monitor-control ([Use regimes](https://benchmaker.ai/handbook/regime/use-regimes)).

### The validation profile

$$
\Pi = (N, E, T, G, C, D)
$$

where:

| Symbol | Name | Meaning | Home |
|---|---|---|---|
| $N$ | Natural error | Confusion matrices, calibration, and slice error on the target distribution | [The validation profile](https://benchmaker.ai/handbook/pressure/validation-profile) |
| $E$ | [Exploitability curve](https://benchmaker.ai/glossary/exploitability-curve) | Evaluator–anchor divergence as optimization exposure grows | [Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity) |
| $T$ | [Transport](https://benchmaker.ai/glossary/transport) | Whether error rates carry over across generators, slices, time, and environments | [The validation profile](https://benchmaker.ai/handbook/pressure/validation-profile) |
| $G$ | [Generator coupling](https://benchmaker.ai/glossary/generator-coupling) | Whether the evaluator errs where the generator errs | [Goodhart and self-confirming loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops) |
| $C$ | [Channel coupling](https://benchmaker.ai/glossary/channel-coupling) | Dependence among supposedly independent assurance channels | [Channels and independence](https://benchmaker.ai/handbook/lifecycle/channels-and-independence) |
| $D$ | [Drift](https://benchmaker.ai/glossary/drift) and saturation | How calibration and discrimination degrade, and when the range is exhausted | [Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites) |

Inside $\Pi$, name the exploitability curve, channel coupling, drift profile, and generator coupling in words; the letters collide with the claim's.

## Collision rules

The same letter often carries different meanings in different layers. The rules below keep one meaning per page; where a formula uses a letter locally, its "where" list fixes the meaning.

| Symbol | Meanings in this handbook | Rule |
|---|---|---|
| $\kappa$ | Consumer in $\mathcal{R}$; chance-corrected agreement | Bare $\kappa$ is the consumer, only inside $\mathcal{R}$. Always write [Cohen's κ](https://benchmaker.ai/glossary/cohens-kappa) for agreement. |
| $V$ | Evidence view $V$; verifier $V_\psi$; value $V^\pi$ (and $Q^\pi$); validation layer $\mathcal{V}$ | Plain $V$ is the evidence view. The others always carry their subscript, superscript, or script letter. |
| $R$ | Resolution $R$; regime $\mathcal{R}$ | Plain $R$ is resolution, and the regime is always $\mathcal{R}$. An expert's value is $Q_z$ and a judge's rubric is $\Psi$, never $R$. |
| $A$ | Anchor $A$; information access $A_{\text{access}}$; anchor channel A of O/A/P; exemplar anchors $A_i$; an action $a$ | Qualify every "anchor" in prose: reference anchor, anchor channel, exemplar anchors, or linking items. |
| $P$, $p$ | Population $P$; pass or base rate $p$; nominal density $p(x)$ in importance sampling; failure probability $P_f$ | Write probabilities as $\Pr[\cdot]$ so that $P$ stays the population. |
| $\Psi$, $\psi$ | Specification $\Psi$; the same specification as a predicate $\psi$ | Never write the specification as $\Sigma$. |
| $E$, $C$, $D$, $T$, $G$ | Claim: environment, construct, validity domain. $\Pi$: exploitability, channel coupling, drift, transport, generator coupling. Validity statement: evaluator system, claim, domain, trust root. Coupling diagnostics: evaluator $E$, generator $G$. Chain links: task success $T$, positive grade $G$ | Write $\Pi$'s components and the statement's slots in words. |
| $S$ | Modification surface $S$; system behavior $S(x)$ | The aggregate score is written $\Gamma(z)$, never $S$. |
| $\tau$, $\theta$ | Trajectory $\tau$; thresholds $\theta_i$ and a flag threshold $\theta$; parameters in $g_\theta$ and $\pi_\theta$ | Thresholds are $\theta_i$, never $\tau_i$; $\theta$ as a subscript names parameters. |
| $t$, $f$ | TPR and FPR in selection and alert formulas; $f$ is also an objective, a density $f_1$, and a learned scorer $f_\phi$ | $t$ and $f$ mean TPR and FPR only where the "where" list says so. |
| $b$, $\delta$ | Exposure budget $b$ and McNemar's discordant count $b$; decision $\delta$, a disturbance $\delta$, and the smallest difference that matters | McNemar's $b$ and $c$ are counts. Read $\delta$ from its formula. |
| $\alpha$, $\beta$ | Significance level, quantile level, conformal miss rate, Beta prior, and system effect $\alpha_m$; Beta prior, power $1 - \beta$, KL price, and expert consistency | Local to each formula. |

## Conventions

- **Pass rates.** $p$ is the true rate on the claim's population, or a base rate; $\hat p$ is an estimate; $p_{\text{obs}}$ is a judge's raw reading; $p_{\text{fail}}$ is a failure probability. A sampled rate always carries its interval and n, as in the illustrative 80% (95% Wilson CI 71–87%, n = 100).
- **Error rates.** TPR and FPR are written $\mathrm{TPR}$ and $\mathrm{FPR}$ in the sensor formulas and $t$ and $f$ in the selection and alert formulas. Specificity, or TNR, is $1 - \mathrm{FPR}$ ([sensitivity and specificity](https://benchmaker.ai/glossary/sensitivity-and-specificity)).
- **Trajectories.** $\tau = (s_0, a_0, s_1, a_1, \dots, s_T)$, with states $s_t$ and actions $a_t$ ([Evaluating agents](https://benchmaker.ai/handbook/agents/evaluating-agents)).
- **Margins.** $\rho(\psi, \tau)$ is a signed [robustness margin](https://benchmaker.ai/glossary/robustness-margin): positive when $\tau$ satisfies $\psi$, negative when it violates it, and its size says by how much ([Specification](https://benchmaker.ai/handbook/instrument/specification)).
- **Exploitability.** $\varepsilon(b)$ is the evaluator's disagreement with an independent anchor at exposure $b$, and $b^{*}$ is the largest tested exposure before that divergence exceeds a predeclared tolerance ([tested exposure](https://benchmaker.ai/glossary/tested-exposure)). Report $b^{*}$ with the tested range, never as "robust".
- **Typography.** Multi-letter names are upright, as in $\mathrm{FPR}$ and $\mathrm{CVaR}$; $\Pr[\cdot]$ is a probability, $\mathbb{E}[\cdot]$ an expectation, and $\mathbb{1}[\cdot]$ is 1 when its condition holds and 0 otherwise.

## Formula card

Each entry gives the formula, a "where" list, when it applies, the published source where there is one, the chapter that develops it, and the calculator that computes it, if one exists.

### Estimating a rate and its interval

#### Target and observed pass rate

$$
p = \mathbb{E}_{x \sim P}\big[\psi(S(x))\big], \qquad \hat p = \frac{1}{n}\sum_{i=1}^{n} y_i
$$

where:

- $S(x)$ is the system's behavior on task $x$, drawn from the claim's population $P$;
- $y_i \in \{0, 1\}$ is the outcome of task $i$ among $n$ independent tasks.

$\hat p$ estimates the never-observed $p$ only when the tasks represent $P$; a suite drawn from another distribution $Q$ estimates $\mathbb{E}_{x \sim Q}[\psi(S(x))]$ instead ([Task distributions and dataset engineering](https://benchmaker.ai/handbook/claim/task-distributions)).

**Chapter:** [Estimands, pass rates, and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals)

#### Wilson interval

$$
\begin{aligned}
\text{center} &= \frac{\hat p + z^2/(2n)}{1 + z^2/n} \\
\text{half-width} &= \frac{z}{1 + z^2/n}\sqrt{\frac{\hat p(1 - \hat p)}{n} + \frac{z^2}{4n^2}}
\end{aligned}
$$

where:

- $\hat p$ is the observed pass rate on $n$ independent tasks;
- $z$ is the standard normal quantile, 1.96 for 95%.

The [Wilson interval](https://benchmaker.ai/glossary/wilson-interval) is the default for independent binary outcomes at moderate n, and unlike the Wald form it keeps a nonzero width when every task passes or every task fails; when trials repeat within tasks, resample whole tasks instead.

**Chapter:** [Estimands, pass rates, and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals) · **Calculator:** [Pass-rate intervals and sample size](https://benchmaker.ai/tools/confidence-interval-calculator)

#### Beta-binomial posterior

$$
\begin{aligned}
p &\sim \mathrm{Beta}(\alpha, \beta) \\
p \mid y &\sim \mathrm{Beta}(\alpha + s,\ \beta + n - s)
\end{aligned}
$$

where:

- $\alpha$ and $\beta$ set the prior, and $\alpha = \beta = 1$ is uniform;
- $s$ is the number of passes among $n$ tasks.

Use the [Beta-binomial](https://benchmaker.ai/glossary/beta-binomial) posterior, which states its prior openly and assumes independent tasks, when the release question is itself a probability, such as $\Pr[p > 0.95]$.

**Chapter:** [Estimands, pass rates, and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals)

#### Stratified estimate

$$
\begin{aligned}
\hat p_{\text{strat}} &= \sum_h w_h\, \hat p_h \\
\mathrm{SE}(\hat p_{\text{strat}}) &= \sqrt{\sum_h w_h^{2}\, \frac{\hat p_h (1 - \hat p_h)}{n_h}}
\end{aligned}
$$

where:

- $h$ indexes the strata, and $w_h$ is stratum $h$'s share of the claim's population, with the shares summing to 1;
- $\hat p_h$ and $n_h$ are the stratum's observed pass rate and number of tasks.

The [stratified estimate](https://benchmaker.ai/glossary/stratified-estimate) reweights a suite that over-samples some slices on purpose, with weights taken from the claim's population; no weighting reaches a stratum the suite never sampled.

**Chapter:** [Estimands, pass rates, and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals)

#### Kish effective sample size

$$
n_{\text{eff}} = \frac{\left(\sum_i w_i\right)^2}{\sum_i w_i^2}
$$

where $w_i$ is the weight on item $i$.

The [Kish effective sample size](https://benchmaker.ai/glossary/kish-effective-sample-size) prices any reweighting as the number of equally weighted items that would carry the same information; when it falls to a small fraction of n, the suite's mix is far from the population's.

**Chapter:** [Estimands, pass rates, and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals)

#### Covariate-shift weight

$$
w(x) = \frac{P(x)}{Q(x)}
$$

where:

- $P(x)$ is the probability of task $x$ under the claim's population;
- $Q(x)$ is its probability under the distribution the suite actually samples.

The [covariate-shift weight](https://benchmaker.ai/glossary/covariate-shift-weight) works only where $Q$ puts probability wherever $P$ does, because weighting cannot repair missing support.

**Chapter:** [Estimands, pass rates, and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals)

### Correcting a noisy judge

#### Judge sensor model

$$
p_{\text{obs}} = p\,\mathrm{TPR} + (1 - p)\,\mathrm{FPR}
$$

where:

- $p$ is the system's true pass rate, relative to the specification;
- $\mathrm{TPR}$ and $\mathrm{FPR}$ are the judge's error rates on this system's outputs, measured against adjudicated labels;
- $p_{\text{obs}}$ is the pass rate the judge reports.

The model applies to binary verdicts, as in estimating prevalence from an imperfect screening test, and its error rates belong to the outputs they were measured on: a system optimized against the judge can have different ones, which is [differential misclassification](https://benchmaker.ai/glossary/differential-misclassification).

**Chapter:** [The judge as a noisy sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor) · **Calculator:** [Correcting a noisy judge](https://benchmaker.ai/tools/judge-noise-correction)

#### Rogan–Gladen correction

$$
\hat p = \frac{p_{\text{obs}} - \mathrm{FPR}}{\mathrm{TPR} - \mathrm{FPR}}
$$

where $\hat p$ is the corrected estimate of the true pass rate, and the other symbols are as in the sensor model.

The [Rogan–Gladen correction](https://benchmaker.ai/glossary/rogan-gladen-correction) ([Rogan & Gladen, 1978](https://benchmaker.ai/handbook/reference/sources#rogan-gladen-1978)) applies when the error rates transport to the eval set and TPR ≠ FPR; clip a result outside 0 to 1 and treat the clipping as evidence that they do not transport, and note that no correction rescues a rubric that operationalizes the wrong construct.

**Chapter:** [The judge as a noisy sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor) · **Calculator:** [Correcting a noisy judge](https://benchmaker.ai/tools/judge-noise-correction)

#### Variance of the corrected rate

$$
\begin{aligned}
\operatorname{Var}(\hat p) \approx \frac{1}{J^{2}} \Big[\, & \frac{p_{\text{obs}}(1 - p_{\text{obs}})}{n} \\
&+ \hat p^{2} \operatorname{Var}(\mathrm{TPR}) \\
&+ (1 - \hat p)^{2} \operatorname{Var}(\mathrm{FPR}) \,\Big]
\end{aligned}
$$

where:

- $J = \mathrm{TPR} - \mathrm{FPR}$ is the judge's separation between true passes and true failures;
- $n$ is the number of eval items the judge graded;
- $\operatorname{Var}(\mathrm{TPR})$ and $\operatorname{Var}(\mathrm{FPR})$ are the sampling variances of the calibration estimates.

This first-order approximation sizes the problem but is not a release-grade interval: it blows up as $J \to 0$, and decision-grade intervals on corrected rates need a construction built for the purpose, such as the one by [Lang and Reiczigel (2014)](https://benchmaker.ai/handbook/reference/sources#lang-reiczigel-2014).

**Chapter:** [The judge as a noisy sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor)

### Selection and alert precision

#### Benign selection model

$$
\Pr[C \mid +] = \frac{p\,t}{p\,t + (1-p)\,f}
$$

where:

- $C$ is the event that a candidate is correct, and $+$ the event that the evaluator flags it;
- $p$ is the share of candidates that are correct;
- $t$ and $f$ are the evaluator's TPR and FPR on these candidates.

The formula gives the precision of [best-of-N](https://benchmaker.ai/glossary/best-of-n) selection through a binary evaluator as a model, not a theorem, assuming independent candidates, values of $t$ and $f$ that hold for every candidate at every N, and a uniform choice among flagged candidates; N does not appear in it, so more samples cannot raise the ceiling.

**Chapter:** [Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity) · **Calculator:** *Best-of-N with an imperfect verifier*

#### Winner correctness under top-score selection

$$
\Pr[C_{\text{winner}}] = N \int p\, f_1(s)\, G(s)^{N-1}\, ds
$$

where:

- $F_1$ and $F_0$ are the score distributions of correct and incorrect candidates, and $f_1$ is the density of correct scores;
- $G(s) = p\,F_1(s) + (1-p)\,F_0(s)$ is the chance that one other candidate scores below $s$;
- $N$ is the number of candidates.

This stylized expression applies when the selector keeps the highest score; the result depends on the extreme tail of the score distributions, so precision can fall, rise, or plateau as N grows, and ordinary calibration does not answer the question.

**Chapter:** [Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity) · **Calculator:** *Best-of-N with an imperfect verifier*, with continuous scores

#### Alert precision

$$
\Pr[\text{danger} \mid \text{alert}] = \frac{p\,t}{p\,t + (1 - p)\,f}
$$

where:

- $p$ is the base rate of truly dangerous events;
- $t$ is the monitor's true-positive rate on dangerous events, and $f$ its false-positive rate on harmless ones.

[Alert precision](https://benchmaker.ai/glossary/alert-precision) is the selection arithmetic at a production base rate, where even a small false-positive rate can dominate the alerts when $p$ is tiny, so validate a monitor at that base rate, not on a balanced test set.

**Chapter:** [Monitoring and control in production](https://benchmaker.ai/handbook/regime/production-monitoring) · **Calculator:** [Alert precision at production base rates](https://benchmaker.ai/tools/alert-precision-calculator)

#### Conformal flag threshold

$$
\theta = s_{(k)}, \qquad k = \lfloor (n + 1)\,\alpha \rfloor
$$

where:

- $s_{(1)} \le \dots \le s_{(n)}$ are the monitor's badness scores on $n$ labeled true failures, sorted;
- $\alpha$ is the target miss rate, and events that score at or above $\theta$ are flagged.

Under exchangeability, at most about $\alpha$ of future true failures score below the threshold, but the guarantee of [conformal prediction](https://benchmaker.ai/glossary/conformal-prediction) is marginal, not per slice, breaks under drift, and says nothing about false positives.

**Source:** Kochenderfer et al. (2026) · **Chapter:** [Monitoring and control in production](https://benchmaker.ai/handbook/regime/production-monitoring)

### pass@k and pass^k

#### Independent pass@k and pass^k

$$
\text{pass@}k = 1 - (1 - p)^k, \qquad \text{pass}^k = p^k
$$

where:

- $p$ is the per-attempt success probability;
- $k$ is the number of attempts.

[pass@k](https://benchmaker.ai/glossary/pass-at-k) is the probability that at least one of k attempts succeeds and [pass^k](https://benchmaker.ai/glossary/pass-power-k) the probability that all k succeed, both idealized for independent attempts at one rate, so compute them per task and then average when difficulty varies. The Berkeley slides label the all-k quantity "pass@k" ([Song, 2025](https://benchmaker.ai/handbook/reference/sources#song-2025), slides 55, 57, and 61); read it as pass^k.

**Chapter:** [Estimands, pass rates, and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals) · **Calculator:** [pass@k versus pass^k](https://benchmaker.ai/tools/pass-at-k-calculator)

#### Unbiased pass@k estimator

$$
\widehat{\text{pass@}k} = 1 - \binom{n - c}{k} \Big/ \binom{n}{k}
$$

where:

- $n$ is the number of samples drawn for one task, at least $k$;
- $c$ is the number of those samples that pass.

The [unbiased estimator](https://benchmaker.ai/glossary/unbiased-pass-at-k) of [Chen et al. (2021)](https://benchmaker.ai/handbook/reference/sources#chen-2021) is computed per task, averaged across tasks, and given an interval by resampling tasks; the plug-in $1 - (1 - \hat p)^k$ reads low on average.

**Source:** Chen et al. (2021) · **Chapter:** [Estimands, pass rates, and intervals](https://benchmaker.ai/handbook/validation/estimands-and-intervals) · **Calculator:** [pass@k versus pass^k](https://benchmaker.ai/tools/pass-at-k-calculator)

> [!OPEN]
> This handbook gives an unbiased estimator for pass@k but none for pass^k. τ-bench reports the all-k quantity from repeated i.i.d. runs (Song, 2025, slide 55); its paper ([Yao et al., 2024](https://benchmaker.ai/handbook/reference/sources#yao-2024)) is the place to check how a number was computed. This card does not supply an estimator.

### Comparing two systems

#### McNemar's test

$$
\chi^2 = \frac{(b - c)^2}{b + c}
$$

where:

- $b$ is the number of tasks that system A passes and system B fails;
- $c$ is the number of tasks that B passes and A fails.

[McNemar's test](https://benchmaker.ai/glossary/mcnemar-test) compares two systems on the same tasks through the [discordant pairs](https://benchmaker.ai/glossary/discordant-pairs), and under equal performance its statistic is approximately chi-square with one degree of freedom; use the exact binomial test when $b + c$ is small, and report the effect, $(b - c)/N$ on $N$ paired tasks, with its interval, not only a p-value.

**Chapter:** [Comparing systems](https://benchmaker.ai/handbook/validation/comparing-systems) · **Calculator:** [Comparing two systems on the same tasks](https://benchmaker.ai/tools/paired-comparison-calculator)

#### Paired-comparison sample size

$$
N \approx \frac{\left(z_{1-\alpha/2} + z_{1-\beta}\right)^2 d}{\delta^2}
$$

where:

- $N$ is the number of paired tasks;
- $\delta$ is the smallest true difference in pass rates that would change the decision;
- $d$ is the expected share of tasks on which the two systems disagree;
- $z_{1-\alpha/2}$ and $z_{1-\beta}$ are standard normal quantiles for the significance level and the [power](https://benchmaker.ai/glossary/statistical-power) $1 - \beta$.

This is a rough sizing rule from McNemar's normal approximation, slightly optimistic against the exact test. It sizes a comparison for the smallest difference that would change the decision; the chapter derives it.

**Chapter:** [Comparing systems](https://benchmaker.ai/handbook/validation/comparing-systems)

#### Family-wise false alarms

$$
\Pr[\text{at least one false alarm}] = 1 - (1 - \alpha)^m
$$

where:

- $m$ is the number of gates;
- $\alpha$ is the significance level of each gate.

The formula assumes independent gates, and $m\alpha$ bounds the [family-wise false-alarm rate](https://benchmaker.ai/glossary/family-wise-false-alarm) under any dependence; pre-register the gates or correct for the family, for example by testing each gate at $\alpha/m$.

**Chapter:** [Comparing systems](https://benchmaker.ai/handbook/validation/comparing-systems) · **Calculator:** [Comparing two systems on the same tasks](https://benchmaker.ai/tools/paired-comparison-calculator), lower panel

> [!OPEN]
> Arena-style leaderboards rank many systems from pairwise votes. This handbook gives no treatment of ranking uncertainty under Bradley–Terry or Elo models, so this card has no formula for it.

### Rare events

#### Trials a rare failure needs

Under naive sampling from the claim's population, seeing even one failure at rate $p$ takes about $1/p$ trials, and estimating $p$ to within about 10% of its value takes about $100/p$ ([Kochenderfer et al., 2026](https://benchmaker.ai/handbook/reference/sources#kochenderfer-2026)).

**Chapter:** [Rare-event assurance](https://benchmaker.ai/handbook/validation/rare-events)

#### Rule of three

$$
p_{\text{fail}} \lesssim \frac{3}{n}
$$

where:

- $p_{\text{fail}}$ is the true probability that one trial fails;
- $n$ is the number of independent, representative trials, none of which failed.

The [rule of three](https://benchmaker.ai/glossary/rule-of-three) is a rough 95% upper bound, exactly $1 - 0.05^{1/n}$; its Bayesian form updates a uniform prior to $\mathrm{Beta}(1, n + 1)$, with 95% upper bound $1 - 0.05^{1/(n+1)} \approx 3/(n+1)$, and extends to $s$ failures as $\mathrm{Beta}(1 + s, 1 + n - s)$. Clustered or correlated trials make every one of these bounds optimistic.

**Chapter:** [Rare-event assurance](https://benchmaker.ai/handbook/validation/rare-events) · **Calculator:** [What zero failures buys you](https://benchmaker.ai/tools/rule-of-three-calculator)

#### Importance sampling

$$
\begin{aligned}
P_f &= \mathbb{E}_{q}\!\left[\frac{p(x)}{q(x)}\, \mathbb{1}\{\text{fail}(x)\}\right] \\
&\approx \frac{1}{N} \sum_{i=1}^{N} \frac{p(x_i)}{q(x_i)}\, \mathbb{1}\{\text{fail}(x_i)\}
\end{aligned}
$$

where:

- $P_f$ is the failure probability under the nominal distribution;
- $p(x)$ is the probability of task $x$ under the nominal distribution, not a pass rate;
- $q(x)$ is its probability under the proposal from which the $N$ tasks $x_i$ are drawn.

[Importance sampling](https://benchmaker.ai/glossary/importance-sampling) samples where failures are common and reweights each outcome by its likelihood ratio, which keeps the estimate unbiased for the nominal distribution only if the proposal covers the whole failure region; a poorly matched proposal makes the variance blow up.

**Source:** Kochenderfer et al. (2026) · **Chapter:** [Rare-event assurance](https://benchmaker.ai/handbook/validation/rare-events)

#### Multilevel splitting

$$
p_{\text{fail}} = \prod_{\ell=1}^{L} \Pr\big[f \le \gamma_\ell \mid f \le \gamma_{\ell-1}\big]
$$

where:

- $f$ is a robustness-style objective, with failure exactly when $f \le 0$;
- $\gamma_1 > \gamma_2 > \dots > \gamma_L = 0$ are the thresholds, with $\gamma_0$ above every outcome.

[Multilevel splitting](https://benchmaker.ai/glossary/multilevel-splitting) writes a rare probability as a product of common conditional ones, each estimated among the trials that reached the previous level: the honest form of a staged red-team funnel, whose end count is never a rate from uniform sampling.

**Source:** Kochenderfer et al. (2026) · **Chapter:** [Rare-event assurance](https://benchmaker.ai/handbook/validation/rare-events)

#### Falsification objective

$$
\min_{x,\,\delta}\ \rho\big(\psi,\ S(x;\delta)\big)
$$

where:

- $x$ is an input and $\delta$ a disturbance, such as a paraphrase, a tool failure, or an adversarial suffix;
- $S(x;\delta)$ is the trajectory the system produces, and $\rho$ its robustness margin.

[Falsification](https://benchmaker.ai/glossary/falsification) searches for failures rather than counting them: any negative value is a counterexample, the search needs a slope that a pass/fail grader does not give, and a failure found by search says nothing about how often it occurs.

**Source:** Kochenderfer et al. (2026) · **Chapter:** [Falsification and failure regions](https://benchmaker.ai/handbook/pressure/falsification)

#### Most likely failure

$$
\begin{aligned}
\tau^{*} = {} & \arg\max_{\tau}\ \log p_{0}(\tau) \\
& \text{subject to}\ \ \rho(\psi, \tau) < 0
\end{aligned}
$$

where:

- $p_0$ is the nominal distribution over trajectories, the traffic production actually serves;
- $\tau^{*}$ is the violating trajectory that distribution is most likely to produce.

Weight the search by plausibility when nature is the adversary; drop the weight against a strategic adversary, who chooses the input.

**Source:** Kochenderfer et al. (2026) · **Chapter:** [Falsification and failure regions](https://benchmaker.ai/handbook/pressure/falsification)

> [!OPEN]
> This handbook states the importance-sampling and splitting estimators but gives no worked numerics, variance diagnostics, or guidance on designing a proposal or a threshold ladder for an AI system; that machinery belongs to the validation-algorithms literature.

### Aggregation

#### Five geometries

For criterion-level signals $z = (z_1, \dots, z_k)$, the common geometries are:

| Geometry | Rule | What it asserts |
|---|---|---|
| Weighted utility | $\Gamma(z) = \sum_i w_i z_i$ | Every criterion trades against every other at the stated weights |
| [Conjunctive gate](https://benchmaker.ai/glossary/conjunctive-gate) | $\Gamma(z) = \prod_i \mathbb{1}[z_i \ge \theta_i]$ | Some failures are fatal |
| [Lexicographic order](https://benchmaker.ai/glossary/lexicographic-rule) | $z_{\text{safety}} \succ z_{\text{quality}} \succ z_{\text{cost}}$ | Strict priorities |
| Pareto vector | Keep all $k$ signals; compare by dominance | No trade-off has been agreed yet |
| Risk-sensitive rule | Accept only if $\mathrm{CVaR}_\alpha(L) \le \lambda$ | The tail overrides the average |

where $w_i$ are weights, $\theta_i$ thresholds, $L$ the per-task loss, and $\lambda$ a tolerance. Each geometry is a normative claim about [compensability](https://benchmaker.ai/glossary/compensability), not a statistical detail. This handbook writes the aggregate as $\Gamma(z)$, the thresholds as $\theta_i$, and the tolerance as $\lambda$, keeping $S$ and $\tau$ for the modification surface and the trajectory.

**Chapter:** [Aggregation geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry) · **Calculator:** *Aggregation geometry*

#### VaR and CVaR

$$
\mathrm{CVaR}_\alpha(L) = \frac{1}{m} \sum_{j=1}^{m} L_{(j)}
$$

where:

- $L_{(1)} \ge L_{(2)} \ge \dots \ge L_{(n)}$ are the task losses, sorted from worst to best;
- $m = \lceil (1 - \alpha)\, n \rceil$ is the number of tasks in the tail;
- $\mathrm{VaR}_\alpha(L) = F_L^{-1}(\alpha)$ is the $\alpha$-quantile of the loss: $\mathrm{VaR}_{0.95}$ of latency is its p95.

[CVaR](https://benchmaker.ai/glossary/cvar), the average severity of the worst $1 - \alpha$ share, sees how bad the tail is and not only where it starts, so gate the CVaR of a harm score rather than its mean, remembering that with few tasks the tail rests on a handful of observations.

**Source:** Kochenderfer et al. (2026) · **Chapter:** [Aggregation geometry](https://benchmaker.ai/handbook/instrument/aggregation-geometry) · **Calculator:** *Aggregation geometry*

> [!CONVENTION] CVaR averages the worst share
> This handbook and the aggregation calculator define CVaR as the worst-share average above, not as the conditional mean $\mathbb{E}[X \mid X \ge \mathrm{VaR}_\alpha(X)]$. When losses tie at the quantile, that conditional mean takes in every tied value and understates the tail. In the aggregation chapter's illustrative release with harm 1.0 on 2 of 100 tasks and 0 on the rest, $\mathrm{VaR}_{0.95} = 0$, the conditional mean is 0.02, and the worst-5% average is 0.4. The two forms agree when exactly a $1 - \alpha$ share of the losses lies at or above $\mathrm{VaR}_\alpha$, as it does for any continuous loss distribution.

### Reliability

#### Cohen's κ

$$
\text{Cohen's }\kappa = \frac{p_o - p_e}{1 - p_e}
$$

where $p_o$ is the observed agreement between two scorers and $p_e$ the agreement expected by chance.

Cohen's κ depends on prevalence, so the same scorer posts very different values at different base rates; report the confusion matrix, TPR and TNR, and the calibration set's prevalence with it, never the coefficient alone.

**Chapter:** [The judge as a noisy sensor](https://benchmaker.ai/handbook/validation/judge-as-noisy-sensor)

#### Variance components

$$
\begin{aligned}
Y_{mtrje} = {} & \mu + \alpha_m + u_t + v_r + w_j + q_e \\
& + \text{interactions} + \varepsilon
\end{aligned}
$$

where:

- $Y_{mtrje}$ is the score of system $m$ on task $t$, in rollout $r$, graded by judge $j$, in environment snapshot $e$;
- $\mu$ is the grand mean, $\alpha_m$ the system's effect, and $u_t$, $v_r$, $w_j$, $q_e$ the task, rollout, judge, and snapshot effects;
- $\varepsilon$ is the residual.

[Generalizability theory](https://benchmaker.ai/glossary/generalizability-theory) decomposes a score's variability by facet, with a mixed-effects model estimating each share, and the claim decides which facets count as error.

**Source:** Brennan (2001) · **Chapter:** [Reliability and generalizability](https://benchmaker.ai/handbook/validation/reliability)

#### Generalizability coefficient

$$
\Phi = \frac{\sigma^2_{\text{object}}}{\sigma^2_{\text{object}} + \sigma^2_{\Delta}}
$$

where:

- $\sigma^2_{\text{object}}$ is the variance attributable to the object of measurement, such as the systems compared;
- $\sigma^2_{\Delta}$ collects the variance from sources that should not change the interpretation.

The [absolute coefficient](https://benchmaker.ai/glossary/generalizability-coefficient) is the share of reported-score variance that comes from real differences between systems, and the design question it forces matters more than the number: which sources of variation are legitimate, and how many replications of each make the decision stable.

**Source:** Brennan (2001) · **Chapter:** [Reliability and generalizability](https://benchmaker.ai/handbook/validation/reliability)

#### Variance budget for a pass rate

$$
\begin{aligned}
\operatorname{Var}(\hat p) = {} & \frac{\sigma^2_q}{n_q} + \frac{\sigma^2_d}{n_d} \\
& + \frac{\sigma^2_r}{n_q n_d n_r} + \frac{\sigma^2_j}{n_q n_d n_r n_j}
\end{aligned}
$$

where:

- $\sigma^2_q$, $\sigma^2_d$, $\sigma^2_r$, and $\sigma^2_j$ are the variances contributed by one query, one documentation version, one rollout, and one judge call;
- $n_q$, $n_d$, $n_r$, and $n_j$ are how many of each the design samples.

Each source's variance shrinks only with draws of that source, ignoring the query-by-version interaction; the chapter derives this [variance budget](https://benchmaker.ai/glossary/variance-components) for its DocsAgent design.

**Chapter:** [Reliability and generalizability](https://benchmaker.ai/handbook/validation/reliability)

### Exploitability and independence

#### Exploitability curve and tested exposure

$$
\varepsilon(b) = \Pr\big[\,\text{evaluator} \neq \text{anchor} \mid Q_b\,\big]
$$

where:

- $b$ is the exposure budget, in the optimizer's own unit: N for best-of-N, KL for regularized RL, iterations for developers;
- $Q_b$ is the distribution of outputs after optimization at exposure $b$;
- "evaluator ≠ anchor" means disagreement with an independent reference anchor the optimizer can neither see nor modify.

The exploitability curve traces $\varepsilon(b)$ with the actual optimizer, each point an estimate with its own n and interval, and $b^{*}$ is the largest tested exposure before divergence exceeds a predeclared tolerance; record where possible the induced shift $D(Q_b \Vert P_0)$ from the starting distribution $P_0$, and claim nothing beyond the tested range.

**Chapter:** [Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity) · **Calculator:** *Exploitability under optimization*

#### Generator coupling

$$
\Pr[E \text{ errs} \mid G \text{ errs}] \;\;\text{vs.}\;\; \Pr[E \text{ errs}]
$$

where:

- $E$ is the evaluator and $G$ the generator whose outputs it judges;
- "errs" is measured against adjudicated labels.

If the first term rises well above the second, the evaluator tends to be wrong where the generator is wrong and is not an independent corrective channel; self-confirmation needs no adversary, only this correlated bias.

**Chapter:** [Goodhart pressure and self-confirming loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops)

#### Channel coupling

$$
\Pr[\text{A misses} \mid \text{O misses}] \quad \text{versus} \quad \Pr[\text{A misses}]
$$

where a miss is a true failure, confirmed against the trust root, that a channel passes; A is the anchor channel and O the optimized channel.

The framework asks for joint miss rates wherever they can be estimated, and the chapter reads them in the generator-coupling form: the further the first term rises above the second, the more the channels' blind spots line up.

**Chapter:** [Channels, independence, and anchor burn](https://benchmaker.ai/handbook/lifecycle/channels-and-independence)

### Training signals

#### Preference reward model

$$
\begin{aligned}
\Pr[y_w \succ y_l \mid x] &= \sigma(r_\phi(x, y_w) - r_\phi(x, y_l)) \\
\mathcal{L}_{\text{RM}}(\phi) &= -\mathbb{E}\big[\log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)\big]
\end{aligned}
$$

where:

- $r_\phi(x, y)$ is the reward model's score for response $y$ to prompt $x$;
- $y_w \succ y_l$ means the labeler preferred $y_w$ to $y_l$;
- $\sigma(z) = 1/(1 + e^{-z})$ is the logistic function.

The [Bradley–Terry model](https://benchmaker.ai/glossary/bradley-terry-model) learns an ordering from score differences, not a proof of quality, so a [reward model](https://benchmaker.ai/glossary/reward-model)'s absolute values carry no meaning of their own, and its validity domain is empirical.

**Chapter:** [Evaluators inside training loops](https://benchmaker.ai/handbook/regime/training-loops)

#### KL-regularized objective

$$
\begin{aligned}
\max_\theta\; & \mathbb{E}_{x,\, y \sim \pi_\theta}\big[r_\phi(x, y)\big] \\
& - \beta\, \mathbb{E}_x\Big[D_{\mathrm{KL}}\big(\pi_\theta(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x)\big)\Big]
\end{aligned}
$$

where:

- $\pi_\theta$ is the policy being trained and $\pi_{\text{ref}}$ a fixed reference policy;
- $\beta$ sets the price of divergence from the reference.

The [KL-regularized objective](https://benchmaker.ai/glossary/kl-regularized-objective) is the standard schematic of RLHF's policy step, and its KL term is an exposure control that bounds how far the policy may travel without certifying that the reward stays valid along the way.

**Chapter:** [Evaluators inside training loops](https://benchmaker.ai/handbook/regime/training-loops)

#### Verifiable reward

$$
r_{\text{RLVR}}(x, y) = \mathbb{1}\big[V_\psi(x, y) = 1\big]
$$

where $V_\psi(x, y)$ is the verdict of a verifier for specification $\psi$ on response $y$ to task $x$.

[RLVR](https://benchmaker.ai/glossary/rlvr) grounds the reward in an external check, but grounded is not complete: a test suite samples behavior, and a final-answer check can reward a flawed derivation that reaches the right answer.

**Chapter:** [Evaluators inside training loops](https://benchmaker.ai/handbook/regime/training-loops)

#### Step scores and rollout values

$$
\hat r_t = f_\phi(s_t, a_t, \text{ctx}), \qquad \hat V^{\pi}(s_t) = \frac{1}{K} \sum_{j=1}^{K} y_j
$$

where:

- $\hat r_t$ is a learned [PRM](https://benchmaker.ai/glossary/prm)'s score for step $t$, with state $s_t$, action $a_t$, and context ctx;
- $K$ is the number of completions sampled from the current policy $\pi$ starting at $s_t$, and $y_j$ is 1 if completion $j$ succeeds.

A rollout label estimates a policy-conditioned [value](https://benchmaker.ai/glossary/value-model) that changes whenever the policy does, not a judgment of the step: a PRM score, a value estimate, and a robustness margin $\rho(\psi, \tau)$ are three different objects.

**Chapter:** [Signals and resolution](https://benchmaker.ai/handbook/instrument/signal-and-resolution)

### Inverse evaluation design

#### Posterior over judgments

$$
p(z \mid \mathcal{D}) \propto p(\mathcal{D} \mid z)\, p(z)
$$

where:

- $z$ parameterizes the latent judgment: criteria, weights, gates, thresholds, and interactions;
- $\mathcal{D}$ is the record of expert behavior, not the validity domain $D$.

[Inverse evaluation design](https://benchmaker.ai/glossary/inverse-evaluation-design), this handbook's own synthesis, keeps a distribution over candidate judgments whose spread shows where asking experts is worth most, but it cannot deliver formal soundness or validate the construct from inside the preference data.

**Chapter:** [Inverse evaluation design](https://benchmaker.ai/handbook/practice/inverse-evaluation-design)

#### Boltzmann choice model

$$
\Pr[a \mid s, z, \beta] = \frac{\exp\big(\beta\, Q_z(s, a)\big)}{\sum_{a'} \exp\big(\beta\, Q_z(s, a')\big)}
$$

where:

- $s$ is the case in front of the expert and $a$ the expert's decision;
- $Q_z(s, a)$ is how good decision $a$ is under judgment $z$;
- $\beta \ge 0$ is the expert's consistency.

Experts usually choose what their judgment favors and err most when options are close, and with two options and $\beta = 1$ a pairwise preference reduces to the Bradley–Terry form. The value is written $Q_z$ because $R$ is resolution.

**Chapter:** [Inverse evaluation design](https://benchmaker.ai/handbook/practice/inverse-evaluation-design)

### Other expressions

These expressions name relations rather than compute quantities:

| Expression | Meaning | Chapter |
|---|---|---|
| $V_\psi(y) = 1 \Rightarrow \psi(y) = 1$ | [Soundness](https://benchmaker.ai/glossary/soundness): whatever the verifier accepts satisfies the property | [The verification hierarchy, demoted](https://benchmaker.ai/handbook/foundations/verification-hierarchy) |
| $\Pr[T = 1 \mid C, E, H]$, $\Pr[G = 1 \mid T, \hat g, H]$ | Task success given the capability, and a positive grade given success: the probabilistic links | [Evaluation is measurement](https://benchmaker.ai/handbook/foundations/evaluation-is-measurement) |
| $z = g_\theta(x, y, \Psi, e)$ | A rubric-conditioned judge: every input, including the evidence $e$ and the configuration $\theta$, is part of the instrument | [LLM judges in practice](https://benchmaker.ai/handbook/instrument/llm-judges) |
| $J(x, y) \;\to\; (c_i) \;\to\; (e_i) \;\xrightarrow{\;g_i\;}\; z \;\xrightarrow{\;\Gamma\;}\; \delta$ | Expert judgment, compiled into criteria, evidence, criterion outcomes, and a decision | [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards) |
| $\mathcal{E} = \langle C, \Psi, M, Z, R, U \rangle$ | The [six-axis type](https://benchmaker.ai/glossary/six-axis-type): construct, specification, mechanism, signal, resolution, use | [Vocabulary without category errors](https://benchmaker.ai/handbook/foundations/vocabulary) |
| $\mathsf{G}$, $\mathsf{F}$, $\mathsf{U}$, as in $\neg\,\text{send}\ \mathsf{U}\ \text{confirmed}$ | [Temporal patterns](https://benchmaker.ai/glossary/temporal-pattern): always, eventually, until | [Specification](https://benchmaker.ai/handbook/instrument/specification) |
| $A \notin S$ and $I(A \to O)$ controlled | The anchor condition: the anchor is outside the modification surface, and what flows from it to the optimizer is limited | [Exposure, modification, and access](https://benchmaker.ai/handbook/regime/exposure-and-surfaces) |

---
Source: https://benchmaker.ai/handbook/reference/notation-and-formulas