# Aggregation geometry

> Weighted means assume strengths can offset weaknesses; gates assert some failures are fatal. How criteria combine is a normative decision, so make it explicitly.

**Aggregation geometry**, written $\Gamma$, is the rule that turns criterion-level signals into a decision: a weighted utility, a conjunctive gate, a lexicographic order, a Pareto vector, or a tail-risk rule. It is normative, not a statistical detail: each geometry asserts which failures can be offset and which cannot.

> [!KEY] In one sentence
> A weighted mean assumes strengths can offset weaknesses and a gate asserts that some failures are fatal, so choose how criteria combine as the normative decision it is, and write it down.

## Five geometries

The point compresses into three words: aggregation is normative. Each common geometry gives a different answer to the question of what may make up for what. For criterion-level signals $z = (z_1, \dots, z_k)$, such as per-criterion pass rates:

- **Weighted utility**, $\Gamma(z) = \sum_i w_i z_i$, trades every criterion against every other at the stated weights $w_i$. It fits only trade-offs that are explicitly elicited and defensible.
- **[Conjunctive gate](https://benchmaker.ai/glossary/conjunctive-gate)**, $\Gamma(z) = \prod_i \mathbb{1}[z_i \ge \theta_i]$, returns 1 only if every criterion clears its threshold $\theta_i$, so some failures are fatal. It fits safety and compliance criteria.
- **[Lexicographic order](https://benchmaker.ai/glossary/lexicographic-rule)**, $z_{\text{safety}} \succ z_{\text{quality}} \succ z_{\text{cost}}$, compares options on safety first and lets quality count only among options that safety does not separate. It fits strict priorities.
- **Pareto vector** keeps all $k$ signals and compares options by dominance: one option dominates another if it is at least as good on every criterion and better on at least one. It fits any stage before a trade-off has been agreed.
- **Risk-sensitive rule** accepts only if $\mathrm{CVaR}_\alpha(L) \le \lambda$: the per-task loss $L$, averaged over the worst $1 - \alpha$ share of tasks, stays within a tolerance $\lambda$. Worst-slice or rare-event risk overrides average performance, which fits harms, where a fat tail matters more than the mean.

Here $\mathbb{1}[\cdot]$ is 1 when its condition holds and 0 otherwise.

## Compensability is a claim

Putting helpfulness and safety into one weighted sum quietly assumes [compensability](https://benchmaker.ai/glossary/compensability): with enough helpfulness, a safety violation is paid for. That can suit aesthetic preferences and be disastrous for safety constraints. The judgment compiler, this handbook's procedure for turning expertise into graders, says it in one line at its decision-semantics step: safety and compliance criteria are often non-compensable, and five points of style should not buy back a forbidden customer action. A hard safety gate, a conjunctive gate on the safety criteria, states the intended decision semantics directly.

The geometry decides rankings even when nothing fails. In the playground's balanced scenario, two illustrative systems clear every gate and neither takes a forbidden action. System A is stronger on correctness (0.86 against 0.80); System B on helpfulness (0.84 against 0.70) and cost efficiency (0.78 against 0.60). With weights 0.35, 0.25, 0.25, and 0.15, the weighted mean ranks B first (0.857 against 0.816). The lexicographic order, tied on safety, decides on correctness and ranks A first. Neither system dominates, so the Pareto view leaves a trade-off that needs a stated value judgment, and both pass the tail rule. Now give B two forbidden actions in a hundred tasks, a safety score of 0.98. The weighted mean still ranks B first (0.852 against 0.816), while the gate, the lexicographic order, and the tail rule all turn against it: B's CVaR rises from 0.19 to 0.51, above $\lambda = 0.5$.

The playground starts in the balanced scenario; drag System B's safety to 0.98 and watch which verdicts change.

```widget
aggregation-playground
preset: balanced
```

## Choose the functional

When a criterion is scored over many tasks, its $z_i$ is itself an aggregate. A stochastic system run over a task distribution induces a distribution of metric values, and any single number quoted from it, whether mean, median, p95, or worst observed, is a chosen statistical functional of that distribution. The choice has teeth: mean latency, p95 latency, and worst-case latency protect different users.

For quantities that matter most when they are worst, look at the tail rather than the middle. Value at risk, $\mathrm{VaR}_\alpha$, is the $\alpha$-quantile of a metric's distribution, so p95 latency is $\mathrm{VaR}_{0.95}$; [conditional value at risk](https://benchmaker.ai/glossary/cvar) is the mean beyond that quantile. Over $n$ tasks,

$$
\mathrm{CVaR}_\alpha(L) = \frac{1}{m} \sum_{j=1}^{m} L_{(j)}
$$

where:

- $L_{(1)} \ge L_{(2)} \ge \dots \ge L_{(n)}$ are the task losses, sorted from worst to best;
- $m = \lceil (1 - \alpha)\, n \rceil$ is the number of tasks in the tail, so at $\alpha = 0.95$ the average covers the worst 5% of tasks;
- for a continuous loss distribution this is the sample version of the formula card's $\mathbb{E}[L \mid L \ge \mathrm{VaR}_\alpha(L)]$; with tied losses, as in Release 2 below, the two differ, so average the worst share as written here.

In words: CVaR is the average severity of the worst $1 - \alpha$ share, so it sees how bad the tail is, not only where it starts.

Two illustrative releases of DocsAgent, the documentation agent of [Case G](https://benchmaker.ai/handbook/cases/docs-agent), are scored on 100 tasks for the harm of following their instructions, from 0 (harmless) to 1 (severe):

| | Release 1 | Release 2 |
|---|---|---|
| Harm scores | 0.2 on 10 tasks, 0 on 90 | 1.0 on 2 tasks, 0 on 98 |
| Mean | 0.02 | 0.02 |
| $\mathrm{VaR}_{0.95}$ | 0.2 | 0 |
| $\mathrm{CVaR}_{0.95}$ | 0.2 | 0.4 |

The mean cannot tell the releases apart, and the 95th percentile prefers the one with two severe failures; only CVaR sees how bad its tail is. Hence the rule: gate the CVaR of a harm score, not its mean, so that a fat tail can no longer hide behind a good average. With 100 tasks the tail rests on five of them, so decision-grade tail estimates need the methods of [Rare-event assurance](https://benchmaker.ai/handbook/validation/rare-events).

## Plot the frontier, then elicit the weights

When a release trades several metrics, such as quality, cost, latency, and safety, a composite score scalarizes a multi-objective decision. An option is Pareto-optimal if no metric can improve without another worsening, and the defensible choices lie on the [Pareto frontier](https://benchmaker.ai/glossary/pareto-frontier). Plot the frontier before scalarizing, because a meeting that argues about a single blended score has already hidden the decision inside the weights.

When a CI gate needs a single verdict, derive the weights from the specification owner's concrete choices between pairs of options, not from numbers typed into a configuration file. Each choice between two frontier points is evidence about the exchange rate the weights must encode. Suppose, illustratively, that two DocsAgent configurations both clear the grounding gate: X produces working procedures on 90% of tasks at 2 cents per answer, and Y on 86% at 1 cent. If the docs product manager picks X, four points of working procedures are worth more to them than a cent per answer, and a handful of such choices brackets the weights. Setting the weights is the product decision; eliciting them is how that decision gets recorded honestly. Treating such choices as data about a latent judgment is the synthesis this handbook calls [inverse evaluation design](https://benchmaker.ai/handbook/practice/inverse-evaluation-design).

## Should one number exist?

Aggregating heterogeneous skills into one number makes a structural claim: the components can be combined in the proposed way, and the total has a coherent interpretation. Testing that claim calls for internal-structure evidence. If a benchmark has slices for planning, tool selection, memory, policy compliance, and recovery from failures, check whether their correlations and the model rankings they produce match the conceptual structure: a model can dominate on tool use and fail on policy compliance, and a simple mean can hide it. [Sources of validity evidence](https://benchmaker.ai/handbook/validation/validity-evidence) places internal structure among the other sources.

Until that evidence is in, prefer the vector: for engineering decisions it is often better to retain a vector of scores and use explicit gates or a Pareto analysis than to pretend that every capability trade-off is commensurable. Keep criteria apart long enough to see how they trade off, and combine them only when the decision needs a scalar and the weights are explicit.

> [!OPEN]
> Factor analysis, item-response models, and hierarchical models are candidate ways to test whether a headline score is defensible, but this handbook does not yet give a worked procedure for fitting them to AI evaluations.

## Recover the expert's decision geometry

Measured criteria are not yet codified judgment; the aggregation rule is a second layer of it. The expert must also recover the decision geometry: which deficiencies are fatal, which are tradeable, which require escalation, and how uncertainty changes the action. A [criterion card](https://benchmaker.ai/glossary/criterion-card) records the answer as severity semantics, namely fatal, tradeable, advisory, or escalate, and each maps onto a geometry; an escalation region counts as a geometry in its own right.

| Severity | Geometry | DocsAgent (Case G) |
|---|---|---|
| Fatal | Conjunctive gate, or first place in a lexicographic order | Citation grounding: a release needs at least 95% of answers fully grounded, and an unsupported safety-relevant instruction fails the answer |
| Tradeable | Weighted utility among admissible outputs, with elicited weights | Factual correctness, procedural faithfulness, and abstention when evidence is absent |
| Advisory | Reported beside the decision, never changing it | Cost and latency per answer *(illustrative)* |
| Escalate | A routing region that sends the case to a person or a stronger channel | The grounding judge abstains, or two documentation versions conflict *(illustrative)* |

The fatal and tradeable rows follow the Studio preset for [Case G](https://benchmaker.ai/handbook/cases/docs-agent), which gates grounding for the whole release. Within one answer, the DocsAgent grounding card is finer: only an unsupported safety-relevant instruction fails the whole answer, which is why [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards) suggests splitting that card in two.

## In practice

1. **List the criteria with the specification owner**, and mark each fatal, tradeable, advisory, or escalate.
2. **Choose each criterion's functional**, such as a pass rate for ordinary quality and a worst-slice rate or CVaR for harms, and state it as part of the estimand.
3. **Gate the fatal criteria** conjunctively, or put them first in a lexicographic order. Thresholds are product decisions, and a gate is incomplete until it also binds the uncertainty, not only the point estimate (see [Decision claims](https://benchmaker.ai/handbook/claim/decision-claims)).
4. **Keep the tradeable criteria as a vector**, and plot the Pareto frontier of the candidates.
5. **Elicit weights only if one verdict is needed**, from the owner's pairwise choices among frontier points, and record the choices.
6. **Define the escalation region** and where escalated cases go.
7. **Check internal structure** before publishing a headline number over heterogeneous slices.
8. **Version the geometry** with the rubric, judge, checkers, and thresholds, since aggregation logic is part of one measurement version.

The artifact is an aggregation specification (severities, functionals, gates and thresholds, weights with their elicitation record, and the escalation route), which completes the aggregation part of the specification field in the evaluator dossier.

## Common failure modes

- **The compensation trap** — a weighted mean lets strong scores elsewhere buy back a forbidden action → gate the fatal criteria and reserve weights for the tradeable ones.
- **Weights from a configuration file** — nobody can say why a criterion is worth 0.25 → elicit the weights from the owner's pairwise choices, and record them.
- **A mean for a harm** — a good average hides a fat tail → gate CVaR or the worst slice, and size the sample for the tail.
- **A headline over incoherent slices** — one average hides a model that fails policy compliance → test internal structure and report the vector. See [one-number reporting](https://benchmaker.ai/handbook/practice/anti-patterns#one-number-reporting).

## Check yourself

**Analyze.** A leaderboard averages five slices: planning, tool selection, memory, policy compliance, and recovery. What evidence would tell you whether the average should exist, and what would you report if it should not?

> [!DETAILS] Hint
> Look for internal-structure evidence: slice correlations, whether model rankings agree across slices, and models that lead on one slice while failing another. Without it, report the vector with gates or a Pareto view.

**Evaluate.** SQLAgent, the text-to-SQL agent of [Case C](https://benchmaker.ai/handbook/cases/sql-data-agent), is ranked by a weighted mean of four criteria (correct result set, no unauthorized reads or writes, ambiguity handling, and cost per question) with weights 0.4, 0.3, 0.2, and 0.1. Which criteria should never enter a compensating mean, and what would you use instead?

> [!DETAILS] Hint
> Unauthorized access is non-compensable: gate it, or put it first in a lexicographic order. Then decide with the owner whether correctness is a gate or a trade-off for the rollout decision (Case C gates it at 95%), and ask where the remaining weights came from.

**Create.** Write the aggregation specification for DocsAgent's four criteria plus cost per answer: severities, functionals, gates and thresholds, two elicitation questions for the docs product manager, and the escalation route.

> [!DETAILS] Hint
> Start from the Case G preset, in which grounding is fatal and the other three criteria are tradeable. Write each elicitation question as a choice between two frontier points that differ in working procedures and cost, and say where an abstaining judge's cases go.

## Further reading

- [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards) — severity semantics on the canonical criterion card.
- [Decision claims](https://benchmaker.ai/handbook/claim/decision-claims) — thresholds as product decisions, and gates that account for uncertainty.
- [Sources of validity evidence](https://benchmaker.ai/handbook/validation/validity-evidence) — internal-structure evidence for headline scores.
- [Rare-event assurance](https://benchmaker.ai/handbook/validation/rare-events) — estimating the tails that risk-sensitive rules depend on.
- [Kochenderfer et al. (2026)](https://benchmaker.ai/handbook/reference/sources#kochenderfer-2026), *Algorithms for Validation* — the validation framework this chapter follows for tail functionals and Pareto trade-offs.

---
Source: https://benchmaker.ai/handbook/instrument/aggregation-geometry