Estimands, pass rates, and intervals

An estimand is the exact quantity an evaluation sets out to estimate: which outcome, over which population of tasks, summarized by which functional. Write it down before choosing an interval: many statistical mistakes are actually estimand mistakes, where the formula is correct but answers the wrong question.

Start with the estimand#

Many evaluations estimate a pass rate, which this handbook writes as an expectation over the claim's population:

$$p = \mathbb{E}_{x \sim P}\big[\psi(S(x))\big]$$

where:

• $P$ is the population of tasks named in the claim;
• $S(x)$ is the behavior of the system under test on task $x$: an answer, an artifact, or a trajectory;
• $\psi$ is the specification read as a predicate: 1 if the behavior satisfies it and 0 if not.

In words: $p$ is the probability that the system's behavior on a task drawn from the claim's population meets the specification. It is never observed directly. A run of $n$ independent tasks gives outcomes $y_i \in \{0, 1\}$ and the empirical pass rate $\hat p = \frac{1}{n}\sum_i y_i$, which estimates $p$ only when the tasks represent $P$.

The pass rate is one estimand among several. Each pairs a population with a functional: the summary taken over the distribution of outcomes, such as a mean, a probability over k attempts, a quantile, or a difference. The common ones are below, with examples, several of them for SQLAgent, the text-to-SQL agent of Case C:

A SQLAgent team that averages over its regression suite has answered a legitimate question about its known failure modes, but a suite that over-samples known failures on purpose is not a population estimate. The decision in the claim selects the estimand: a release gate, a choice between versions, and an unattended workflow each need a different one (decision claims).

Report n and an interval#

Always report n and an uncertainty interval. For binary pass rates, Wilson or Bayesian intervals both serve, and the Wilson interval is a good frequentist default for moderate n:

$$\begin{aligned} \text{center} &= \frac{\hat p + z^2/(2n)}{1 + z^2/n} \\ \text{half-width} &= \frac{z}{1 + z^2/n}\sqrt{\frac{\hat p(1 - \hat p)}{n} + \frac{z^2}{4n^2}} \end{aligned}$$

where:

• $\hat p$ is the observed pass rate on $n$ independent tasks;
• $z$ is the standard normal quantile for the confidence level, 1.96 for 95%.

In words: the interval is pulled slightly toward 50% and keeps a nonzero width even when every task passes or every task fails. The Wald form $\hat p \pm 1.96\sqrt{\hat p(1-\hat p)/n}$ serves as a mental estimate in the middle of the range but breaks down near 0 and 1: for an illustrative clean run of 20 of 20 it gives an interval of zero width, while Wilson gives 84–100%.

In an illustrative release run, SQLAgent passes 80 of 100 tasks: report 80% (95% Wilson CI 71–87%, n = 100). Near 80%, the 95% half-width is roughly ±11 points at n = 50, ±8 at 100, ±4 at 400, and ±2.5 at 1,000, so at n = 100 an 84% and an 81% are usually not evidence of a real difference. The half-width shrinks in proportion to $1/\sqrt{n}$: halving it takes four times the tasks.

The calculator starts at SQLAgent's 80 of 100. Drag n to 400 and the half-width falls to about ±3.9 points; in Plan a sample size, ±5 points at 80% needs 245 independent tasks.

Slice-level claims need slice-level sample sizes and intervals. If 12 of SQLAgent's 100 tasks ask about business-calendar dates and 9 of them pass, the slice reads 75% (95% Wilson CI 47–91%, n = 12): compatible with the headline, and also with a slice that fails half its tasks.

When the question is a probability#

A Bayesian alternative, the Beta-binomial model, states its prior openly:

$$\begin{aligned} p &\sim \mathrm{Beta}(\alpha, \beta) \\ p \mid y &\sim \mathrm{Beta}(\alpha + s,\ \beta + n - s) \end{aligned}$$

where:

• $\alpha$ and $\beta$ set the prior, and $\alpha = \beta = 1$ is uniform;
• $s$ is the number of passes among $n$ tasks, and $y$ is the observed outcomes.

In words: the prior acts like $\alpha$ passes and $\beta$ failures seen before the run, and the run adds its own passes and failures.

The posterior is useful because many release questions are themselves probabilities, such as "What is the probability the true pass rate exceeds 95%?" In an illustrative release, a candidate must exceed 95% and passes 194 of 200 tasks: 97% (95% Wilson CI 94–99%, n = 200). Under a uniform prior the posterior is Beta(195, 7), and $\Pr[p > 0.95] \approx 0.88$. The observed rate clears the bar, yet the true rate falls short with probability of about one in eight; whether to accept that risk is a product decision.

Trials cluster within tasks#

Repeated trials of a stochastic agent are not new tasks: when each task is run multiple times, outcomes within a task are clustered, so do not pretend n × k trials are independent. Clustered trials call for resampling whole tasks, each with all its trials, or for a hierarchical model.

SQLAgent's evaluation in Case C runs multiple rollouts on ambiguous and multi-step tasks and treats tasks as clusters. In an illustrative run of 50 tasks with 4 rollouts each, 30 tasks pass all four rollouts, 15 fail all four, and 5 pass two of four: 130 of 200 trials, or 65%.

The honest interval is nearly twice as wide. Most of the variation lies between tasks, not between rollouts of one task, so the 200 trials carry about as much information as 55 independent tasks would, and more rollouts would mostly pin down each task's own rate. How variance divides among tasks, rollouts, judges, and environments is the subject of reliability and generalizability.

Retries: pass@k and pass^k#

Retrying a stochastic agent can serve two opposite purposes. pass@k is the probability that at least one of k attempts succeeds; pass^k is the probability that all k succeed. If each attempt succeeds independently with probability $p$:

$$\text{pass@}k = 1 - (1 - p)^k, \qquad \text{pass}^k = p^k$$

where:

• $p$ is the per-attempt success probability;
• $k$ is the number of attempts.

In words: pass@k needs one success among k tries, so it rises with k; pass^k needs success on every try, so it falls.

At p = 0.75 and k = 3, pass@3 is about 98% and pass^3 about 42%: one system can serve an inspected coding workflow well and still be unfit to run a customer workflow unattended. pass@k fits settings where search or retries are allowed and something reliably recognizes the success, as in the coding-agent case; repeated or unattended work needs pass^k.

Both formulas assume independent attempts at one rate. Retries in practice are often dependent, and difficulty varies by task, so compute the functional per task, then average. In an illustrative suite where half the tasks always succeed and half succeed half the time, the mean rate is 75%, but the average pass@3 is 93.8% and the average pass^3 56.3%, not the 98.4% and 42.2% the mean rate implies.

Estimating pass@k without bias#

Computed from a few trials per task, the plug-in $1 - (1 - \hat p)^k$ is biased. With $n$ trials sampled on a task, $c$ of which succeed, the standard unbiased estimator is (Chen et al., 2021):

$$\widehat{\text{pass@}k} = 1 - \binom{n - c}{k} \Big/ \binom{n}{k}$$

where:

• $n$ is the number of samples drawn for the task, at least $k$;
• $c$ is the number of those samples that pass;
• $\binom{n-c}{k}$ counts the sets of $k$ samples that contain no pass, and $\binom{n}{k}$ counts all sets of $k$ samples.

In words: the estimate is the share of all k-sample subsets that contain at least one pass. Compute it per task, average across tasks, and resample tasks for an interval.

For an illustrative task with 3 passes in 10 samples, the unbiased pass@5 is $1 - 21/252 \approx 91.7\%$ and the plug-in $1 - 0.7^5$ gives 83.2%. The plug-in is concave in $c/n$, so it reads low on average: if the true rate were 30%, its expected value would fall about 7 points short of the true pass@5.

The calculator opens at p = 0.75 and k = 3, with that 3-of-10 task in its lower panel. Raise k to 10 and pass@10 rounds to 100% while pass^10 falls to about 6%; then set the lower panel's k to 1, where the unbiased and plug-in estimates agree at 30%.

Reweight to the population#

Suites are often unbalanced on purpose: error analysis grows them from real failures, and this handbook recommends over-sampling hard slices. The reconciliation is deliberate bias with honest books: sample where the information is, then reweight to the population. With strata $h$, production shares $w_h$ that sum to 1, and per-stratum pass rates $\hat p_h$ on $n_h$ tasks, the stratified estimate and its standard error are:

$$\begin{aligned} \hat p_{\text{strat}} &= \sum_h w_h\, \hat p_h \\ \mathrm{SE}(\hat p_{\text{strat}}) &= \sqrt{\sum_h w_h^{2}\, \frac{\hat p_h (1 - \hat p_h)}{n_h}} \end{aligned}$$

where:

• $h$ indexes the strata, the slices the suite was built from;
• $w_h$ is stratum $h$'s share of the claim's population;
• $\hat p_h$ and $n_h$ are the stratum's observed pass rate and number of tasks.

In words: estimate each slice on its own, then combine the slices in the population's proportions, not the suite's.

An illustrative SQLAgent suite over-samples multi-table joins: 200 join tasks and 100 single-table lookups, while production traffic is 20% joins and 80% lookups. Joins pass 120 of 200 (60%, 95% Wilson CI 53–67%, n = 200) and lookups 95 of 100 (95%, 95% Wilson CI 89–98%, n = 100).

The unweighted figure describes a user base in which two questions in three need joins: unweighted reporting of a deliberately unbalanced suite is not a simplification; it is an estimate of a fictional user base. The standard error also shows where more labels would help most, because a stratum with a large weight but few tasks dominates it: here the lookups, with 80% of the weight and 100 tasks, contribute 86% of the variance.

The same idea works item by item. If the suite's tasks come from a distribution $Q$ but the claim concerns the population $P$, you can sometimes weight each task by the covariate-shift weight $w(x) = P(x)/Q(x)$. Reweighting has a price, which the Kish effective sample size states:

$$n_{\text{eff}} = \frac{\left(\sum_i w_i\right)^2}{\sum_i w_i^2}$$

where $w_i$ is the weight on item $i$. In words: unequal weights make some items count for more than others, and $n_{\text{eff}}$ is the number of equally weighted items that would carry the same information.

In the SQLAgent suite each join carries weight 0.3 and each lookup 2.4, so $n_{\text{eff}} \approx 152$: reweighting spent about half of the 300 tasks. If $n_{\text{eff}}$ falls to a small fraction of $n$, the suite's mix is far from production's, and a precise-looking headline is mostly noise. Nor can weighting repair missing support: for task regions the suite never samples, there is no evidence to reweight (see task distributions).

In practice#

Write an estimate record for each reported claim.

1. Write the estimand. Name the population, the outcome, and the functional the decision needs: a single-trial rate, pass@k, pass^k, a tail functional, or a paired difference.
2. Fix the sampling unit. Decide what counts as one independent observation; if tasks repeat, resample tasks or fit a hierarchical model.
3. Size the suite. Use the Wilson half-width to choose n for the precision the decision needs, and give each slice you will claim about its own n.
4. Choose the estimator. For pass@k with k > 1, use the unbiased estimator per task; for pass^k, state the method and its assumptions.
5. Reweight. If the suite is deliberately unbalanced, report the stratified estimate with its standard error, the production weights, and $n_{\text{eff}}$.
6. Report. Give every rate its n and interval, as in "80% (95% Wilson CI 71–87%, n = 100)", and give each slice its own.

The artifact is an estimate record per claim: the estimand in words, the sampling unit and weights, n per slice, the estimator, and each estimate with its interval. It covers the statistics item of the reporting checklist.

Common failure modes#

• Bare percentages — "SQLAgent: 80%" with no n or interval → report n, a Wilson interval, and per-slice counts. See one-number reporting.
• Trials counted as tasks — 50 tasks with 4 rollouts each reported as n = 200 → resample whole tasks or fit a hierarchical model. See correlated trials.
• The wrong functional — pass@k quoted for unattended work, or an all-k "pass@k" compared with an at-least-one pass@k → choose the functional from the decision, and translate labels before comparing.
• A fictional user base — the headline averages a suite that over-samples hard cases on purpose → stratify to the production mix and report $n_{\text{eff}}$.

Check yourself#

Analyze. Two papers report "pass@5" for coding agents, 62% and 41%. List what you need to know before you compare them.

Hint

Which functional (at least one of five, or all five)? Which estimator (unbiased per task, or plug-in)? How many samples per task, how many tasks, and what interval? Which task set and harness?

Evaluate. A team reports SQLAgent v2 at "65% (95% Wilson CI 58–71%, n = 200)" and concludes that it clears a 60% bar. The 200 outcomes are 4 rollouts on each of 50 tasks. Is the conclusion supported?

Hint

The interval treats rollouts as independent tasks. In this chapter's illustration, resampling whole tasks gives about 52–77%, which includes 60%.

Create. Write the estimand and reporting plan for a release gate on DocsAgent, the documentation agent of Case G, requiring 80% of help requests to be answered correctly and with grounding, with separate claims for version-sensitive and unsupported requests.

Hint

Name the population and its weights, the functional (a single-trial rate), and the slices you will claim. About 245 independent tasks give ±5 points at 80%; plan more if rollouts or judge calls cluster, and size each slice for its own claim.

Further reading#

• Task distributions and dataset engineering — where the strata, weights, and support come from.
• Comparing systems — the estimand for "is the candidate better?" and its paired test.
• Rare-event assurance — when the failures that matter almost never appear.
• The judge as a noisy sensor — correcting a pass rate that an imperfect judge graded.
• Chen et al. (2021), "Evaluating large language models trained on code" — the origin of the pass@k name and its unbiased estimator.
• Yao et al. (2024), "τ-bench: A benchmark for tool-agent-user interaction in real-world domains" — the source of the all-k pass^k.