Skip to content
Benchmaker

Calculator

pass@k and pass^k calculator

pass@k is the chance that at least one of k attempts succeeds, pass^k the chance that all k do. Compute both, plus the unbiased pass@k estimator.

Interactivepass@k versus pass^k
75%
3

pass@3: at least one of 3 succeeds

98.4%

1 − (1 − p)^k. Useful when a reliable check, or a person, picks out the success.

pass^3: all 3 succeed

42.2%

p^k. The reliability of unattended or repeated work.

pass@k = 1 - (1-p)^kpass^k = p^k
0%25%50%75%100%05101520Attempts k · current k = 3Probability
  • pass@k (at least one)
  • pass^k (all k)

Convention

Here pass@k means at least one of k attempts succeeds, and pass^k means all k succeed. Some sources, including write-ups of τ-bench results, call the all-k quantity “pass@k”. Check the definition before comparing numbers.

Estimate pass@k from samples

You drew n samples for one task and c of them passed.

10
3
5

Unbiased pass@5

91.7%

The share of k-subsets of your n samples that contain at least one pass.

Plug-in 1 − (1 − c/n)^k

83.2%

Concave in c/n, so on average it reads low (Jensen's inequality).

Plug-in's average error

−7.2 pts

Expected plug-in minus true pass@k if the true rate were c/n. The unbiased estimator's average error is zero.

widehatpass@k = 1 - binomn-ck Big/ binomnk

Needs at least k samples per task. Estimate per task, then average across tasks; for an interval, resample tasks, not samples.

Unattended or repeated work needs pass^k. At p = 75%, 3 attempts contain a success 98.4% of the time, which helps only if something reliably picks it out, but all 3 succeed only 42.2% of the time. Both curves assume independent attempts at one rate. Across a suite, difficulty varies by task, so compute per task and average: with uneven difficulty, average pass@k comes out lower and average pass^k higher than these curves at the mean rate.

pass@k is the probability that at least one of k attempts succeeds; pass^k is the probability that all k succeed. They answer different questions, and the gap between them widens as k grows. Set the per-attempt success rate and k to compare them, then estimate pass@k from your own samples.

What it computes#

For independent attempts at one success rate pp:

pass@k=1−(1−p)k,passk=pk\text{pass@}k = 1 - (1 - p)^k, \qquad \text{pass}^k = p^k

pass@k is useful when a reliable check, or a person, picks out the success from k tries. pass^k is the reliability of unattended or repeated work, where every run has to succeed.

Estimating pass@k from samples#

If you drew nn samples for one task and cc of them passed, the unbiased estimator of Chen et al. (2021) is

pass@k^=1−(n−ck)/(nk)\widehat{\text{pass@}k} = 1 - \binom{n - c}{k} \Big/ \binom{n}{k}

with n≥kn \ge k. The plug-in 1−(1−c/n)k1 - (1 - c/n)^k reads low on average. Compute the estimate per task, average across tasks, and give it an interval by resampling tasks.

Cautions#

  • Both formulas assume independent attempts at one rate. When difficulty varies across tasks, compute them per task and then average.
  • Some sources label the all-k quantity "pass@k"; check which one a reported number means before comparing it.
  • The handbook gives no unbiased estimator for pass^k.

The chapter Estimands, pass rates, and intervals develops both quantities.

Terms on this page

The chapter behind it

Handbook · ValidationEstimands, pass rates, and intervalsWrite the estimand first, then report n, an honest interval and the right functional, from Wilson to pass^k to reweighted estimates.

More calculators