Glossary · Signal and resolution
Unbiased pass@k estimator
For a task with c passes among n ≥ k samples, 1 − C(n − c, k)/C(n, k): the share of k-sample subsets containing a pass. Compute per task, then average; the plug-in 1 − (1 − c/n)^k reads low.
Defined in
Handbook · ValidationEstimands, pass rates, and intervalsWrite the estimand first, then report n, an honest interval and the right functional, from Wilson to pass^k to reweighted estimates.Related terms
- pass@k
- The probability that at least one of k sampled candidates succeeds: 1 − (1 − p)^k under independence. Not to be confused with pass^k.
- pass^k
- The probability that all k independent attempts succeed: p^k. The right quantity for unattended or repeated work. Some sources label it pass@k; check which functional a leaderboard computed.
Where it comes up
- Notation and formula cardHandbook · Reference
- pass@k versus pass^kCalculator