Glossary · Signal and resolution
pass^k
The probability that all k independent attempts succeed: p^k. The right quantity for unattended or repeated work. Some sources label it pass@k; check which functional a leaderboard computed.
Defined in
Handbook · ValidationEstimands, pass rates, and intervalsWrite the estimand first, then report n, an honest interval and the right functional, from Wilson to pass^k to reweighted estimates.Related terms
- pass@k
- The probability that at least one of k sampled candidates succeeds: 1 − (1 − p)^k under independence. Not to be confused with pass^k.
Where it comes up
- Evaluating agentsHandbook · Agents and self-improvement
- Reading agent benchmarksHandbook · Agents and self-improvement
- A: Mathematical reasoningHandbook · Case studies
- B: Coding agentHandbook · Case studies
- Notation and formula cardHandbook · Reference
- pass@k versus pass^kCalculator