Tools
Calculators for evaluation statistics
The arithmetic behind defensible eval results, one calculator per question: how uncertain a pass rate is, what pass@k and pass^k mean, whether one system really beats another, what a clean run rules out, how to correct a noisy judge, and how many alerts are real.
- Wilson confidence interval calculator for pass ratesThe Wilson 95% interval for an observed eval pass rate, and how many independent tasks a target half-width needs.4Validation
- pass@k and pass^k calculatorpass@k is the chance that at least one of k attempts succeeds, pass^k the chance that all k do. Compute both, plus the unbiased pass@k estimator.4Validation
- McNemar test calculator for comparing two systemsIs system A really better than system B on the same tasks? McNemar's test on the discordant pairs, plus the false-alarm rate across several gates.4Validation
- Rule of three calculator: zero failures in n trialsAfter n clean trials, how high could the failure rate still be? The rule of three, the exact 95% bound, and the trials a target bound needs.4Validation
- LLM judge noise correction calculator (Rogan–Gladen)Recover a system's true pass rate from the rate an imperfect judge reports, given the judge's true- and false-positive rates.4Validation
- Alert precision calculator for rare eventsHow many of a monitor's alerts are real when the event it watches for is rare: precision from the base rate and the monitor's error rates.3Regime
More interactive examples, from aggregation to exploitability, sit inside the chapters that explain them; the handbook lists them all.