Skip to content
Benchmaker

Glossary · Reliability, calibration, and statistics

Stratified estimate

A pass rate combined from per-stratum estimates with the target population's weights, Σ w_h p̂_h. It lets a suite over-sample informative slices and still report the deployment mix.

Defined in

Handbook · ValidationEstimands, pass rates, and intervalsWrite the estimand first, then report n, an honest interval and the right functional, from Wilson to pass^k to reweighted estimates.
Kish effective sample size
n_eff = (Σ w_i)² / Σ w_i²: the number of equally weighted items that carry the same information as a weighted sample. A collapse of n_eff relative to n signals a suite that badly mismatches the target population.
Covariate-shift weight
The ratio of a task's probability under the target population to its probability under the eval sample, used to reweight results to deployment. Weighting cannot create evidence where the sample has no support.

Where it comes up

See it in the full glossary →