Skip to content

Glossary · Reliability, calibration, and statistics

Optional stopping

Checking a comparison repeatedly and stopping once it looks significant. Each look is another chance for noise to cross the threshold, so false positives inflate; use a fixed sample size or an explicitly sequential design.

Defined in

Handbook · ValidationComparing systemsPair tasks, count discordant pairs, size for the smallest difference that matters, and never let significance stand in for validity.

See it in the full glossary →