Alert precision calculator for rare events

A monitor with a 1% false-positive rate sounds accurate until the event it watches for happens once in ten thousand checks. Alert precision is the share of alerts that are real at the production base rate. Set the base rate, the volume of checks and the monitor's error rates to see how many alerts are real and how many events slip through.

What it computes#

With base rate $p$ of truly dangerous events, true-positive rate $t$ on dangerous events and false-positive rate $f$ on harmless ones,

$$\Pr[\text{danger} \mid \text{alert}] = \frac{p\,t}{p\,t + (1 - p)\,f}$$

When $p$ is tiny, the $(1 - p)\,f$ term dominates, so even a small false-positive rate can make most alerts false alarms.

What follows#

• Validate a monitor at the production base rate, not on a balanced test set: precision measured at equal prevalence does not transport.
• The calculator also counts alerts and missed events per day at your volume, which is what the people answering the alerts experience.

The chapter Monitoring and control in production puts this arithmetic to work.