Rare-event assurance

Rare-event assurance is the work of supporting a claim that a failure is very unlikely, so unlikely that an evaluation run of ordinary size will probably never show one. A zero-failure test does not establish zero risk: it bounds the failure rate at a level set by the number of trials.

The arithmetic of rare failures#

A clean test set is weak evidence for extremely rare harms. Seeing even one failure at rate $p$ takes about $1/p$ trials, and pinning $p$ down to within roughly 10% of its value takes about $100/p$ (Kochenderfer et al., 2026). The last column below anticipates the next section: the number of clean, independent trials whose exact 95% upper bound reaches the rate.

A claim that a catastrophic failure occurs less than once in a million trials would need millions of representative clean trials if sampling were the only evidence. Sampling alone cannot carry such a claim; the rest of this chapter is about what can.

Some claims concern how bad the worst outcomes are rather than how often failure occurs, and tail functionals answer them: VaR is the α-quantile of a loss, and CVaR, the mean loss beyond that quantile, is the gate this handbook recommends for harm-adjacent qualities (see aggregation geometry). Their estimates meet the same arithmetic: at the 0.99 level, 500 trials leave five observations in the tail.

What zero failures buys#

With zero failures in $n$ independent trials, a rough 95% upper bound on the failure probability is

$$p_{\text{fail}} \lesssim \frac{3}{n}$$

where:

• $p_{\text{fail}}$ is the true probability that one trial fails;
• $n$ is the number of independent, representative trials, none of which failed.

In words: after n clean trials, the failure rate could still be as high as about three in n without the clean run being surprising. This is the rule of three. The exact one-sided bound solves $(1 - p_{\text{fail}})^n = 0.05$, which gives $1 - 0.05^{1/n}$; because $-\ln 0.05 \approx 3$, that is close to $3/n$.

The rule has a Bayesian reading, which is strictly more useful. Updating a uniform prior on n clean trials gives a Beta(1, n + 1) posterior, and its 95% upper credible bound is $1 - 0.05^{1/(n+1)} \approx 3/(n+1)$. The posterior gives a whole curve rather than one bound, accepts an informative prior, and still works after one or two failures, where the rule of three stops. With $s$ failures the posterior is Beta(1 + s, 1 + n − s), the Beta-binomial update applied to failures. For an illustrative run of 500 trials:

A single failure raises the bound by more than half.

A drill makes the stakes concrete. A team proposes a 99.9% reliability requirement, a failure rate of at most 0.1%, and offers a clean 500-task run as evidence. The run bounds the failure rate at about 0.6% (exact one-sided bound 0.597%), six times the requirement. A system three times worse than the requirement, failing 0.3% of trials, would produce a clean 500-task run about 22% of the time (illustrative). Sampling alone would need about 2,995 clean trials, and if one failure appeared, about 4,740 trials in all.

The calculator starts at the drill's clean 500-task run and its 0.1% target. Drag n toward 3,000 and the bound reaches the target; then set the target to 10⁻⁶ and the clean trials needed approach three million.

Every number in this section assumes independent trials drawn from the distribution the claim is about. When trials share infrastructure, dependencies, or state, their outcomes move together and independent-sample calculations no longer hold; repeated runs of one task are clustered too. Either way the bound is too optimistic. A clean run on tasks unlike production bounds the failure rate on those tasks only.

Oversample, then correct#

When naive sampling cannot reach the rate, change what you sample. The simplest version oversamples high-risk slices and reweights them to the population, as in stratified estimates. Importance sampling generalizes it: sample tasks from a proposal $q$ chosen to produce failures more often, then weight each outcome by its likelihood ratio, which keeps the estimate unbiased for the nominal distribution $p$ (Kochenderfer et al., 2026):

$$\begin{aligned} P_f &= \mathbb{E}_{q}\!\left[\frac{p(x)}{q(x)}\, \mathbb{1}\{\text{fail}(x)\}\right] \\ &\approx \frac{1}{N} \sum_{i=1}^{N} \frac{p(x_i)}{q(x_i)}\, \mathbb{1}\{\text{fail}(x_i)\} \end{aligned}$$

where:

• $P_f$ is the failure probability under the nominal distribution;
• $p(x)$ is the probability (or density) of task $x$ under the nominal distribution, the claim's population; here $p$ is not a pass rate;
• $q(x)$ is its probability under the proposal, from which the $N$ tasks $x_i$ are drawn;
• $\mathbb{1}\{\text{fail}(x)\}$ is 1 if the system fails on $x$ and 0 otherwise.

In words: sample where failures are common, then shrink each failure's weight by how much more often the proposal produced its task than the population would. Two caveats apply: the proposal must put probability on the whole failure region, and a poorly matched proposal makes the variance blow up.

Multilevel splitting attacks the same problem differently. It writes a very small failure probability as a product of larger conditional probabilities, one per severity level (Kochenderfer et al., 2026). With a robustness-style objective $f$ that is at or below zero exactly when the system fails, and a ladder of thresholds $\gamma_1 > \gamma_2 > \dots > \gamma_L$:

$$p_{\text{fail}} = \prod_{\ell=1}^{L} \Pr\big[f \le \gamma_\ell \mid f \le \gamma_{\ell-1}\big]$$

where:

• $f$ is the objective, such as a robustness margin, with failure exactly when $f \le 0$;
• $\gamma_\ell$ are the thresholds, the last one at the failure boundary ($\gamma_L = 0$), with $\gamma_0$ above every outcome so that the first factor is unconditional;
• each factor is estimated by plain sampling among the trials that reached the previous level.

In words: instead of waiting for a rare failure, estimate a chain of common events, such as how often a trial comes near the boundary and how often a near miss comes nearer, and multiply them.

Splitting is the honest version of a staged red-team funnel, which multiplies the share of outputs that are concerning, the share of those that come near a violation, and the share of near-violations that become violations. The rule is blunt: multiply the conditionals, and never report the funnel's end count as if it came from uniform sampling. Search is not counting either. Adversarial search finds failures efficiently, but a failure found by search carries no information about how often it occurs, which is why falsification maps failure regions rather than rates.

Prevent, search, bound, monitor#

Naive Monte Carlo is expensive at very low failure probabilities, so serious safety assurance combines methods: make failures unreachable by construction, hunt for counterexamples adversarially, oversample risky slices and reweight, use importance sampling or splitting where a simulator allows, and monitor production while counting exposure. The correct overall shape is prevent, search, bound, monitor, not "run a bigger random benchmark."

Case F shows the four steps on one claim: a tool-using agent can send external messages, and no message may reach an unauthorized destination.

The order matters. Whatever construction guarantees no longer needs samples, so the sampling budget goes to the properties only measurement can reach, and the safety case says, property by property, which are guaranteed (naming the mechanism) and which are estimated (giving n and the interval). Monitoring closes the loop, and its alerts need calibration at production base rates, not only on a balanced test set.

In practice#

Build a rare-event safety case for each property whose failure must be very unlikely.

1. State the rate. Translate "99.9% reliable" into a failure rate per trial, task, or session, over a named population.
2. Prevent what you can. Make the failure unreachable by construction where the claim allows it, and move those properties out of the sampling budget, citing the mechanism.
3. Search. Run adversarial search on what remains, and record failure regions and minimal reproductions, not counts.
4. Bound. Report the residual failure rate with n and a 95% upper bound, from the rule of three or the Beta posterior once failures appear, and state the independence and representativeness assumptions. Where a simulator supports it, use importance sampling or splitting, with their caveats.
5. Monitor. Count production exposure, validate alerts at production base rates, and revisit the case when the validity domain drifts.

The artifact is a rare-event safety case: every property marked guaranteed (with its mechanism) or estimated (with n, the bound, and its assumptions), the search record, and the monitoring plan with its exposure accounting.

Common failure modes#

• Zero failures reported as zero risk — "no incidents in 500 runs" → report the bound, under 0.6% at 95%, and the trials it rests on. See one-number reporting.
• Correlated clean runs — runs that share a sandbox image, a seed, or a tool mock are counted as independent trials → count independent units and model the clustering. See correlated trials.
• The funnel's end count as a rate — violations found by a staged red-team funnel are divided by the prompts it started with and reported as the deployment rate → estimate each conditional on the nominal distribution and multiply, or report the search as search.
• A bigger random benchmark as the plan — more uniform samples for a 10⁻⁶ claim → prevent, search, bound, and monitor instead.

Check yourself#

Analyze. A dashboard reports "0 incidents in 200 runs: incident rate 0%". The runs are 50 tasks with 4 seeds each, in one shared sandbox. What can be claimed?

Hint

Even for 200 independent trials, the 95% bound would be about 1.5%, not 0%. These runs cluster by task and share infrastructure, so they carry less evidence than 200 independent trials; if all the variation were between tasks, they would count closer to 50, a bound near 6%.

Evaluate. A red-team report says an attacker agent found 3 policy violations in 50,000 attempts and concludes that the violation rate is about 0.006%. What does the evidence support?

Hint

That violations are reachable, and where. The attempts were chosen by a search, not drawn from deployment, so the count says nothing about frequency; a rate needs likelihood-ratio weights relative to the nominal distribution, or a splitting design, with their assumptions stated.

Create. For the drill above, a 99.9% requirement with a clean 500-task run in hand, design a layered assurance program using construction, search, estimation, and monitoring.

Hint

Partition the requirement's failure modes into those construction can make unreachable (cite the mechanism) and the rest. Search the rest adversarially; for the residual, plan about 3,000 clean representative trials or oversample with weights; and monitor production with exposure accounting.

Further reading#

• Guarantee before you estimate — making failures unreachable rather than rare.
• Falsification and failure regions — searching for failures and mapping where they live.
• Monitoring and control in production — base rates, alerts, and exposure once deployed.
• Aggregation geometry — VaR, CVaR, and tail-sensitive gates.
• Kochenderfer, Katz, Corso, and Moss (2026), Algorithms for Validation — the validation-algorithms framework behind the rare-event machinery.