# Falsification and failure regions

> Estimation counts failures; falsification finds them. Search for the most likely breaks and map the region, not the anecdote.

**Falsification** is the search for inputs and conditions that make a system violate its specification. Random testing asks how often failures occur under a sampling distribution; falsification asks whether a failure can occur at all, and what makes it happen. A robust evaluation program needs both.

> [!KEY] In one sentence
> Estimation counts failures and falsification finds them: search for the most likely breaks with a graded margin, then map the region around each one instead of collecting anecdotes.

## Two questions, two procedures

Take the release evaluator of SQLAgent, the text-to-SQL agent of [Case C](https://benchmaker.ai/handbook/cases/sql-data-agent). A random sample of 400 logged analyst questions estimates its pass rate, and the interval says how precisely *(illustrative)*. If none of them fails, the [rule of three](https://benchmaker.ai/glossary/rule-of-three) puts a rough 95% upper bound of 3/400, under 1%, on the failure rate under that sampling distribution. The sample cannot say whether some phrasing of a date range, an unfamiliar table, or a tool timeout in the middle of a query makes the agent return wrong rows. That question needs a search, not a sample.

| | Estimation | Falsification |
|---|---|---|
| Question | How often does it fail under my sampling distribution? | Can it fail at all, and where? |
| Procedure | Sample from the target population and count | Search over inputs and disturbances |
| Output | A rate with n and an interval | Counterexamples, minimal reproductions, failure regions |
| What it cannot tell you | Whether rare or adversarial failures exist | How often they occur |

Falsification has two targets. The first is the [system under test](https://benchmaker.ai/glossary/system-under-test): inputs and disturbances under which it violates the [specification](https://benchmaker.ai/glossary/specification). The second is the evaluator itself. Searching for high-scoring failures and evaluator blind spots is a step of its own in building an evaluator, and a learned judge should face adversarial cases and high-scoring failures before it is trusted. The Berkeley course gives benchmark builders the same advice: an eval can be gamed through shortcuts, so remove the ones it probably has, and write hard test cases that show the evaluating agent is reliable ([Song, 2025](https://benchmaker.ai/handbook/reference/sources#song-2025), slide 46).

In the framework's coordinates, red teaming is a validation stress test whose product is falsification search and boundary evidence. When the search is run by the actual optimizer at measured budgets, it becomes an exploitability study ([Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity)).

## Search needs a slope

A specification's [robustness margin](https://benchmaker.ai/glossary/robustness-margin) $\rho(\psi, \tau)$ is positive when a [trajectory](https://benchmaker.ai/glossary/trajectory) $\tau$ satisfies the specification $\psi$ and negative when it violates it, and its magnitude says by how much (see [Specification](https://benchmaker.ai/handbook/instrument/specification)). With a margin in hand, falsification is an optimization problem:

$$
\min_{x,\,\delta}\ \rho\big(\psi,\ S(x;\delta)\big)
$$

where:

- $x$ is an input from the task space, such as an analyst question;
- $\delta$ is a disturbance: a paraphrase, a tool failure, a state perturbation, or an adversarial suffix;
- $S(x;\delta)$ is the trajectory the system under test produces on $x$ under $\delta$;
- $\rho(\psi, \tau)$ is the robustness margin of that trajectory against the specification.

In words: find the input and disturbance that push the system furthest toward violating the specification. Any negative value is a counterexample.

Search methods include fuzzing, evolutionary search, gradient-based optimization when a smooth surrogate exists, planning, and attacker agents ([Kochenderfer et al., 2026](https://benchmaker.ai/handbook/reference/sources#kochenderfer-2026)). All of them need something to descend, and a hard pass/fail grader is a step function no optimizer can descend, which is one more reason graders should emit margins.

Most binary checks have a natural margin nearby: score minus threshold, budget minus usage, distance from the unsafe set. Take a bounded-response rule for SQLAgent: every database error is fixed or escalated within three retries. Its margin is the number of retries left. A search that drives that number toward zero is closing in on a violation that a pass/fail grade would reveal only after it happened.

Search also makes demands on the [eval harness](https://benchmaker.ai/glossary/eval-harness). Each family of methods needs a capability the harness must provide, so decide which adversaries the risk profile requires before you build it:

| Search method | What the harness must support |
|---|---|
| Direct sampling, fuzzing, derivative-free search | Black-box batch rollouts |
| Disturbance sweeps | Hooks that corrupt a tool result, delay an API call, or change retrieved context mid-episode |
| Tree search over conversation actions | Snapshot and restore, since a search cannot branch an episode it cannot fork |
| RL-style attackers | Cheap resets and per-step margins |
| Regression on found failures | Deterministic replay of the environment |

## Two adversary regimes

Not every counterexample is equal evidence. A long adversarial suffix of Unicode noise and a plausible, typo-ridden question typed at five in the morning can both break an agent, but they are not the same discovery. Validation theory sharpens the objective: look for the most likely failure, the violating trajectory with the highest probability under the nominal distribution (Kochenderfer et al., 2026). Written as an optimization problem in this handbook's notation:

$$
\begin{aligned}
\tau^{*} = {} & \arg\max_{\tau}\ \log p_{0}(\tau) \\
& \text{subject to}\ \ \rho(\psi, \tau) < 0
\end{aligned}
$$

where:

- $p_0$ is the nominal distribution over trajectories: the inputs and disturbances production actually serves;
- $\rho(\psi, \tau) < 0$ says that $\tau$ violates the specification;
- $\tau^{*}$ is the most likely failure.

In words: among all the ways to break the specification, find the one the world is most likely to produce. A softer version descends a combined objective of robustness plus log-likelihood, and a further variant weights the search by severity where actions differ in blast radius.

Whether to weight by plausibility depends on who chooses the inputs. Separating two **adversary regimes** resolves a standing confusion in industry red-teaming:

- **Nature.** Ordinary users and honest traffic. Weight the search by plausibility, because the failures worth engineering effort are the ones production will serve. Finds reweight the dataset and drive fixes ranked by expected loss.
- **Strategic adversary.** Jailbreakers and prompt injectors. Drop the weight: the attacker chooses the input, so the existence of a worst case is the relevant fact, and implausibility is no defense. Finds drive guardrails and permanent counterexamples.

Run both searches, label every find by regime, and route the finds differently. For SQLAgent, a confusing fiscal-calendar question is a nature-regime find. An instruction planted in a table comment that tells the agent to read a restricted column is an adversary-regime find *(both illustrative)*, and the right response is a permission that makes the read unreachable, not a reweighted test set ([Guarantee before you estimate](https://benchmaker.ai/handbook/practice/structural-guarantees)).

## From a failing input to a failure region

A single adversarial example is useful for debugging but weak for understanding risk. Once a search finds a failure, map its neighborhood:

1. **Minimal reproduction.** Find the smallest plausible edit that flips the outcome. Prefer sparse, plausible, actionable changes: a "minimal" reproduction reached through an implausible edit explains nothing.
2. **Clusters and prototypes.** Cluster the failure traces and read the trace nearest each cluster center, not a random sample of everything.
3. **Predictive features.** Fit which task features predict failure, and treat the fitted coefficients as leads for the next search, not as causal conclusions.
4. **Conditional sampling.** Where a generative environment allows it, start from a known failure and propose local, plausibility-weighted mutations that remain failures. This maps the connected region, not one point.
5. **Severity ladders.** Arrange cases from benign to harmful.

The goal is to infer a failure region, not collect a museum of anecdotes. A **failure region** is the set of inputs and conditions around a found failure that also fail, described by the features that define it.

An illustrative map for SQLAgent: a search finds that "revenue last quarter" returns calendar-quarter rows from a warehouse that reports by fiscal quarter. The minimal reproduction is one phrase, since naming the fiscal quarter explicitly gives the correct result set. Clustering the other finds shows the same pattern for "year to date" and "month end", the feature model points at relative date phrases over fiscal-calendar tables, and conditional sampling from the first failure maps a region of relative-date questions. That region is a [construct facet](https://benchmaker.ai/glossary/construct-facet) that Case C lists for SQLAgent, date and business-calendar semantics, and the task set under-samples it. The fix is a new slice weighted to its production share, not one more test case.

Mapping also serves construct validity: repeated local failures can reveal the hidden nuisance feature a benchmark is actually measuring. A region defined by question length or column naming, rather than by anything the construct includes, belongs on the [nuisance list](https://benchmaker.ai/glossary/nuisance-variable), which is the discriminant test plan ([Constructs, facets, and nuisances](https://benchmaker.ai/handbook/claim/constructs)).

## What a found failure can and cannot tell you

Falsification deliberately gives up the ability to count: a failure found by search says nothing about frequency, because the search went looking for it. When a rate is needed too, return to weighted estimation: sample the failure region at its production weight, or use importance sampling or multilevel splitting where a simulator supports them ([Rare-event assurance](https://benchmaker.ai/handbook/validation/rare-events)).

> [!WARNING]
> A red-team funnel's end count is not a rate. Multiply the conditional rates along the funnel's stages, and never report the final count as if it came from uniform sampling. See [one-number reporting](https://benchmaker.ai/handbook/practice/anti-patterns#one-number-reporting).

Falsification also explains when feedback helps. In code self-repair, a failing test is an executable counterexample. The RSI survey reports a placebo-controlled study that credits the counterexample, not renewed exposure to the problem, with the value of feedback, and a controlled decomposition that separates real feedback value from resampling, format correction, and extra test-time compute, all of which inflate naive comparisons ([Chen, Wang & Qu, 2026](https://benchmaker.ai/handbook/reference/sources#chen-wang-qu-2026), §3.2). The design transfers: before crediting an evaluator's feedback with an improvement, compare it against a control that shows the system the task again without the counterexample.

> [!OPEN]
> Adaptive search changes the statistical object. Evaluations that choose the next task from a system's earlier behavior gain sample efficiency, but their observations are no longer an i.i.d. sample from a fixed distribution, and clear estimands and inference for adaptive task generation remain open.

## In practice

Run a falsification campaign alongside the estimation suite, not instead of it.

1. **Give each property a margin.** For every property you will search against, write the specification and its natural margin, and replace pass/fail checks with margins where a slope exists.
2. **Name the adversary regimes.** Decide whether nature, a strategic adversary, or both choose inputs in this deployment, and which disturbances belong to each: paraphrases, tool failures, state perturbations, injected instructions.
3. **Build the harness the search needs.** Add batch rollouts, disturbance hooks, snapshot and restore, cheap resets, and deterministic replay as the chosen methods require.
4. **Search the system and the evaluator.** Look for violations by the system and for failures the evaluator scores as passes, and label every find by regime.
5. **Map each region.** Reduce every find to a minimal reproduction, cluster the traces, fit predictive features, sample conditionally where the environment allows, and grade severity.
6. **Route the finds.** Nature-regime regions become weighted slices and fixes ranked by expected loss; adversary-regime finds become guardrails, structural controls, and permanent regression cases.
7. **Estimate separately.** If a rate is needed, estimate it with a weighted design, never from search counts.

The artifact is a failure-region register: for each region, its minimal reproduction, defining features, adversary regime, severity, estimated rate or "not estimated", and the slice or control it produced.

## Common failure modes

- **Search counts reported as rates** — "12 jailbreaks in 1,000 attempts" becomes a 1.2% failure rate → report finds as evidence of existence, and estimate rates with a weighted design. See [one-number reporting](https://benchmaker.ai/handbook/practice/anti-patterns#one-number-reporting).
- **A museum of anecdotes** — the list of failing prompts grows, but nobody can say what they share → reduce each to a minimal reproduction and map the region before adding more.
- **Searching a step function** — pass/fail graders give the search nothing to descend, so it degrades to random sampling → emit margins. See [Specification](https://benchmaker.ai/handbook/instrument/specification).
- **The grader is never searched** — only the system is red-teamed, and the optimizer finds the evaluator's blind spots first → red-team the grader before the system trains against it. See [grader gaming](https://benchmaker.ai/handbook/practice/anti-patterns#grader-gaming).

## Check yourself

**Analyze.** A red-team report lists 40 successful jailbreaks of a support agent found over two weeks and concludes that the agent "fails 4% of the time". Separate what the report supports from what it does not.

> [!DETAILS] Hint
> The finds are evidence that strategic-adversary failures exist. The 4% has no sampling distribution behind it. Ask for minimal reproductions and regions and, if a rate matters, a weighted estimate.

**Evaluate.** A coding agent is graded by whether a hidden test suite passes. A colleague proposes to falsify it by random search over rephrasings of the issue text. Judge whether the search can work as designed, and what you would change.

> [!DETAILS] Hint
> A single pass/fail verdict gives the search no slope. Look for natural margins, such as the share of tests passed or the time budget left, and decide which disturbances belong to which adversary regime.

**Create.** Design a falsification campaign for the "grounded answer" criterion of DocsAgent, the documentation agent of Case G: the margin you will search against, the disturbances for each adversary regime, the harness capabilities you need, and how finds become slices or controls.

> [!DETAILS] Hint
> Candidate margins include the judge's score minus its threshold and the share of claims with a supporting passage. Changing the retrieved context mid-episode needs a disturbance hook; documents from an outdated product version are a nature-regime disturbance.

## Further reading

- [Rare-event assurance](https://benchmaker.ai/handbook/validation/rare-events) — turning a failure region into a rate with weighted sampling, importance sampling, and splitting.
- [Specification](https://benchmaker.ai/handbook/instrument/specification) — robustness margins and temporal patterns that give a search its slope.
- [Adaptive validity](https://benchmaker.ai/handbook/pressure/adaptive-validity) — the same search run by the real optimizer, at measured budgets.
- [Task validity and outcome validity](https://benchmaker.ai/handbook/validation/task-and-outcome-validity) — checks that catch shortcuts and weak graders before a search has to.
- Kochenderfer, Katz, Corso, and Moss (2026), *Algorithms for Validation* — the validation-algorithms framework behind the handbook's falsification and failure-region machinery.
- Chen, Wang, and Qu (2026), "Recursive self-improvement in AI" — §3.2 on crediting the value of feedback to falsification in code self-repair.

---
Source: https://benchmaker.ai/handbook/pressure/falsification