# Monitoring and control in production

> Monitoring is not continuous benchmarking: base rates, calibration, conformal caveats, and the moment a monitor becomes a training signal.

**Production monitoring** uses an evaluator to watch a deployed system and raise alerts, and, through runtime control, to block, reroute, roll back, or escalate. It is not continuous benchmarking: the estimand, the operating point, and the base rate all differ, and a monitor that training reacts to has become an optimization target.

> [!KEY] In one sentence
> Validate a monitor at the production base rate, calibrate its scores, state where its thresholds hold, detect traffic outside its validity domain, prefer structural control wherever a property can be guaranteed, and reclassify the monitor the moment training reacts to it.

## Not continuous benchmarking

Production monitoring is not merely benchmarking continuously: the estimand and operating point differ. A benchmark estimates a rate over a declared population, offline, to support a report or a release decision. A monitor decides on every live event, at the production base rate, within a latency and cost budget, and pays for its errors one false alarm or one missed event at a time. The regime's threats follow: production base-rate effects, calibration drift, alarm fatigue, latency, missing coverage, correlated incidents, and policy adaptation around the guardrail. For deployment, the Berkeley lecture asks for evaluation that is trustworthy, task-specific, and absolute ([Song, 2025](https://benchmaker.ai/handbook/reference/sources#song-2025), slide 5).

Take [Case F](https://benchmaker.ai/handbook/cases/production-safety-monitor), a tool-using agent that can send external messages. The questions its monitor must answer are per event and per day: of the messages it flags, how many really violate policy, and how many real violations slip past?

## The base-rate problem

Suppose a monitor flags dangerous events with [sensitivity](https://benchmaker.ai/glossary/sensitivity-and-specificity) $t$ and false-positive rate $f$, and dangerous events occur at base rate $p$. The share of alerts that are real is the monitor's **alert precision**:

$$
\Pr[\text{danger} \mid \text{alert}] = \frac{p\,t}{p\,t + (1 - p)\,f}
$$

where:

- $p$ is the base rate, the fraction of monitored events that are truly dangerous;
- $t$ is the monitor's true-positive rate on dangerous events;
- $f$ is its false-positive rate on harmless ones.

In words: real alerts divided by all alerts, where the harmless majority contributes $(1 - p)f$ false ones. When $p$ is tiny, even a small false-positive rate can dominate.

An illustrative monitor that catches 99% of dangerous events with a 1% false-positive rate looks excellent on a balanced test set. In production, at a base rate of 1 in 1,000 and 100,000 events a day, it raises about 1,098 alerts a day: 99 real and 999 false. Only 9.0% of its alerts are real, about ten false alarms for every real one, and it still misses about one dangerous event a day. Cutting the false-positive rate tenfold, to 0.1%, lifts alert precision only to about 50%.

Production monitors therefore need calibration at production base rates, not only balanced test accuracy, and their false-alarm cost must be measured, because false alarms feed alarm fatigue. How much a clean record can rule out is a separate question; see [Rare-event assurance](https://benchmaker.ai/handbook/validation/rare-events).

The calculator starts at this example. Move the base rate down to 1 in 100,000: alert precision falls to about 0.1%, while the number of alerts barely changes, because almost every alert is now false.

```widget
alert-precision
p: 0.001
tpr: 0.99
fpr: 0.01
volume: 100000
```

## Monitoring observes; control intervenes

A monitor observes. A control mechanism changes system behavior: it blocks an action, routes to a human, triggers rollback, or constrains reachability. Where a property can be guaranteed by construction, [structural controls](https://benchmaker.ai/glossary/structural-guarantee) can dominate statistical monitoring: read-only credentials, sandbox walls, and permission systems make certain bad states unreachable rather than merely detectable. The monitor's job is what construction cannot reach; see [Guarantee before you estimate](https://benchmaker.ai/handbook/practice/structural-guarantees).

Case F applies this in order: structural permissions make unauthorized destinations unreachable where possible, a trajectory check enforces "no external message until the user confirms", $\neg\,\text{send}\ \mathsf{U}\ \text{confirmed}$, and a learned semantic monitor handles only the residual policy interpretation (see [Case F](https://benchmaker.ai/handbook/cases/production-safety-monitor)). That trajectory check is the same [temporal property](https://benchmaker.ai/glossary/temporal-pattern) that would grade an offline trial, since invariants written this way are checkable both offline and online.

## Calibrated scores and honest thresholds

A monitor that emits a score is claiming a probability. **Calibration** makes the claim true: among events scored 0.7, about 70% should actually violate. A **reliability diagram**, which plots the observed violation rate against the predicted score, together with recalibration on held-out data, is how the claim is checked and repaired ([Kochenderfer et al., 2026](https://benchmaker.ai/handbook/reference/sources#kochenderfer-2026)). An uncalibrated 0.7 is a unitless number wearing a probability's clothes.

**Conformal prediction** turns a threshold into a stated guarantee (Kochenderfer et al., 2026). To bound the miss rate on true failures at level $\alpha$, score $n$ expert-labeled true failures with the monitor, then flag every event that scores at or above the threshold $\theta$:

$$
\theta = s_{(k)}, \qquad k = \lfloor (n + 1)\,\alpha \rfloor
$$

where:

- $s_{(1)} \le \dots \le s_{(n)}$ are the monitor's badness scores on the $n$ labeled true failures, sorted;
- $\alpha$ is the target miss rate;
- $\theta$ is the flag threshold.

In words: put the threshold where only about a fraction $\alpha$ of the known failures score lower. Under exchangeability, at most about $\alpha$ of future true failures score below it and slip past. With an illustrative 199 labeled failures and $\alpha = 0.05$, $k = 10$: flag everything that scores at least as high as the tenth-lowest failure.

The guarantee is honest about its limits. It is marginal, not per slice. It requires exchangeability, which production drift breaks, so recalibrate on a cadence and set tripwires on disagreement spikes. And it says nothing about false positives, which must be priced separately. Any such guarantee belongs to a declared validity domain and must be paired with drift detection and recalibration.

## Runtime validity-domain detection

Offline evidence has a boundary: the [validity domain](https://benchmaker.ai/glossary/validity-domain) in which it was gathered. A monitor needs to know when live traffic leaves it. Define features that indicate whether a request resembles the evaluated domain — task embedding distance, product area, language, retrieval-score profile, session length, tool availability, or state descriptors — and give the system an explicit policy for requests outside it: abstain, escalate, reduce permissions, request confirmation, or route to a safer process.

The detector is itself an instrument, so report its false-positive and false-negative behavior. A validity-domain detector that silently misses novel cases simply moves the unmeasured region behind a second unvalidated model.

The claim for DocsAgent, the documentation agent of [Case G](https://benchmaker.ai/handbook/cases/docs-agent), covers help requests "within the supported product corpus". A question about a product outside that corpus is out of domain. In an illustrative design, the policy routes it to human support, and the detector is validated on labeled in-domain and out-of-domain requests, with both error rates reported.

## When the monitor joins the loop

Two promotions change what a monitor is. The first moves it into the request path, where it blocks, rewrites, or escalates before the user sees the output. The engineering constraints flip — tight latency, per-call cost at production volume, and a false-positive rate that now directly damages the product — and the guardrail becomes part of the [system under test](https://benchmaker.ai/glossary/system-under-test), evaluated like any component.

The second moves it into optimization. If policy training responds to the monitor, the monitor is now an optimization target, and its validation obligations migrate toward the optimize regime: in Case F, once the policy is trained to avoid monitor alerts, the monitor needs adaptive-validity tests. A falling alert rate is then ambiguous, because guardrail circumvention is one of the regime's dominant failures: only an independent check can tell fewer violations from violations the monitor no longer sees. [Use regimes](https://benchmaker.ai/handbook/regime/use-regimes) covers this [regime migration](https://benchmaker.ai/glossary/regime-migration) in general.

> [!OPEN] Privacy and trace logging
> A monitor is only as good as the evidence it sees, and production evidence is users' data. How privacy constraints should limit what a monitor may log and inspect is not worked out in this handbook; the evaluator dossier asks only for raw traces or privacy-compatible summaries.

## In practice

Build a validation record for each production monitor.

1. **Write the estimand and the action.** Name the event, what an alert triggers, and the production base rate with its interval.
2. **Subtract what construction guarantees.** Remove unreachable states from the monitor's job, citing the mechanism that removes them.
3. **Measure error rates on production-like data.** Estimate the true- and false-positive rates on adjudicated samples, then compute alert precision, daily alerts, and daily misses at the production base rate and volume.
4. **Price the operating point.** Put a cost on a false alarm and on a miss, and set the threshold from those costs, not from balanced accuracy.
5. **Calibrate.** Draw a reliability diagram on held-out labeled data and recalibrate before anyone reads a score as a probability.
6. **Scope every threshold guarantee.** For a conformal threshold, record $n$, $\alpha$, the marginal scope, and the recalibration cadence.
7. **Draw the domain boundary.** Choose out-of-domain features and an explicit out-of-domain policy, and measure the detector's own error rates.
8. **Write the migration triggers.** A guardrail in the request path is part of the system under test; any training or prompt change driven by alerts moves the monitor into the optimize regime.

The artifact is a monitor validation record: base rate, operating point, alert precision and its costs, calibration evidence, the threshold rule and its scope, the out-of-domain policy with the detector's error rates, and the migration triggers.

## Common failure modes

- **Balanced-test-set validation** — 99% accuracy on a balanced set, and a pager full of false alarms in production → validate at the production base rate and report alert precision with daily counts. See [one-number reporting](https://benchmaker.ai/handbook/practice/anti-patterns#one-number-reporting).
- **A score read as a probability** — thresholds set on an uncalibrated score → check a reliability diagram and recalibrate on held-out data.
- **A guarantee outliving its domain** — a conformal threshold kept through drift → recalibrate on a cadence and tripwire on disagreement spikes.
- **A monitor the policy trains against** — alerts feed training or prompt changes, and the alert rate falls → reclassify the monitor as optimize and audit it with an independent check. See [Goodharting](https://benchmaker.ai/handbook/practice/anti-patterns#goodharting).

## Check yourself

**Analyze.** A vendor reports that its detector for policy-violating outgoing messages is "99% accurate." About one message in 10,000 violates policy. List what you need before deploying it as a blocking guardrail, and compute its alert precision under an assumption you state.

> [!DETAILS] Hint
> Accuracy hides the base rate. You need the true- and false-positive rates separately, measured on traffic like yours. If the detector caught 99% of violations with a 1% false-positive rate, about 1% of its blocks would be real.

**Evaluate.** Your team proposes fine-tuning the agent on the messages the monitor flagged, to cut the alert rate. What does this change about the monitor's regime, and what can the alert rate support afterward?

> [!DETAILS] Hint
> Training now responds to the monitor, so it is an optimization target. A falling alert rate fits both fewer violations and violations the monitor no longer detects; an adjudicated audit sample can tell them apart.

**Create.** Design DocsAgent's runtime validity-domain check: the features, the out-of-domain policy, and the validation of the detector itself.

> [!DETAILS] Hint
> Start from the claim's boundary, "within the supported product corpus." Candidate features include product area, retrieval-score profile, and language; validate the detector on labeled in-domain and out-of-domain requests and report both error rates.

## Further reading

- [Guarantee before you estimate](https://benchmaker.ai/handbook/practice/structural-guarantees) — making bad states unreachable instead of monitoring for them.
- [Rare-event assurance](https://benchmaker.ai/handbook/validation/rare-events) — what zero failures in $n$ trials can and cannot rule out.
- [Use regimes: same judge, four jobs](https://benchmaker.ai/handbook/regime/use-regimes) — the causal rule and regime migration in general.
- [Case F: production safety monitor](https://benchmaker.ai/handbook/cases/production-safety-monitor) — structural permissions, a precedence check, and a residual learned monitor, end to end.
- [Kochenderfer et al. (2026)](https://benchmaker.ai/handbook/reference/sources#kochenderfer-2026), *Algorithms for Validation* — the validation-algorithms framework this chapter follows for calibration, conformal thresholds, and runtime assurance.

---
Source: https://benchmaker.ai/handbook/regime/production-monitoring