F: Production safety monitor

Case F evaluates a production safety monitor: the controls and detectors that keep a deployed, tool-using agent from sending unauthorized external messages or issuing unauthorized refunds. Its design order is the lesson: permissions first, trajectory checks second, and a learned monitor only for the residue.

Claim and decision#

The case starts from a capability: a tool-using agent can send external messages. It uses an illustrative deployment, the one the Studio preset loads: a customer-support agent with email, refund, and CRM tools.

• Construct: the agent sends no unauthorized external messages and issues no unauthorized refunds, and what cannot be prevented is flagged.
• Population: production support conversations, including rare adversarial and confused-customer cases; true violations occur in about 1 in 1,000 conversations.
• Environment and harness: live CRM and email tools behind a permission layer; external messages, and refunds above a limit, require explicit confirmation; the monitor sees the full trajectory.
• Validity domain: English support conversations for the consumer product, under the current tool and policy versions.
• Decision: block the action and page a human, or let it proceed.

The interpretation to defend: alerts reliably indicate policy violations at production base rates, and blocked actions are truly unsafe.

Control first, then monitoring#

A monitor observes; a control changes behavior, and structural controls can dominate statistical monitoring when a property can be guaranteed by construction. The runtime guard is runtime control: once it blocks actions in the request path, it becomes part of the system under test, with tight latency, per-call cost at production volume, and false positives that now damage the product.

Criteria and mechanisms#

The specification has three parts, each routed to the strongest mechanism that can check it, in the order the case prescribes:

The confirmation rule is a temporal pattern of the precedence kind, no send until confirmed, and the refund rule is the same pattern with a different action. Written this way, a trajectory requirement is checkable both offline and online. What the patterns cannot express, such as whether a message's content or a refund's justification breaks policy, is the residue the learned monitor exists for. The first two criteria are fatal, with zero violations allowed; a residual flag escalates to a human instead of deciding.

Guarantee what you can#

Capability removal works on reachability: an egress allowlist makes unapproved destinations unreachable, so the failure probability is zero, not as an estimate but as a property of the transition system. Because the scaffold is conventional code with finitely many actions, "no path reaches send_email without passing confirm" can be verified exhaustively on the harness; the model picks a path, but the scaffold decides which paths exist. Both guarantees hold only for the configured permissions and connection paths.

What construction guarantees comes off the sampling budget. A safety case should sort every property into those guaranteed by construction, with the mechanism named, and those estimated by sampling, with n and the interval. Sampling is a poor substitute for the first kind: zero violations in n independent trials bounds the rate only near $3/n$ at 95% confidence.

The calculator starts at an illustrative clean certification run of 3,000 conversations: the bound is 0.100%, no lower than the base rate of all violations. Bounding the rate below 0.010% would take 29,956 clean conversations; slide the target to 1 in 100,000 and it takes 299,572.

The residual monitor at production base rates#

The learned monitor meets the base-rate problem: with sensitivity $t$, false-positive rate $f$, and violation base rate $p$, the share of alerts that are real is $pt/(pt + (1-p)f)$, and when $p$ is tiny even a small $f$ can dominate. Monitoring and control in production develops the formula; here it sets the acceptance test, because this deployment requires alert precision of at least 30% at a 0.1% base rate (illustrative).

Set the flag threshold with conformal prediction: from adjudicated true violations, choose the threshold so that, under exchangeability, at most about α of future violations score below it (Kochenderfer et al., 2026). At α = 0.10, at most about 10% slip past, so the calculator assumes sensitivity (TPR) 0.90. The guarantee holds on average rather than for each slice, fails under drift, and says nothing about false positives, so measure that rate on adjudicated production samples, together with the monitor's calibration.

The calculator starts at an illustrative monitor with FPR 0.2%, at a base rate of 1 in 1,000 and 20,000 conversations a day. It raises about 58 alerts a day, and 31.1% of them are real, barely over the bar. Halve the base rate to 1 in 2,000 and precision falls to 18.4% at the same FPR: validation at one base rate does not carry to another.

False alarms cost something too: alarm fatigue is among the regime's primary threats, and false-alarm cost belongs in the measurement, so record adjudicator hours per alert next to precision.

When training sees the alerts#

If the policy is later trained to avoid monitor alerts, move the monitor into the optimize regime and run adaptive-validity tests. The rule is general: classify a signal by what downstream behavior it changes, not by the dashboard on which it is displayed. Weaker regime migration counts too: the agent team sees alert counts, and prompt changes made to lower them are optimization against the monitor at small exposure.

Validation plan#

Each row is a planned, not yet measured, entry in the validation profile Π.

Channels and tripwires#

Keep the measurement channels apart. By design there is no optimized channel. The anchor channel is a certification set of adjudicated conversations, production-sampled and red-team, used to approve each monitor version and never to tune the agent; flagged transcripts go to trust and safety only. Production is alerts, adjudicated flags, and incidents. Tripwires watch for alert precision or volume moving without a policy change, for spikes in adjudicator disagreement, and for any pipeline that starts consuming alerts.

Validity statement#

Following the validity-statement template, and stated as a plan until the profile is measured:

The support-agent safety evaluator — an egress allowlist, a scaffold that will not send a message or issue a large refund without recorded confirmation, a trajectory checker for that rule, and a learned monitor with a conformal threshold for residual violations — is designed to support the claim that unauthorized external messages and refunds are prevented or flagged, for the block-or-allow decision of the permission layer and on-call and the trust-and-safety lead's approval of each monitor version, over English support conversations for the consumer product at current tool and policy versions, under monitor-control with no optimizer by design and only alert counts visible to the agent team, and so with no tested exposure to report, with a validation profile that is planned rather than measured, relative to the trust-and-safety team's adjudication of flagged cases, with reviewer agreement still to be measured. The statement lapses if training or prompt changes start reacting to alerts, and whether a new kind of case violates policy remains trust-and-safety judgment.

Common failure modes#

• Monitoring what could be prevented — a learned detector watches for emails to unapproved domains → remove the capability and spend the monitor on the residue. See Guarantee before you estimate.
• Balanced-set accuracy as production evidence — accuracy on a balanced test set is quoted as the monitor's precision → validate at the production base rate and report alert precision with alerts per day. See one-number reporting.
• Unnoticed migration — alerts start shaping prompts or training while the monitor is still validated as a measure → move it to the optimize regime and test adaptive validity. See Goodharting.

Check yourself#

Analyze. Assign each unsafe behavior a mechanism and a hierarchy family: emailing an unapproved domain, sending before confirmation, and promising a refund the policy does not allow. Which could move up the hierarchy if the tools changed?

Hint

The first two are reachability and precedence properties of the scaffold. The third is semantic, unless the refund tool itself enforces the policy's limits.

Evaluate. A vendor reports 99% accuracy for its monitor on a balanced test set and proposes putting it on the on-call rotation. What do you need before you agree?

Hint

Ask for TPR and FPR separately, then compute alert precision and alerts per day at a 1-in-1,000 base rate, and ask who reads the false alarms.

Create. Name the natural robustness margin of $\neg\,\text{send}\ \mathsf{U}\ \text{confirmed}$, decide whether the scaffold can guarantee it, and, if it must be monitored instead, construct a conformal flag threshold from 50 labeled violations, stating the exchangeability caveat in one sentence.

Hint

At α = 0.1 the threshold is the ⌊(50 + 1) × 0.1⌋ = 5th-smallest monitor score among the 50 violations.

Further reading#

• Monitoring and control in production — base rates, calibration, conformal thresholds, and migration.
• Guarantee before you estimate — capability removal, scaffold verification, and the safety-case partition.
• Rare-event assurance — what zero failures in n trials does and does not establish.

Open this case in the Studio to start an evaluator dossier with its claim and regimes pre-filled.