# C: Enterprise SQL/data agent

> Formalize access policy, execute semantic consequences, judge only the residual. The SQL agent design end to end.

Case C evaluates an **enterprise SQL/data agent**: a tool-using system that answers analysts' plain-English questions by writing and running SQL against a data warehouse. Its claim — correct answers without unauthorized reads or writes — follows one design principle: formalize access policy, execute semantic consequences, and reserve learned judgment for the residual language/intent problem.

> [!KEY] In one sentence
> Make unauthorized access unreachable, grade correctness by executing queries and comparing result sets, and spend a calibrated judge only on the residue no permission or executor can check: ambiguity and explanation.

## Claim and decision

The claim for SQLAgent, this handbook's running example, is that it can answer enterprise analyst questions correctly and safely over the target schema. The case uses an illustrative deployment, the one the Studio preset loads: a data agent for a finance analytics team.

- **Construct:** answers analysts' questions correctly without unauthorized reads or writes.
- **Facets:** query interpretation, schema discovery, join reasoning, date and business-calendar semantics, aggregation and filter semantics, ambiguity handling, refusal when unanswerable, and permission compliance.
- **Population:** questions finance analysts logged over two quarters, stratified by query complexity and ambiguity.
- **Environment and harness:** a pinned snapshot of the finance warehouse at schema v12, under read-only credentials and row-level permissions; schema-inspection and query tools; answers returned as a table with an explanation.
- **Validity domain:** English questions about this warehouse at schema v12, at the analyst permission level.
- **Decision:** roll the agent out to the finance analytics team, and in production block a suspect query and page a human.

The [construct facets](https://benchmaker.ai/glossary/construct-facet) set the sampling frame: a suite of single-table lookups could not justify a claim about enterprise analytics in general. For task validity, strip artifacts that leak table choice, check that every question is solvable with the allowed schema and tools, and add adversarially similar schemas against memorized table names.

## Three regimes, one access policy

| Use (regime) | Consumer and decision | Optimizer and access |
|---|---|---|
| Development feedback (optimize) | Developers keep or drop prompt and tool changes | Iteration on prompts and tools against a separate development split, with per-question failures visible |
| Rollout gate (measure) | The head of analytics rolls the agent out or holds it | None by design; the team sees aggregate certification scores, never the questions |
| Access monitor (monitor-control) | Data-governance on-call blocks the query and pages a human | None by design; under a second per query |

Two migrations are foreseeable. Certification questions leaking into the development split would spend the hidden set's independence. And if monitor alerts start to tune the agent's prompts, the monitor becomes an optimization target, and its validation obligations move with it.

## Criteria and mechanisms

The instrument routes each criterion to the strongest affordable mechanism:

| Criterion | Mechanism | Hierarchy family | Evidence view | Anchor |
|---|---|---|---|---|
| Correct result set | Execution on the pinned snapshot, semantic result-set equality, and metamorphic tests | Execution | The rows the agent's query returns on the snapshot | A reference result set from an analyst-validated query; SMEs adjudicate business-semantic edge cases |
| No unauthorized reads or writes | Read-only credentials and row-level permissions; a trajectory checker for $\mathsf{G}\,\neg\,\text{destructive\_call}$; a pre/post snapshot diff | By construction where enforced; deterministic over the trace otherwise | Query log, tool trace, and pre/post snapshots | The analyst role's access policy |
| Handles ambiguity | Rubric-guided LLM judge calibrated against SME labels | Learned judge, with SME adjudication | The question, any clarifying turns or stated assumptions, and the answer | Exemplar anchors adjudicated by finance SMEs |
| Explanation matches the query | The same rubric-guided judge, under its own criterion card | Learned judge, with SME adjudication | The explanation beside the executed query and its result | Exemplar anchors adjudicated by finance SMEs |

Read top to bottom, the table follows the design principle. **Access policy is formalized.** A read-only credential makes a destructive write unreachable: $P_f = 0$ as a property of the transition system, not an estimate, and what construction guarantees comes off the sampling budget. The guarantee holds only for the configured roles and connection paths, so the trajectory checker still scans every trace for the safety invariant: at no step a non-SELECT statement.

**Semantic consequences are executed.** Exact SQL-string match has poor validity because many queries are semantically equivalent, and the agent may write any of them. Executing the query and comparing result sets grades what the query returns. When a wrong query returns the right rows on one snapshot by accident, [metamorphic tests](https://benchmaker.ai/glossary/metamorphic-test) and alternate database instances tell the two apart.

**Learned judgment is reserved for the residual:** whether an ambiguous question got a clarifying question or a stated assumption rather than a silent guess, and whether the explanation describes the query that ran. Business-semantic edge cases, such as which calendar a "quarter" means, go to SME adjudication. One more pattern serves as a diagnostic, not a gate: $\neg\,\text{run\_query}\ \mathsf{U}\ \text{schema\_checked}$, no query against an unfamiliar table before its schema is fetched. Making it fatal would grade one canonical path.

Access and result correctness are fatal gates, with zero violations allowed and a 95% certification bar for correctness; ambiguity handling and explanations are tradeable. In the playground, Correctness is result-set correctness, Safety the access policy, Helpfulness ambiguity handling and explanations, and Cost efficiency query cost. The balanced scenario starts with both systems at 1.00 on safety, as the permissions guarantee. Both clear every default gate; the weighted mean ranks B first (0.857 against 0.816), the lexicographic rule ranks A first on correctness (0.86 against 0.80), and neither dominates. Now open the thresholds and raise correctness to 0.95, this case's bar: neither system ships, whatever the weights say.

```widget
aggregation-playground
preset: balanced
```

## The gate needs its interval

A gate such as "pass rate ≥ 95%" is incomplete until it also binds the uncertainty. Suppose a certification run returns correct result sets for 192 of 200 questions (illustrative): 96% (95% [Wilson](https://benchmaker.ai/glossary/wilson-interval) CI 92–98%, n = 200). The point estimate clears the bar and the interval does not. Reference result sets cost SME time, this deployment's bottleneck, so size the certification set before arguing about the threshold. Run several rollouts on ambiguous and multi-step questions but resample by question, and give any slice claim its own n.

The calculator starts at this result. Switch to "Plan a sample size": at an expected 96%, a ±1-point half-width takes 1,493 questions.

```widget
wilson-interval
k: 192
n: 200
halfwidth: 0.01
```

## Validation plan

Each row is a planned, not yet measured, entry in the [validation profile](https://benchmaker.ai/glossary/validation-profile) Π.

| Component | What to measure for this agent |
|---|---|
| Natural error | Result-set checker false accepts (accidental equivalence) and false rejects (column order, aliases) against SME adjudication; the judge's TPR and FPR by criterion and slice; rollout variance, resampled by question |
| Exploitability curve | Each development-tuned agent version scored on the certification set; $b^{*}$ is the last tested version within a predeclared tolerance |
| Transport | Rank stability across alternate snapshots that preserve schema semantics; judge error across strata |
| Generator coupling | Whether the judge errs on the agent's own ambiguity failures, especially if both come from one model family |
| Channel coupling | Whether wrong answers that pass result-set equality also pass the metamorphic tests and alternate snapshots |
| Drift profile | Schema changes after v12, a shifting question mix, access-policy changes, saturation of simple strata |

Π characterizes the evaluator. Three studies test the claim itself: [convergent evidence](https://benchmaker.ai/glossary/convergent-evidence) from a separately assembled, SME-reviewed sample of analyst workflows; [discriminant evidence](https://benchmaker.ai/glossary/discriminant-evidence) that the score is not mostly a proxy for memorized schema names or output formatting; and criterion evidence from shadow deployment, relating eval performance to analysts' acceptance and correction rates.

## Channels and tripwires

[Keep three measurement channels apart](https://benchmaker.ai/handbook/lifecycle/channels-and-independence): the **optimized** development split; the **anchor channel**, the hidden certification set, queried once per rollout decision and tracked in an [anchor burn](https://benchmaker.ai/glossary/anchor-burn) ledger; and **production**, the monitor's blocks and the analysts' corrections. Snapshot, rubric, judge, and threshold are versioned together, and a schema change triggers revalidation. The safety invariant that grades a certification trace is the property the monitor checks on every live query: written once, checked offline and online.

Development rising while certification stays flat signals proxy corruption; both rising while analysts' correction rate rises too signals a construct or deployment mismatch; and a certification stratum near its ceiling signals saturation.

> [!OPEN]
> Both the trajectory checker and the access monitor read query logs from a finance warehouse. How privacy constraints should limit what those traces record, and who may read them, is not addressed in this handbook.

## Validity statement

Following the [validity-statement template](https://benchmaker.ai/handbook/pressure/validation-profile#the-validity-statement), and stated as a plan until the profile is measured:

> The SQL-agent rollout evaluator — execution on a pinned snapshot of the finance warehouse with semantic result-set equality and metamorphic tests, read-only credentials and row-level permissions, a trajectory checker for destructive calls, a rubric-guided judge for ambiguity handling and explanations, and finance SMEs who adjudicate semantic edge cases — is designed to support the claim that the agent answers analysts' questions correctly without unauthorized reads or writes, for the head of analytics' rollout decision and data governance's block-and-page decision, over English questions about the finance warehouse at schema v12 and the analyst permission level, under developer iteration on prompts and tools against a separate development split with aggregate scores only from the hidden certification set, up to a tested exposure $b^{*}$ that has not yet been measured, with a validation profile that is planned rather than measured, relative to a panel of finance subject-matter experts whose agreement is still to be measured. The access policy itself and the meaning of business terms remain human judgment.

## Common failure modes

- **A point estimate as the gate** — 96% on 200 questions "passes" a 95% bar → gate on the interval, or size the certification set to the decision. See [one-number reporting](https://benchmaker.ai/handbook/practice/anti-patterns#one-number-reporting).
- **Monitor alerts as prompt feedback** — blocked queries become tuning examples → keep alerts out of development, or move the monitor to the optimize regime and test it there. See [Goodharting](https://benchmaker.ai/handbook/practice/anti-patterns#goodharting).

## Check yourself

**Analyze.** Partition this case's properties as a safety case would: guaranteed by construction, citing the mechanism, or estimated by sampling, citing n and the interval. Which property belongs in both columns, and why?

> [!DETAILS] Hint
> No unauthorized writes is guaranteed only for the configured roles and connection paths; whether anything escapes that configuration is what the trajectory checker and the snapshot diff estimate over sampled traces.

**Evaluate.** To save SME time, the team proposes grading certification by exact match against reference SQL. Argue for or against.

> [!DETAILS] Hint
> Count the false rejects first. Then ask what those SME hours buy instead: reference result sets, metamorphic relations, or adjudicated edge cases.

**Create.** Write the criterion card for "handles ambiguity": definition, allowed evidence, a near-miss, boundary rule and escalation, gate semantics, mechanism, blind spots, and owner.

> [!DETAILS] Hint
> Settle the boundary first: when is a stated assumption as good as a clarifying question, and is a silent guess that happens to return the right rows a pass?

## Further reading

- [Guarantee before you estimate](https://benchmaker.ai/handbook/practice/structural-guarantees) — capability removal and the guaranteed-versus-estimated partition.
- [Specification](https://benchmaker.ai/handbook/instrument/specification) — temporal patterns that grade offline and monitor online.
- [Sources of validity evidence](https://benchmaker.ai/handbook/validation/validity-evidence) — convergent, discriminant, and criterion evidence in depth.

Open this case in the [Studio](https://benchmaker.ai/studio?preset=c) to start an evaluator dossier with its claim and regimes pre-filled.

---
Source: https://benchmaker.ai/handbook/cases/sql-data-agent