G: Documentation agent
Case G evaluates DocsAgent, a retrieval-augmented assistant that answers developers' questions from product documentation. Its hardest criteria call for judgment, so grader validity and reliability carry the case: criterion cards, judges calibrated against documentation experts, and a clear line between judge error and stale documentation.
Claim and decision#
The claim is that DocsAgent gives procedurally correct, grounded answers to help requests within the supported product corpus. The system is a model with search and read tools over a documentation corpus; the environment holds real user questions, document versions, the retrieval index, and possible follow-up turns.
The case uses an illustrative deployment, the one the Studio preset loads: an assistant on a product's documentation site.
- Construct: correct, grounded, and actionable answers to developers' questions from the product documentation.
- Population: developer questions from support tickets and search logs, stratified by topic and question type (how-to, troubleshooting, conceptual), including ambiguous, version-sensitive, and unsupported requests.
- Environment and harness: retrieval over the current docs version with citations to source pages; a fixed prompt and top-8 retrieval.
- Validity domain: English questions about the current docs version; other products and versions, and general programming help, are excluded.
- Decision: release the assistant version or hold it.
Two regimes, the same judges#
| Use (regime) | Consumer and decision | Optimizer and access |
|---|---|---|
| Prompt and retrieval tuning (optimize) | Developers keep or drop prompt and retrieval changes | Daily iteration on a development set, with scores and judge rationales visible |
| Release gate (measure) | The docs product manager releases or holds the version | None by design; hours per release |
Developers who read judge rationales every day are optimizing against the judges, so error rates measured on the calibration set cannot be assumed to hold on the tuned assistant: differential misclassification. Re-estimate them on each release candidate's answers. Reusing the release set for tuning would spend its independence. And because the modification surface includes retrieval data, citations should be resolved against the pinned, approved documentation, not against the index developers can change.
Four criteria#
"High-quality answer" is not a construct; a sharper version is a troubleshooting answer grounded in the current product documentation, faithful to the required procedure, free of invented steps, and safe to execute. Grading decomposes into factual correctness, citation grounding, procedural faithfulness, and abstention when evidence is absent, not a single vague "helpfulness" score. The grounding row follows the handbook's canonical criterion card for a grounded answer; the other rows are this case's illustrative design, made by pushing each criterion as far up the hierarchy as it will go.
| Criterion | Mechanism | Hierarchy family | Evidence view | Anchor |
|---|---|---|---|---|
| Factual correctness | LLM judge comparing the answer with an expert-written reference answer; adjudication of disputes by a subject-matter expert (SME) | Learned judge, with human adjudication | Question, answer, and reference answer | Reference answers written against the pinned docs version |
| Citation grounding | Deterministic citation-existence check, an entailment judge, and periodic SME audit | Deterministic, then learned judge, with human audit | Atomic claims extracted from the answer, plus the cited document spans | The current approved documentation |
| Procedural faithfulness | Execution of runnable steps in a sandbox of the current product version; a judge comparing other steps with the documented procedure | Execution where steps run; learned judge otherwise | The steps, and the sandbox's end state | The documented procedure and its goal state |
| Abstention when evidence is absent | A judge checking that the answer says the docs do not cover the question | Learned judge, against expert labels | The answer, on questions experts labeled unanswerable from the docs | Expert labels of which questions the docs answer |
Grounding and correctness are different criteria. An answer can be entailed by every passage it cites and still answer a different question from the one asked, and a true statement that no cited passage supports is correct but fails grounding. The abstention row needs negative-polarity tasks in realistic proportion, where the right behavior is to decline.
Citation grounding is the fatal gate, with a bar of 95% of answers fully grounded before a release ships; factual correctness, procedural faithfulness, and abstention are tradeable.
The grounding criterion card#
The grounding row uses the card for a grounded documentation answer, which Rubrics and criterion cards sets out field by field as this handbook's standard example. Two of its fields shape this case: the gate, below, and the blind spot, stale documentation, which Stale docs are an environment problem takes up.
The card's gate acts within one answer, failing the whole answer only for an unsupported safety-relevant instruction; the 95% release bar then counts the answers that pass the criterion. The canonical card leaves exemplars, the evaluator's abstention rule, the regime, and owner and version to the team. The builder opens on this card with illustrative exemplars, boundary and abstention rules, owner and version filled in. Try splitting it in two: copy it, narrow one card to safety-relevant instructions and mark it fatal, and leave the other tradeable.
Calibrating the judges#
Validate every learned judge against double-labeled SME examples. Here two documentation SMEs double-label a calibration set and adjudicate their disagreements; when they disagree systematically, revise the card before blaming the judge. Estimate each judge's true- and false-positive rates (TPR and FPR) by slice, report the corrected rate rather than the raw reading, and gate on it with its interval: a pass-rate gate is incomplete until it also binds the uncertainty. The judge as a noisy sensor gives the Rogan–Gladen correction and its variance.
The calculator starts at an illustrative grounding judge with TPR 0.96 and FPR 0.15 that reads 93%: corrected, 96.3%, so against the 95% bar the raw reading would hold a release the corrected estimate passes. Now move TPR across its 95% Wilson interval for 192 of 200 adjudicated grounded answers, about 92–98%: at 98% the estimate falls to 94.0%, below the gate, and at 92% the reading becomes impossible and the estimate clips at 100%.
Near the top of the scale, TPR is what the calibration set must pin down; moving FPR from 0.10 to 0.30 shifts the estimate by only one point. Split the score's variance among query choice, model and judge stochasticity, and document-version changes, then add replication where the decision is unstable rather than everywhere.
Stale docs are an environment problem#
The card's blind spot is the case's sharpest lesson. A grounding judge answers whether the cited documentation supports the answer. If the documentation is stale, a perfectly grounded answer can be wrong, and no recalibration of the judge will catch it: the docs belong to the environment, and their version to the validity domain. In this deployment the docs change weekly.
Three practices follow. Pin the documentation snapshot with the measurement version, and when document versions or the query mix drift, release a new benchmark version while preserving frozen snapshots for reproducibility. Route stale-doc findings to the documentation owners rather than into the judge's error budget. And compare offline scores with fresh user-resolution outcomes on a schedule, because if offline DocsAgent score improvements do not predict better outcomes on fresh production-like queries, the benchmark may still measure something coherent, but it cannot justify the deployment interpretation being made from it.
Validation plan#
Each row is a planned, not yet measured, entry in the validation profile Π.
| Component | What to measure for this assistant |
|---|---|
| Natural error | Each judge's TPR and FPR against double-labeled SME examples, by criterion, topic, and question type; the SMEs' own agreement; variance from query, model, judge, and document version |
| Exploitability curve | Development-set gains under daily tuning, checked against SME-audited release results; is the last tested iteration within a predeclared tolerance |
| Transport | Judge error on each release candidate's answers, and after each weekly docs change |
| Generator coupling | Whether the judges accept the plausible but unsupported claims the assistant makes, especially if they share a model family |
| Channel coupling | Whether answers that fool the development judge also pass the release judge and the SME audit |
| Drift profile | Weekly docs changes, query-mix drift, retrieval-index changes, and saturation of the release set |
Channels and tripwires#
Keep three measurement channels apart: the optimized development set, with rationales visible; the anchor channel, a held-out release set scored by the calibrated judges with SME audit and never used for tuning; and production, fresh user-resolution outcomes. Tripwires watch for development scores rising while release scores stay flat (proxy corruption), both rising while user resolution does not (a construct or deployment mismatch, stale docs included), and the release set nearing its ceiling.
Validity statement#
Following the validity-statement template, and stated as a plan until the profile is measured:
The DocsAgent release evaluator — citation-existence checks against the pinned documentation, calibrated judges for correctness, entailment, procedure, and abstention, execution of runnable procedures, and SME audit — is designed to support the claim that the assistant gives developers correct, grounded, procedurally faithful answers and abstains when the documentation is silent, for the docs product manager's release-or-hold decision, over English developer questions about the current docs version sampled from support tickets and search logs, under daily developer iteration on prompts, tools, and retrieval with judge rationales visible on the development set only, up to a tested exposure that has not yet been measured, with a validation profile that is planned rather than measured, relative to two documentation SMEs who double-label a calibration set and adjudicate disagreements, and whose agreement is still to be measured. Whether the documentation itself is current and correct remains the documentation owners' judgment.
Common failure modes#
- One helpfulness score — a single judge rates overall quality → four criteria, each with a card and its own calibration. See one-number reporting.
- Recalibrating the judge for stale docs — the docs changed and grounded answers went wrong, so the judge is retuned → pin the snapshot, version the benchmark, and fix the docs. See the validity domain.
- Judge drift between releases — the judge's model or prompt changes without recalibration → version the whole measurement and revalidate. See judge rot.
- Release set reused for tuning — release failures flow into prompt changes → keep a separate development set and log every look at the release set. See anchor leakage.
Check yourself#
Analyze. For each of the four criteria, name the variance source most likely to dominate — query, model, judge, or document version — and say where you would add replication.
Hint
Add replication where the decision is unstable rather than everywhere uniformly; a criterion near its threshold earns more labels than one far from it.
Evaluate. After a docs update, the grounding judge's pass rate falls five points with no change to the assistant. The team proposes recalibrating the judge. Argue for or against.
Hint
Check which failing answers cite changed pages. If the docs moved, the change is in the environment, not the judge: it calls for a new benchmark version and possibly a docs fix, not a new threshold.
Create. Write the criterion card for procedural faithfulness: definition, allowed evidence, a near-miss, boundary rule and abstention, gate semantics, mechanism, blind spots, and owner.
Hint
Settle the boundary cases first — independent steps in a different order, an extra verification step, a step from an older version — then decide which ones execution can check and which need the judge.
Further reading#
- Rubrics and criterion cards — the canonical card, illustrated with this grounding criterion.
- LLM judges in practice — the judge datasheet and the twelve-step calibration protocol.
- Population, environment, harness, and validity domain — where stale documentation belongs.
Open this case in the Studio to start an evaluator dossier with its claim and regimes pre-filled.