# Frame review and governance

> Better judges can't tell you whether you're measuring the right thing. Frame review is the governance process that can.

**Frame review** is the governance process that asks whether the construct and objective an evaluator system serves are still the right ones, a question logically prior to verifying performance against them. It is a process, not a measurement channel, and no improvement to a judge can stand in for it.

> [!KEY] In one sentence
> A better judge measures the same thing more accurately; only a governance process with named authority, pre-committed triggers, and a place outside the loop can decide that the thing itself should change.

## The logically prior question

Most of this handbook asks some version of "does the signal mean what we claim?" Frame review asks whether the claim is still the one worth making. The framework counts governance and frame review among the disciplines a mature practice needs, to ask the logically prior question: are we still evaluating the right thing?

It sits above the measurement channels, not among them. Frame review is deliberately not called a channel: it is a meta-level process that can revise what is being measured. The optimized, anchor, and production channels can disagree with one another, and their disagreements are evidence; deciding what a disagreement means for the [construct](https://benchmaker.ai/glossary/construct) is frame review's job ([Channels, independence, and anchor burn](https://benchmaker.ai/handbook/lifecycle/channels-and-independence)).

The research agent of Case D shows the difference. Much of research execution can be progressively instrumented: citation provenance checked deterministically, artifacts reproduced by execution, claim–evidence entailment judged and audited by experts. Direction-setting is different, because it includes the logically prior question of what deserves evaluation at all, a frame-selection problem that the verification hierarchy alone does not solve (see [Case D](https://benchmaker.ai/handbook/cases/research-agent)). The RSI survey goes further, reading its corpus as showing that the difficulty of choosing research directions and the difficulty of verifying open-ended work are a single bottleneck ([Chen, Wang & Qu, 2026](https://benchmaker.ai/handbook/reference/sources#chen-wang-qu-2026), §5.4). At Marr's computational level the same point reads: a more sophisticated grader cannot repair an incorrectly posed computational-level question.

## Frame lock-in

**Frame lock-in** is the failure frame review exists to catch: a system goes on optimizing a fixed criterion after the real objective, or the landscape of opportunities, has moved. The prescription is direct: lock-in requires external frame review, not merely a better judge.

A better judge cannot help because every improvement inside the instrument raises fidelity to the property already specified. Formality improves fidelity to the specified property but does not validate the choice of property, and the same holds for a better judge, a sound formal checker, or a cleaner rubric. Under lock-in the measurement can be excellent. The scores are accurate, reliable, and rising, and they are about the wrong thing.

Picture DocsAgent, the documentation agent of [Case G](https://benchmaker.ai/handbook/cases/docs-agent), after its product team decides that what matters now is resolving support issues end to end, not answering questions about the documentation. The suite still rewards grounded answers, the development and certification scores still rise together, and every number is accurate *(illustrative)*. Nothing in the instrument is broken; the frame is. The framework's tripwire for this pattern is repeated success on lines of work that no longer matter, and the only response is a review of the frame.

Automated vigilance does not reach this level. The A-Evolve-Training loop noticed that its own development metric had stopped tracking an external target, but the objective and the search space it optimized within were set by people (Chen, Wang & Qu, 2026, §6.2); catching a corrupted proxy is a channel question, and choosing the objective is a frame question.

## When to convene a review

A frame review that waits for someone to feel uneasy will not happen. The evaluator dossier asks for written triggers for construct and objective review, and for the name of whoever has authority to revise the construct. Three triggers come from the tripwires and the decision log, and two more follow from the claim and regime layers:

| Trigger | Where it shows up | Basis |
|---|---|---|
| Metrics improve but real outcomes do not | The mismatch tripwire: O and A up, P flat or falling | Divergence tripwire |
| Repeated success on lines of work that no longer matter | The lock-in tripwire | Divergence tripwire |
| The eval has not changed a decision recently | The decision log | The eval-theater diagnostic |
| The decision or consumer the evaluator serves changes | The claim's decision | Inference from the claim layer |
| The loop can now modify its own evaluator or criteria | The modification surface and loop closure | Inference from the regime layer |

The first two triggers are [divergence tripwires](https://benchmaker.ai/glossary/divergence-tripwire) over the [O/A/P channels](https://benchmaker.ai/glossary/oap-channels). The fourth follows from the claim layer: the same measurement can be adequate for one decision and inadequate for another. The fifth follows from the RSI framing: once a loop can rewrite its own machinery of "better," the question shifts from whether the system improves against a target to what makes the changing target trustworthy. Add a date on the calendar as a backstop, so that review does not depend on a wire firing.

When DocsAgent's mismatch wire fires, with offline scores rising and fresh user-resolution outcomes flat, the review's question is whether "procedurally correct, grounded answers to help requests within the supported product corpus" is still the construct that users need.

## Who may revise the frame

A trigger without an owner is an alarm nobody answers. Frame governance names who has authority to revise the construct and objective, and that authority is narrower than the right to contribute evidence: in a suite, cases are open to everyone while one named owner arbitrates the [specification](https://benchmaker.ai/glossary/specification), the rule of "cases wide, specs narrow" ([Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites)).

At the end of the chain sits the [trust root](https://benchmaker.ai/glossary/trust-root), where the evaluate-the-evaluator recursion stops; for normative, open-ended quality it may be an expert panel with measured reliability and an explicit governance mandate. Measured reliability makes the panel a mechanism with known error, and the mandate is what lets it change the frame ([The validation profile and validity statement](https://benchmaker.ai/handbook/pressure/validation-profile)).

The coding-agent case writes this into its [validity statement](https://benchmaker.ai/glossary/validity-statement): two senior engineers adjudicate disputed cases and own the specification, and revisions to the specification remain human judgment ([Case B](https://benchmaker.ai/handbook/cases/coding-agent)). By the same logic as the anchor condition, the body that reviews the frame should not be the team that optimizes against the current score: the independence argument extends from the channels to the frame.

A review has four possible outcomes: keep the frame, revise the construct, reconnect the eval to a decision, or retire the eval. A revised construct is a new claim. It brings a new measurement version, revalidation of the evidence the change reopens, and a new validity statement ([Harness architecture and reproducibility](https://benchmaker.ai/handbook/lifecycle/harness-and-reproducibility)).

## Loop closure and the human role

Who validates a change is the second axis of the RSI survey's taxonomy (Chen, Wang & Qu, 2026, §2.2). In a human-in-the-loop design a person approves every change; in a [human-on-the-loop](https://benchmaker.ai/glossary/human-on-the-loop) design an automatic signal drives the loop while people audit outcomes and gate deployment; in a closed loop no person reviews the improvements the system makes and applies. The most consequential cell is self-evaluation in a closed loop, where the system can alter its own definition of better and bounded self-refinement shades into [open-ended RSI](https://benchmaker.ai/glossary/open-ended-rsi) (see [Self-improvement and evaluator co-evolution](https://benchmaker.ai/handbook/agents/self-improvement)).

For frame review, loop closure settles a practical question: is there still a place outside the loop to stand? The survey's safety discussion reports an argument that controls living inside an agent's own runtime are within reach of whatever influences the agent, so a system that can reach far enough into its runtime could in principle rewrite its own guardrails (Chen, Wang & Qu, 2026, §7.3). Whatever governs the frame, whether the review body, the trust root, or the record of triggers, has to live outside what the loop can modify.

The human role in that place has to be designed, not left over. A pre-specified study of human-in-the-loop research in economics found that reliability depended less on model capability than on how the cognitive work was divided between people and machine: pre-commitment, the order of decisions, accountability, and the allocation of attention ([Zhu et al., 2026](https://benchmaker.ai/handbook/reference/sources#zhu-2026), via Chen, Wang & Qu, 2026, §6.3). Read as requirements for frame review, in this handbook's synthesis:

- **Pre-commitment:** write the triggers, the evidence each needs, and the decision rule before the results arrive.
- **Sequencing:** set the frame before optimization begins, and review it before results are interpreted, so that a good number cannot redefine the goal after the fact.
- **Accountability:** name who signs off on keeping, revising, or retiring the frame.
- **Attention:** spend scarce expert time on residual judgment and on the frame, not on rechecking what machines already check well.

The survey closes on the same point: until systems become reliable evaluators of open-ended research judgment, the people in the loop remain the field's last line of verification (Chen, Wang & Qu, 2026, §9).

## Eval theater is a governance failure

[Eval theater](https://benchmaker.ai/glossary/eval-theater) is the terminal pathology of an eval program: suites that are run before launches and quoted in reviews but connected to no decision. Its diagnostic is a single question, when the eval last changed a decision; if the answer is never, the eval is set dressing. It is a governance failure rather than a measurement failure, because the instrument itself may be excellent. What is missing is a decision for it to serve ([Decision claims](https://benchmaker.ai/handbook/claim/decision-claims)).

Frame review is where theater gets caught, because it is the one process that looks at the whole portfolio against the decisions it is supposed to inform. A DocsAgent tone score that appears on every dashboard but has never held a release is a candidate for retirement, or for a decision rule that would let it hold one *(illustrative)*.

> [!WARNING]
> Ask of every eval in the portfolio when it last changed a decision. If the answer is never, give it a decision rule or retire it; do not keep it as decoration. See [eval theater](https://benchmaker.ai/handbook/practice/anti-patterns#eval-theater).

The hardest version of the governance question has no instrument yet.

> [!OPEN] Governance-grade measurement
> Proposals for credible slowdowns of self-improving systems need [governance-grade measurement](https://benchmaker.ai/glossary/governance-grade-measurement): evidence that could show, auditably, what a training loop is and is not getting better at, including that it is not improving past a threshold. The RSI survey finds almost no technical work on it and calls it the clearest gap between stated stakes and research effort that it found (Chen, Wang & Qu, 2026, §7.3, §8). Frame review can ask the question; no current instrument answers it.

## In practice

Write a frame charter for one evaluator system.

1. **State the frame.** Copy the construct, objective, decision, and consumer from the claim, and list the interpretations it excludes.
2. **Name the authority.** Name who may revise the construct and the specification, the mandate they hold, and the trust root for disputed cases.
3. **Pre-commit the triggers.** Write down the triggers and the evidence each needs before results arrive, including the mismatch and lock-in tripwires, a decision-log check, changes to the decision or to loop closure, and a calendar date.
4. **Assemble the review packet.** Gather the tripwire history, production outcomes, the decision log, the exposure ledger, and what has changed in the world.
5. **Choose one of the four outcomes.** Keep the frame, revise the construct and revalidate, reconnect the eval to a decision, or retire it.
6. **Record what remains human judgment.** Write it into the validity statement.

The artifact is a frame charter, which fills the frame-governance entry of the evaluator dossier, the triggers for construct and objective review and who has authority to revise it, and supplies the validity statement's account of what remains human judgment ([Templates](https://benchmaker.ai/handbook/reference/templates)).

## Common failure modes

- **Frame lock-in** — scores keep rising on a criterion nobody needs anymore → pre-commit lock-in triggers and route them to frame review, not to a better judge. See [frame lock-in](https://benchmaker.ai/handbook/practice/anti-patterns#frame-lock-in).
- **Eval theater** — the eval is run and quoted but changes no decision → ask when it last changed one, then reconnect it or retire it. See [eval theater](https://benchmaker.ai/handbook/practice/anti-patterns#eval-theater).
- **Review by the optimizer** — the team that optimizes against the score also decides whether the score is still right → give frame authority to someone outside the loop. See [Channels, independence, and anchor burn](https://benchmaker.ai/handbook/lifecycle/channels-and-independence).
- **A better judge as the answer** — a frame question is answered with a judge upgrade → ask the computational-level question first. See [Marr's levels for evaluation](https://benchmaker.ai/handbook/foundations/marr-levels).

## Check yourself

**Analyze.** Classify each problem as one of measurement, lifecycle, or frame: (a) a judge's false-positive rate rose after a model swap; (b) the certification split's score jumped after developers saw its failures; (c) scores keep rising on question types that users stopped asking.

> [!DETAILS] Hint
> (a) is natural error, fixed by recalibration; (b) is anchor burn; (c) is frame lock-in, and no judge upgrade fixes it.

**Evaluate.** An autonomous post-training loop notices that its development metric has decoupled from an external benchmark, and adapts. Does that make human frame review unnecessary for the loop?

> [!DETAILS] Hint
> The survey stresses that the loop optimized within an objective and search space chosen by people (Chen, Wang & Qu, 2026, §6.2). Detecting proxy corruption compares O with A; choosing the objective is the frame.

**Create.** Write the frame-governance entry for the research agent of Case D: the construct, the authority, the triggers, and what remains human judgment.

> [!DETAILS] Hint
> Direction-setting is the frame-selection problem that the hierarchy does not solve, and for open-ended quality the trust root is an expert panel with measured reliability and a governance mandate.

## Further reading

- [Channels, independence, and anchor burn](https://benchmaker.ai/handbook/lifecycle/channels-and-independence) — the three measurement channels that frame review sits above.
- [Self-improvement and evaluator co-evolution](https://benchmaker.ai/handbook/agents/self-improvement) — loop closure, open-ended RSI, and the non-verifiable frontier.
- [Decision claims](https://benchmaker.ai/handbook/claim/decision-claims) — writing the decision that an eval must change.
- [Templates](https://benchmaker.ai/handbook/reference/templates) — the dossier's frame-governance field and the validity statement.
- Chen, Wang & Qu (2026), *Recursive Self-Improvement in AI* — §2.2 for loop closure, §6.3 and §7.3 for the human role and controls outside the runtime, and §8 for governance-grade measurement.

---
Source: https://benchmaker.ai/handbook/lifecycle/frame-review