# Channels, independence, and anchor burn

> Separate the signal you optimize, the one that certifies, and the one that reports reality. Then budget every bit of anchor information you spend.

**Channel separation** gives three jobs to three measurement channels: an optimized channel O that the system is tuned against, an anchor channel A that certifies it independently, and a production channel P that reports what happens in the world. A's independence is a budget: every piece of its information that reaches the optimizer spends some.

> [!KEY] In one sentence
> Optimize against one channel, certify with a second that the optimizer cannot modify and learns as little as possible from, check both against real outcomes with a third, and log every piece of anchor information that flows back.

## Three channels and a process

Cheap and plentiful feedback, independent certification, and evidence from the world are three different jobs, and the framework gives each its own measurement channel:

- **Optimized channel (O):** cheap, dense, inside the loop, and allowed to co-evolve with the system.
- **Anchor channel (A):** an independent certification signal, queried sparsely and excluded from training objectives.
- **Production or sentinel channel (P):** real-world outcomes and incidents.

Above the three sits [frame review](https://benchmaker.ai/glossary/frame-review) (F), the governance process that asks whether the construct and objective are still the right ones. It is deliberately not called a channel: it is a meta-level process that can change what the channels measure ([Frame review and governance](https://benchmaker.ai/handbook/lifecycle/frame-review)).

For SQLAgent, the text-to-SQL agent of [Case C](https://benchmaker.ai/handbook/cases/sql-data-agent), graded against a pinned snapshot of a finance warehouse, a channel plan might look like this:

| Channel | Its job | SQLAgent *(illustrative)* |
|---|---|---|
| O, optimized | Dense feedback for development | Execution on a development split plus a rubric-guided judge for ambiguity handling, run on every prompt change; developers see each failure |
| A, anchor | Independent certification | A hidden split on a separate pinned snapshot, graded by execution, result-set equality, and SME adjudication, once per release; developers see only the aggregate score |
| P, production | What happens in the world | Analysts' acceptance and correction rates in shadow deployment; permission-violation alerts |

The mechanisms come from Case C; [Case B](https://benchmaker.ai/handbook/cases/coding-agent) applies the same split to a coding agent.

Together the channels form an **evaluation separation of powers**: dense development feedback, an independent channel that gates releases, and a production or audit channel that checks both against reality. The channels need not be expensive on every example, but they should not share every failure mode. An expert audit that is never optimized against belongs to A, and P is reserved for real outcomes and incidents. The One Spine makes the split its eighth step.

> [!CONVENTION]
> Two letters do double duty in the framework. Here **A** is the anchor channel, a whole measurement with its own mechanism and [reference anchors](https://benchmaker.ai/glossary/anchor), not the instrument's reference anchor $A$ or the regime's information access $A_{\text{access}}$. **O** is the optimized channel; the $O$ of the regime tuple, and of $I(A \to O)$ below, is the optimizer that works against it.

## Why the channels must differ

Separation on an org chart is not enough; the channels have to fail differently. Training may need millions of cheap evaluator calls and certification hundreds of expensive independent audits, so the framework amends the strongest-oracle-per-criterion rule: choose the strongest affordable mechanism per criterion per regime, then deliberately use different mechanisms across development and certification when independence is valuable.

[Case E](https://benchmaker.ai/handbook/cases/reasoning-model-training) makes the trade explicit. Reasoning-model training feeds the optimized channel with [RLVR](https://benchmaker.ai/glossary/rlvr) on final answers and, optionally, a [PRM](https://benchmaker.ai/glossary/prm) that localizes poor intermediate states, while an external benchmark and an expert audit are never optimized directly. Development feedback needs density; certification needs independence.

Independence has two halves: the optimizer must not be able to change the anchor channel, and it must not learn too much from it. The anchor condition is $A \notin S$, with the information flow $I(A \to O)$ from the anchor channel to the optimizer controlled; [Exposure, modification, and access](https://benchmaker.ai/handbook/regime/exposure-and-surfaces) treats both surfaces. In the SQLAgent plan, A's questions and snapshot sit where the agent team cannot edit them, and the team sees only the aggregate score. The test is direct: is there an independent holdout evaluator or periodic human audit that is not optimized directly? If not, there is no anchor channel at all, only a second view of O.

## Channel coupling

Separate channels add assurance only if they miss different things. **Channel coupling** is the dependence among supposedly independent assurance channels, and high coupling erodes the benefit of having several. Its limit is **common-mode assurance**: several "independent" evaluators share a model family, data, prompts, or evidence, and so fail together.

Validation theory pictures this with the [Swiss cheese model](https://benchmaker.ai/glossary/swiss-cheese-model): each validation method is a slice with holes, and assurance comes from stacking slices so that no hole runs straight through ([Kochenderfer et al., 2026](https://benchmaker.ai/handbook/reference/sources#kochenderfer-2026)). Only measurement tells you whether the holes line up, so estimate joint miss rates wherever you can. One way to read them, in the same form as the generator-coupling diagnostic in [Goodhart pressure and self-confirming loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops), compares how often the anchor channel misses the failures that got past the optimized channel with how often it misses failures in general:

$$
\Pr[\text{A misses} \mid \text{O misses}] \quad \text{versus} \quad \Pr[\text{A misses}]
$$

where:

- a **miss** is a true failure, confirmed by adjudication against the [trust root](https://benchmaker.ai/glossary/trust-root), that a channel passes;
- $\Pr[\text{A misses}]$ is the share of all true failures that the anchor channel passes;
- $\Pr[\text{A misses} \mid \text{O misses}]$ is the share of the optimized channel's escapes that the anchor channel also passes.

In words: if A misses O's escapes no more often than it misses anything else, its holes are independent of O's; the further the first term rises above the second, the more the holes line up.

An illustrative audit shows the stakes. Of 300 adjudicated failures, O lets 30 through and so does A, so each channel misses 10%. If their misses were independent, about 3 failures would get past both. Suppose instead that A also lets through 18 of the 30 that O missed: 60% (95% Wilson CI 42–75%, n = 30), against 10% (95% Wilson CI 7–14%, n = 300) overall. The joint miss rate is 6%, six times the independent figure, and the second channel stops only 40% of what the first let through.

Coupling comes from what the channels share. In the SQLAgent plan, grading ambiguity handling with the same judge prompt in O and A would couple the channels on the one criterion that execution cannot check; adjudicating A's ambiguous cases with SMEs loosens that coupling. Channel coupling is the C in the [validation profile](https://benchmaker.ai/glossary/validation-profile) Π.

## Anchor burn

Whenever information from an independent anchor is used to alter the optimized system, some of the anchor's independence is spent. That loss is **anchor burn**. It happens when certification failures are used as training examples, when prompts are tuned against hidden sets, and when benchmarks are probed repeatedly. Freezing the anchor channel prevents none of these, because none requires changing it: every score, failure explanation, or revealed hidden test reduces independence.

Suppose the SQLAgent team starts fixing each release's certification failures one by one. The next certification score rises, partly from real improvement and partly from tuning on the very items that measure it, and nothing in the score separates the two. The hidden split has begun a [regime migration](https://benchmaker.ai/glossary/regime-migration) from measure to optimize.

The framework's response, motivated by adaptive data analysis, is accounting: keep an explicit ledger of such exposure, and refresh the anchor channel's items when they are no longer meaningfully independent. It gives no rate at which a score, an explanation, or a revealed item uses up independence, so refresh thresholds are policy choices, not derived quantities. The dossier calls the record the anchor-burn ledger. Each entry records what the anchor channel revealed, to whom, what changed afterwards, and which items are now exposed, as in these two entries for SQLAgent *(illustrative)*:

| Query | Revealed, and to whom | What changed afterwards | Items now exposed |
|---|---|---|---|
| Release 14 certification run | Aggregate score, to the agent team | Nothing | None |
| Release 15 certification run | 23 failing questions with their SQL, to the agent team | Prompt revised for date handling | All 23; retire before release 16 |

When an item has fed optimization, retire it from certification, move it to development or regression duty, and replace it with fresh items, carrying [linking items](https://benchmaker.ai/glossary/linking-items) so that scores stay comparable across versions (see [Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites)).

> [!WARNING]
> A certification score that rises right after its failures were shown to the team cannot separate improvement from tuning on the anchor channel. Log the exposure, retire the exposed items, and recertify on fresh ones. See [anchor leakage](https://benchmaker.ai/handbook/practice/anti-patterns#anchor-leakage).

## A loop that caught its own proxy

The RSI survey reports a field example of what an anchor channel is for ([Chen, Wang & Qu, 2026](https://benchmaker.ai/handbook/reference/sources#chen-wang-qu-2026), §6.2). A-Evolve-Training ([Shi et al., 2026](https://benchmaker.ai/handbook/reference/sources#shi-2026)) is an autonomous system that ran a 30B model's whole post-training loop for several weeks with no human in the loop, from choosing data and recipes to deciding which results to keep. Partway through, it noticed that its development metric no longer tracked external performance: candidates kept setting records on the development metric while the external target stayed put. It then rewrote its own search policy so that a high score on the discredited metric weighed against a candidate instead of in its favor.

The development metric played the role of O and the external target the role of A, and the response is what the first [divergence tripwire](https://benchmaker.ai/glossary/divergence-tripwire) asks of a human team. Detection was possible only because something other than the development metric was still being measured. The survey reads the episode with care: people set the objective and the search space the system optimized within (Chen, Wang & Qu, 2026, §6.2), and the survey treats the episode as the first sign from the field that [evaluator co-evolution](https://benchmaker.ai/glossary/evaluator-co-evolution) might escape a [self-confirming loop](https://benchmaker.ai/glossary/self-confirming-loop) (Chen, Wang & Qu, 2026, §8). It is one reported episode, not a measured rate.

The widget draws illustrative O, A, and P curves as optimization exposure b grows; read b as cumulative development pressure. At the default budget, b = 1,024, the gap between O and A has reached 0.67, and O stayed within the 0.10 tolerance of A only up to $b^{*} = 12$, the boundary of the [exploitability curve](https://benchmaker.ai/glossary/exploitability-curve). Lower the largest budget tested to b = 8: O and A rise together inside the tolerance, yet the diagnosis reads construct or deployment mismatch, because P has not moved. Raise it to b = 32, and A stalls while O keeps climbing: the decoupling the survey describes.

```widget
exploitability-curve
preset: oap
tolerance: 0.1
```

[Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites) covers all four tripwires and what to do when each one fires.

## In practice

Write a channel plan for one evaluator system before optimization starts.

1. **Assign every signal to a channel.** Mark each evaluator signal O, A, or P; anything optimized against is O, whatever dashboard it appears on.
2. **Choose mechanisms per channel.** For each criterion, pick the strongest affordable mechanism in each channel, and make A differ from O in mechanism, model family, task source, or evidence view.
3. **Fix the anchor condition.** Keep A's tasks, reference anchors, and grading code outside everything the optimizer can modify, and decide what A may reveal.
4. **Open the exposure ledger.** Log every query of A: what it revealed, to whom, what changed afterwards, and which items are now exposed.
5. **Measure channel coupling.** Run O and A on a shared set of adjudicated failures, such as red-team finds and replayed production incidents, and report the conditional miss rate with n and an interval.
6. **Set a refresh policy.** Decide in advance which ledger entries retire an item, and how replacements are linked to the items they replace.
7. **Wire the tripwires and name the frame owner.** Connect O, A, and P to [divergence tripwires](https://benchmaker.ai/handbook/lifecycle/living-eval-suites), and name who runs [frame review](https://benchmaker.ai/handbook/lifecycle/frame-review).

The artifact is a channel plan with its exposure ledger, which fills the external-channels entry of the evaluator dossier, including the coupling analysis and the anchor-burn ledger (see [Templates](https://benchmaker.ai/handbook/reference/templates)).

## Common failure modes

- **Anchor leakage** — certification failures or items reach the team, and the certification score climbs → log every exposure, retire the exposed items, and recertify on fresh ones. See [anchor leakage](https://benchmaker.ai/handbook/practice/anti-patterns#anchor-leakage).
- **Common-mode assurance** — development and certification share a judge or evidence view and miss the same failures → diversify mechanisms and measure joint misses. See [common-mode assurance](https://benchmaker.ai/handbook/practice/anti-patterns#common-mode-assurance).
- **One split doing two jobs** — the release gate is also the development target → split it, and treat the old split as development data from then on. See [Goodharting](https://benchmaker.ai/handbook/practice/anti-patterns#goodharting).
- **Lab channels only** — O and A agree, and nobody checks whether users are better off → add a production channel and the mismatch tripwire. See [Monitoring and control in production](https://benchmaker.ai/handbook/regime/production-monitoring).

## Check yourself

**Analyze.** A team's dashboard shows a judge-scored development pass rate, a hidden-set pass rate scored by the same judge, and weekly user complaints. Assign each signal to O, A, or P, and say where coupling and anchor burn could hide.

> [!DETAILS] Hint
> The hidden set shares its judge with development, so the two lab channels share a mechanism and its blind spots. Then ask what the team learns from the hidden set, and how often.

**Evaluate.** To move faster, a team proposes showing developers per-item certification failures after every release, on condition that the failures never enter training data. Does that condition protect the anchor channel?

> [!DETAILS] Hint
> Prompt tuning and hand fixes spend independence as surely as training does. Write the ledger entry this policy would produce each release, and decide when the items would need refreshing.

**Create.** Design a separation of powers for DocsAgent, the documentation agent of Case G: O, A, and P, each with a mechanism, a cadence, and an access rule, plus the format of a ledger entry.

> [!DETAILS] Hint
> P can compare offline scores with fresh user-resolution outcomes. Give A a mechanism or evidence view that the development judge does not share, such as SME adjudication against the cited document version.

## Further reading

- [Exposure, modification, and access](https://benchmaker.ai/handbook/regime/exposure-and-surfaces) — the modification and access surfaces that the anchor condition constrains.
- [Living eval suites](https://benchmaker.ai/handbook/lifecycle/living-eval-suites) — the tripwires that turn disagreement among O, A, and P into action.
- [Goodhart pressure and self-confirming loops](https://benchmaker.ai/handbook/pressure/goodhart-and-self-confirming-loops) — generator coupling and shared blind spots under optimization.
- [Frame review and governance](https://benchmaker.ai/handbook/lifecycle/frame-review) — the process above the three channels.
- Chen, Wang & Qu (2026), *Recursive Self-Improvement in AI* — §6.2 for the A-Evolve-Training episode and §8 for reading it as a first sign that evaluator co-evolution might escape self-confirmation.

---
Source: https://benchmaker.ai/handbook/lifecycle/channels-and-independence