Marr's levels for evaluation
Marr's levels separate three questions every evaluation answers: at the computational level, what claim and what decision; at the algorithmic level, what representation and procedure; at the implementation level, what machinery. Disagreements about evaluators often turn out to sit at different levels.
Three levels, three questions#
This handbook uses David Marr's framework as an anti-confusion device:
| Level | Question | What lives here | Typical mistake |
|---|---|---|---|
| Computational | What claim and decision are we trying to support? | The latent property, the population of situations, and the decision the evidence should change | Optimizing a proxy benchmark for the wrong construct |
| Algorithmic | What representation of quality and evidence channel operationalize the claim? | Tasks, environments, specifications, rubrics, invariants, graders, reward models, estimators, slices, adversarial search | Collapsing several criteria into one vague score |
| Implementation | What exact harness, model, code, and state execute the procedure? | Judge model version, prompt, sampling settings, parser, test runner, sandbox, database snapshot, permissions, retry policy, logging | Flaky tests, judge-version drift, leakage, non-reproducible state |
The test of a computational-level statement is whether it names a property, a world, and a decision. "Model A scored 82" names none; "The agent can autonomously modify production code without violating approval and security constraints, often enough to justify merge privileges" names all three. The framework puts the same question at the claim layer: what property are we trying to infer, over what world, and for what decision?
SQLAgent, the text-to-SQL agent of Case C, shows all three in one evaluator. Its claim is that it answers enterprise analysts' questions correctly and safely over the target schema. Algorithmically, that becomes real analyst questions plus a balanced regression set, execution against a pinned snapshot with semantic result-set comparison, checks for forbidden writes, and intervals that treat tasks as clusters. Its implementation is the snapshot version, the executor and its timeouts, the comparator's normalization rules, and the trace log (illustrative).
Excellent at one level, useless at another#
The levels are not a ranking: each can fail on its own. An eval can be excellent at one level and useless at another: a reproducible harness can faithfully reproduce an invalid metric. A sophisticated judge can grade the wrong construct, and a statistically precise mean can estimate the wrong deployment distribution.
Fixes do not travel upward: a more sophisticated grader cannot repair an incorrectly posed computational-level question. Correctness does not travel downward either: a well-posed claim with a sound procedure can still fail through a parser bug or an environment leak. A typical case is a "model regression" that was really a timeout bug in the agent's scaffold: an implementation fault read as a computational conclusion.
Triage a disagreement by level#
Hence the rule: the first discipline of eval design is to identify which level a disagreement belongs to before arguing about tools. Three common disagreements, located:
| Disagreement | Level to check first | First question |
|---|---|---|
| "The model regressed after Tuesday's run." | Implementation | Did the harness, judge version, prompt, parser, or snapshot change? Diff the per-trial records before blaming the model. |
| "Our judge and our experts disagree." | Algorithmic, then computational | Is the rubric underspecified, or do the experts hold legitimately different values about what matters? When experts disagree systematically, revise the specification before blaming the judge. |
| "The leaderboard leader is not the model our users prefer." | Computational | Does the benchmark's claim (its construct, population, and decision) match yours? |
The second row shows why the order matters. Rater disagreement has at least four causes: random noise, an underspecified rubric, legitimately plural values or trade-offs, and differences in expertise. An underspecified rubric is an algorithmic problem with an algorithmic fix; plural values are computational, and no judge, however strong, settles which property the decision should rest on.
Reconciling the old catalog#
An earlier version of this site's benchmark catalog used Marr's levels as a classification tree: a computational problem (a capability family such as reasoning or tool use), an algorithmic mechanism (reference-based, human, LLM-as-a-judge, or task-based evaluation), and an implementation artifact (the benchmark itself). MMLU was filed as knowledge and recall, then reference-based evaluation, then MMLU. The handbook keeps the levels and changes what goes in each:
| Level | Old catalog | This handbook | What changes |
|---|---|---|---|
| Computational | A capability family | A claim: property, population, and decision | A capability name is a construct label without a world or a decision |
| Algorithmic | A grading-method family | The whole representation: tasks, specification, reference anchor, evidence view, mechanism, estimator | "Reference-based" names a reference anchor, not a mechanism; the comparator is classified on its own |
| Implementation | The benchmark's name | The harness, versions, prompts, parser, and state that ran it | The same benchmark run under two harnesses is two implementations |
The old tree also treated Raschka's four approaches as alternatives at the algorithmic level, but they name different axes — response format, mechanism, aggregation protocol — so one benchmark can belong to two of them (Vocabulary without category errors). The catalog's evaluator cards now describe benchmarks in coordinates instead: the MMLU card states a claim, separates the answer key from the exact-match comparator, and counts the prompt format, number of shots, and answer-extraction rule as part of the instrument because they change scores. A tree that files benchmarks helps browsing; levels that locate a question help design and argument.
In practice#
Build a three-row Marr design sheet for one evaluation.
- Computational row. Write the claim as one sentence that names a property, a population, and a decision. If it reads like "Model A scored 82", rewrite it.
- Algorithmic row. List the tasks and environments, the specification of each criterion, the mechanism and evidence each criterion uses, and the estimator.
- Implementation row. List what executes the procedure: model and judge versions, prompts, settings, parser, test runner, sandbox, snapshot, and logging.
- Add each row's typical mistake, with your evidence that you avoided it.
- Say which row a change touches. A new judge prompt is an implementation change that needs recalibration; a new criterion is algorithmic; a new decision is computational and can invalidate both rows below it.
The artifact is the design sheet, three rows long. Its rows feed the claim, the specification and mechanisms, and the reproducibility fields of the evaluator dossier (see Templates).
Common failure modes#
- Tools argued at the wrong level — a debate over which judge to buy before anyone has agreed on the claim → settle the computational row first; if no decision depends on the score, stop. See eval theater.
- Implementation drift read as capability change — a score moves after a harness or judge update and is reported as a model regression or gain → diff versions and traces before drawing a computational conclusion. See judge rot.
- Precision mistaken for validity — a reproducible harness and a tight interval around a metric that answers the wrong question → check the rows above the one you perfected. See Decision claims.
Check yourself#
Analyze. Apply Marr's levels to "evaluate whether an AI scientist chooses good research directions".
Hint
The computational row is the hard one: what makes a direction good, and for whose decision? The RSI survey treats research direction-setting as the paradigm case of a task with no checkable signal (Chen, Wang & Qu, 2026, §5.4), so expect the algorithmic row to end in human judgment.
Evaluate. A stronger LLM judge replaces a deterministic checker. Which Marr-level question should be asked before accepting the change?
Hint
Start at the computational level: what property does the decision need checked? Then the algorithmic one: can an executor or a rule check it? Do not use a learned judge where an executor can answer the question more directly.
Create. Build a three-row Marr design sheet for an agent benchmark of your choice, with the typical mistake and your evidence against it in each row.
Hint
For the implementation row, start from what each trial should persist: model and decoding settings, prompts and tool schemas, task and dataset versions, environment snapshot, transcript, grader version and raw output, errors, retries, and seeds.
Further reading#
- Decision claims — writing the computational level as a decision with a threshold and error costs.
- Constructs, facets, and nuisances — the property at the heart of the computational level.
- Harness architecture and reproducibility — the implementation level in depth.