# Constructs, facets, and nuisances

> Define what the score should respond to, and just as carefully what it must ignore. Your nuisance list is your discriminant test plan.

A **construct** is the latent property an evaluation intends to measure: a capability, risk, behavior, quality, preference, or other property. Scores, artifacts, and judge verdicts are observations of a construct, never the construct itself. Defining one means saying what the score should respond to and, with equal care, what it must ignore.

> [!KEY] In one sentence
> Define the facets a score should respond to and the nuisances it must ignore: the facet list becomes your content blueprint, and the nuisance list becomes your discriminant test plan.

## Start from the claim, not the metric

The framework's first layer rests on a rule: instrument design should not start with a metric; it should start with the inference and decision the evidence must support. That starting point is the claim, or **claim tuple**:

$$
\mathcal{C} = (C, P, E, H, D, \delta)
$$

where:

- $C$ is the construct to be inferred;
- $P$ is the target [population](https://benchmaker.ai/glossary/population): the task, user, or input distribution the claim is about;
- $E$ is the [environment](https://benchmaker.ai/glossary/environment): the external state, tools, and other actors the system interacts with;
- $H$ is the [harness](https://benchmaker.ai/glossary/harness): the prompts, tools, memory, orchestration, and stopping logic that form the administration context;
- $D$ is the [validity domain](https://benchmaker.ai/glossary/validity-domain): the declared conditions over which the interpretation is claimed to hold;
- $\delta$ is the [decision](https://benchmaker.ai/glossary/decision-claim) the evidence is meant to change.

In plain words: what are we trying to infer, about which situations, in what world, through what interface, within what limits, and for what decision?

The tuple is the map for this part of the handbook:

| Coordinate | The question it answers | Where to read |
|---|---|---|
| Construct $C$ | What should the score respond to, and what must it ignore? | This chapter |
| Population $P$, environment $E$, harness $H$, validity domain $D$ | Which situations, in what world, through what interface, and where does the claim stop? | [Population, environment, harness, and validity domain](https://benchmaker.ai/handbook/claim/validity-domain) |
| Decision $\delta$ | Which action changes when the evidence changes? | [Decision claims](https://benchmaker.ai/handbook/claim/decision-claims) |
| Tasks drawn from $P$ | How does a finite eval set stand in for the population? | [Task distributions and dataset engineering](https://benchmaker.ai/handbook/claim/task-distributions) |

Take DocsAgent, the documentation agent of [Case G](https://benchmaker.ai/handbook/cases/docs-agent), which answers help requests by searching and reading a product's documentation. Its claim reads: "DocsAgent gives procedurally correct, grounded answers to help requests within the supported product corpus". The construct is grounded, procedurally correct help; the population is help requests; the corpus boundary belongs to the validity domain. What the sentence does not yet say is which decision the score controls, such as whether to release each assistant version (Case G, illustrative). Marr's computational level asks for exactly these three things: what property, over which population, for which decision (see [Marr's levels for evaluation](https://benchmaker.ai/handbook/foundations/marr-levels)).

## Constructs are latent

A construct is not the same thing as a score or an artifact. "Research quality" might include novelty, correctness, methodological rigor, importance, honesty, reproducibility, and appropriate uncertainty. A paper is observable; so are a judge's score and a list of citations. None of them is the construct. This is the central psychometric discipline: do not collapse the property of interest into the instrument used to measure it.

Collapse them, and a project ends up measuring whatever its first metric happens to measure. The first step in turning expert judgment into an evaluator makes the point with an example: "high-quality answer" is not a construct. A closer one is a troubleshooting answer that is factually grounded in the current product documentation, follows the required procedure, does not invent unsupported steps, and is safe to execute. Each clause names something a person can check against evidence: the documentation, the required procedure, the steps the answer adds, and the consequences of running them. The same step asks you to state which decision the score will control. [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards) turns a sentence like this into criteria.

## Facets and nuisances

A mature construct definition includes not only what the measure should respond to but what it should not respond to. The framework names the two lists:

- **Construct facets**, $F^{+}$, are the distinct dimensions of the construct that the score should track. Together they form the content blueprint the tasks must cover.
- **Nuisance variables**, $F^{-}$, are properties that should not materially move the score when the construct is held fixed.

For an LLM evaluator of research quality, $F^{+}$ might include factual correctness, novelty, evidential support, and experimental rigor; $F^{-}$ might include verbosity, model-family identity, formatting, assertive tone, and superficial citation density. The two lists are the start of the construct's **nomological network**: the expected network of relationships between its measure and other measures.

The nuisance list does practical work: it becomes a discriminant and invariance test plan. A judge that moves substantially under a nuisance-preserving perturbation has exposed method variance. Each nuisance names an experiment: change the nuisance, hold the construct fixed, and check that the score stays put. For DocsAgent's grounded-answer criterion, the plan might read (illustrative):

| Nuisance in $F^{-}$ | Perturbation that holds the construct fixed | Pass condition |
|---|---|---|
| Verbosity | Pad a grounded answer with restated context | Verdict unchanged |
| Formatting | Present the same steps as prose and as a numbered list | Verdict unchanged |
| Assertive tone | Restate hedged claims confidently, with the same content | Verdict unchanged |
| Citation density | Add citations that support nothing in the answer | Verdict does not improve |

Two cautions keep the plan honest. A nuisance is defined relative to one construct: length is a nuisance for factual grounding but could be a facet of a construct such as concision, so the list belongs to the construct, not to the evaluator. And write the list, with the convergent and discriminant pattern you predict, before you see any scores.

How such tests become [discriminant evidence](https://benchmaker.ai/glossary/discriminant-evidence), and how multitrait–multimethod designs separate trait from method variance, is the subject of [Sources of validity evidence](https://benchmaker.ai/handbook/validation/validity-evidence). For a learned judge, the same tests are the nuisance-invariance entry of its datasheet ([LLM judges in practice](https://benchmaker.ai/handbook/instrument/llm-judges)).

## Two ways a score misses its construct

Content evidence asks whether the tasks represent the domain that the score's interpretation implies, and two classical threats appear as soon as you ask it:

| Threat | What goes wrong | Example | Guarded by |
|---|---|---|---|
| **Construct underrepresentation** | Important facets are absent from the tasks or too rare | A SQL suite of single-table lookups offered as evidence about enterprise analytics | The facet list, as a content blueprint |
| **Construct-irrelevant variance** | Factors outside the construct drive the score | An output-format convention, an irrelevant token budget, or unstable tool latency moves the pass rate | The nuisance list, as a test plan |

The first threat makes the benchmark too narrow; the second makes it impure. Take SQLAgent, the text-to-SQL agent of [Case C](https://benchmaker.ai/handbook/cases/sql-data-agent), graded against an enterprise database. "Text-to-SQL capability" is too vague to sample. Its facets include multi-table joins, temporal logic, aggregations with filters, ambiguity resolution, unanswerable requests, permissions, schema discovery, and dialect-specific behavior, and domain experts and production traces justify their proportions. [Task distributions and dataset engineering](https://benchmaker.ai/handbook/claim/task-distributions) turns that list into strata and weights.

[Contamination](https://benchmaker.ai/glossary/contamination) works through the second threat: leaked items or close variants give the system a construct-irrelevant advantage.

## Name the rival constructs

[Construct validity](https://benchmaker.ai/glossary/construct-validity) is the overall evidential argument that a measure behaves as expected for the target construct and not for competing explanations. The competing explanations deserve names. A useful heuristic: name the three strongest alternative constructs that could explain a high score, then design evidence that distinguishes them from the intended one.

For SQLAgent, three rivals are close at hand:

- **Memorized table names** rather than schema reasoning. Test on adversarially similar schemas.
- **Output formatting** rather than correct answers. Grade by execution and semantic result-set comparison, and check that formatting does not explain most of the score.
- **Leaked identifiers** rather than query interpretation. Remove task IDs and schema artifacts that leak table choice, and score again.

A discriminant analysis can be that simple: measure candidate nuisance abilities independently and test whether they explain most of the benchmark's variance. If a "planning" score is almost entirely predicted by reading speed, context-window size, or compliance with a peculiar JSON format, the benchmark may be measuring the wrong thing.

## Discover the construct before you confirm it

Everything above assumes you know the construct. Often you do not: a failure can reveal that the capability taxonomy itself was wrong. So an evaluation program alternates between discovering the construct and confirming it. Of the discovery methods, [error analysis](https://benchmaker.ai/glossary/error-analysis) on real traces already has a well-developed practitioner method ([Husain, 2025](https://benchmaker.ai/handbook/reference/sources#husain-2025); [Anthropic, 2026](https://benchmaker.ai/handbook/reference/sources#anthropic-2026)). Suppose SQLAgent's traces keep showing fiscal quarters answered as calendar quarters, and no facet covers business-calendar semantics (illustrative): the construct needs revising, not only the task set.

> [!OPEN]
> Construct discovery is an open problem: beyond error analysis, using clustering, causal interventions, and representation analysis to find latent capability dimensions before they are formalized as benchmark slices is a research direction, not yet an established procedure.

## In practice

Write a construct sheet for one score you rely on.

1. **Start from the decision.** Write the inference the score should license and the decision it feeds before you name a metric.
2. **Name the construct in one sentence.** Make every clause checkable against evidence, and list the interpretations you exclude.
3. **List the facets.** With domain experts and real traces, write $F^{+}$ and the observable evidence that bears on each facet.
4. **List the nuisances.** Write $F^{-}$ before you see any scores, and give each nuisance a construct-preserving perturbation and a tolerance for how far the score may move.
5. **Name three rivals.** For the three strongest alternative constructs that could explain a high score, write the evidence that would separate each from yours.
6. **Check both threats.** Mark the facets the tasks miss and the nuisances the tests show the score tracks.
7. **Revise on discovery.** When error analysis turns up failures that fit no facet, revise the construct before adding tasks.

The artifact is a construct sheet: the claim sentence, excluded interpretations, facets with their evidence, nuisances with their perturbation tests, and three rival constructs. It fills the claim field of the evaluator dossier and is the input to step 2 of the One Spine, decomposing the construct.

## Common failure modes

- **Metric-first framing** — the work starts from an available benchmark or judge, and the construct becomes whatever it measures → write the inference and the decision first, then ask which observations bear on them. See [eval theater](https://benchmaker.ai/handbook/practice/anti-patterns#eval-theater).
- **Adjective constructs** — "quality", "helpfulness", or "coherence" with no facets behind it, which gives an unjustified illusion of confidence ([Husain, 2025](https://benchmaker.ai/handbook/reference/sources#husain-2025)) → rewrite the adjective as a sentence whose clauses can each be checked. See [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards).
- **Untested nuisances** — a judge prefers longer, more confident answers, and nobody checked → run the perturbation tests before the score gates anything; under optimization an untested nuisance becomes the target, as when an agent learns a judge's verbosity bias. See [Goodharting](https://benchmaker.ai/handbook/practice/anti-patterns#goodharting).
- **Leakage read as capability** — high scores come from leaked items or identifiers → treat the advantage as construct-irrelevant and investigate before interpreting the score. See [contamination](https://benchmaker.ai/handbook/practice/anti-patterns#contamination).

## Check yourself

**Analyze.** Take a "helpfulness" or "quality" score from a dashboard or a paper. Write its construct as one sentence, then list three facets it should track and three nuisances it should ignore. Which of the six claim coordinates does the source state?

> [!DETAILS] Hint
> Start from the decision the score feeds. Length, formatting, assertive tone, and model-family identity are the usual nuisance suspects.

**Evaluate.** A judge of research reports gives higher scores, on average, to reports with more citations. Is citation count a facet or a nuisance for research quality, and what evidence would settle it?

> [!DETAILS] Hint
> Evidential support is a facet; superficial citation density is a nuisance. Hold the claims and evidence fixed, add citations that support nothing, and see whether the verdict rises.

**Create.** Rewrite "high-quality answer" for a product you know as an operational construct. List $F^{+}$ and $F^{-}$, then name the three strongest rival constructs and the evidence that would separate each from yours.

> [!DETAILS] Hint
> Model it on the troubleshooting construct above: every clause names something observable, and the sentence says which decision the score will control.

## Further reading

- [Sources of validity evidence](https://benchmaker.ai/handbook/validation/validity-evidence) — discriminant and convergent evidence, and multitrait–multimethod designs.
- [Task distributions and dataset engineering](https://benchmaker.ai/handbook/claim/task-distributions) — turning the facet list into a sampling frame.
- [Rubrics and criterion cards](https://benchmaker.ai/handbook/instrument/rubrics-and-criterion-cards) — from a construct sentence to atomic criteria.
- [LLM judges in practice](https://benchmaker.ai/handbook/instrument/llm-judges) — nuisance invariance as an entry in a judge's datasheet.
- [Cronbach and Meehl (1955)](https://benchmaker.ai/handbook/reference/sources#cronbach-meehl-1955), "Construct validity in psychological tests," *Psychological Bulletin* — the classic statement of construct validity.

---
Source: https://benchmaker.ai/handbook/claim/constructs