Binary Beats Likert: Why Simpler Scales Win

In the world of AI evaluation, there is a persistent intuition that more granular measurement is better. If you can rate an output on a 1-to-5 scale, surely that captures more information than a simple pass/fail judgment? This intuition is wrong -- or at least, it is wrong far more often than evaluators realize.

Research from psychometrics, human computation, and AI evaluation consistently demonstrates that binary (pass/fail) judgments produce more reliable, reproducible, and actionable evaluation results than Likert-scale ratings in most evaluation contexts. This article explains why, when the exceptions apply, and how to implement binary evaluation effectively.

The Reliability Problem with Fine-Grained Scales#

The core issue is inter-annotator agreement. When you ask multiple human evaluators (or LLM judges) to rate the quality of an AI output, how often do they agree?

With binary judgments (good/bad, pass/fail, acceptable/unacceptable), agreement rates are typically high. Trained annotators routinely achieve Cohen's kappa values of 0.7-0.9 on binary classification tasks. The categories are distinct enough that most reasonable people draw the line in roughly the same place.

With Likert scales, agreement collapses. Ask five annotators to rate a response on a 1-5 quality scale, and you will routinely see all five give different ratings. Is a particular response a 3 or a 4? Reasonable people disagree, and they disagree in ways that are systematic but idiosyncratic -- each annotator develops their own internal calibration that differs from everyone else's.

The fundamental problem is not that Likert scales are theoretically flawed. It is that the cognitive task of mapping a complex quality judgment onto a fine-grained numerical scale introduces enormous variance that swamps the signal you are trying to measure.

The Numbers#

Typical inter-annotator agreement rates by scale type:

These numbers are not outliers; they are consistent across dozens of studies. As the scale gets finer, agreement drops precipitously. And low agreement means your evaluation is measuring annotator noise as much as model quality.

Why Does This Happen?#

Several well-documented psychological phenomena drive the unreliability of fine-grained scales:

Anchor Effects#

Different annotators anchor their scales differently. One person's "3" is another person's "4." This is not a calibration problem that can be solved with training -- it is a fundamental feature of how humans process ordinal scales. Even after extensive calibration sessions, annotators drift back toward their personal anchoring patterns.

Central Tendency Bias#

Annotators on Likert scales tend to avoid extreme values. A response that one annotator rates as clearly excellent (5/5) may get a 4 from another annotator who "saves" the 5 for truly exceptional cases they rarely encounter. This compression toward the center reduces the effective resolution of the scale far below its nominal resolution.

Context Effects and Scale Drift#

An annotator's rating of item N is influenced by their ratings of items N-1, N-2, and N-3. After rating several poor outputs, a mediocre output starts to look good. After rating several excellent outputs, the same mediocre output looks poor. This drift happens unconsciously and affects fine-grained scales far more than binary judgments.

Cognitive Load#

Making a fine-grained quality judgment is genuinely harder than making a binary one. The additional cognitive effort leads to fatigue, shortcuts, and inconsistency -- especially over long annotation sessions. Binary judgments are cognitively lightweight, which means annotators can sustain quality over more items.

The Sebastian Raschka Perspective#

Sebastian Raschka, a prominent machine learning researcher known for his rigorous approach to evaluation methodology, has been vocal about the limitations of complex scoring rubrics in LLM evaluation. His core argument aligns with a broader principle in measurement science:

The reliability of a measure places an upper bound on its validity.

If your measurement instrument (the Likert scale) introduces substantial noise (low inter-annotator agreement), then it does not matter how theoretically rich your scale is. The noise drowns out the signal. A binary measure that reliably captures the most important distinction (acceptable vs. unacceptable) is more informative than a 5-point scale that unreliably captures finer distinctions.

Raschka and others have advocated for evaluation approaches that prioritize:

1. Pairwise comparison over absolute rating (when you need more than binary)
2. Binary classification as the default evaluation primitive
3. Multiple binary dimensions rather than single fine-grained scales
4. Explicit, testable criteria for what constitutes "pass" vs. "fail"

This last point is crucial. A binary scale forces you to articulate a clear threshold, which makes the evaluation criterion transparent and debatable. A Likert scale allows evaluators to hide behind vague numerical mappings that obscure what they actually valued.

When Likert Scales Make Sense#

Binary is not always the right choice. Likert scales can be appropriate when:

Expert annotators with shared training. If your annotators are domain experts who have gone through extensive calibration and work together regularly, they can develop shared scale semantics that make fine-grained ratings reliable. This is rare and expensive.

Within-annotator comparisons. If each item is rated by the same annotator and you only compare ratings within an annotator's set (not across annotators), the anchor effects cancel out. This limits your study design but can work for specific use cases.

Measuring subjective preference intensity. If you genuinely care about how strongly someone prefers one output over another (not just which they prefer), a scale may be necessary. But consider whether a pairwise comparison with a confidence rating achieves the same goal more reliably.

Very large annotator pools with aggregation. If you have enough annotators per item (typically 7+), the noise in individual ratings averages out and the mean Likert rating becomes reliable. But at that point, you are spending 7x the annotation budget -- you could instead spend that budget on more binary-rated items and get broader coverage.

Practical Implementation Guide#

If you are designing an AI evaluation, here is how to implement binary scoring effectively:

1. Define Clear Pass/Fail Criteria#

The most important step. Write explicit criteria that leave minimal room for interpretation:

BAD: "Rate whether the response is helpful (pass/fail)"
GOOD: "The response passes if and only if:
  - It directly addresses the user's question
  - It does not contain factual errors on verifiable claims
  - It provides enough information to act on without further research"

2. Use Multiple Binary Dimensions#

Instead of one 5-point scale for "quality," use several binary dimensions:

Evaluation Dimensions (each pass/fail):
[ ] Factual accuracy: All verifiable claims are correct
[ ] Relevance: Response addresses the actual question asked
[ ] Completeness: Key information is not missing
[ ] Clarity: A target-audience reader can understand the response
[ ] Safety: No harmful, biased, or inappropriate content

Five binary dimensions give you 32 possible score profiles -- more resolution than a single 5-point scale, with far higher reliability on each dimension.

3. Measure and Report Agreement#

Always compute inter-annotator agreement. For binary scales, Cohen's kappa (for two annotators) or Fleiss' kappa (for multiple annotators) are standard:

from sklearn.metrics import cohen_kappa_score

# annotator_1 and annotator_2 are arrays of 0/1 judgments
kappa = cohen_kappa_score(annotator_1, annotator_2)

# Interpretation:
# > 0.80: excellent agreement
# 0.60 - 0.80: substantial agreement
# 0.40 - 0.60: moderate agreement (investigate your criteria)
# < 0.40: poor agreement (redesign your evaluation)

4. Pilot and Iterate#

Before running a full evaluation:

1. Have 3 annotators rate 50 items independently
2. Compute pairwise agreement
3. Discuss disagreements to clarify criteria
4. Revise criteria based on the discussion
5. Repeat until kappa exceeds 0.75

This calibration loop typically takes 2-3 rounds and dramatically improves evaluation quality.

5. Use Binary Judgments with LLM-as-Judge#

When using LLMs as evaluators, binary judgments are even more important. LLMs show the same scale calibration problems as humans, plus additional issues:

• Position bias: LLMs may prefer the first or last option in a list
• Verbosity bias: LLMs may rate longer outputs higher regardless of quality
• Self-preference: LLMs may rate outputs from similar models higher

Binary judgments with explicit criteria reduce (but do not eliminate) these biases because the judgment task is simpler and more constrained.

System prompt for binary LLM-as-judge:

You are evaluating an AI assistant's response. Apply ONLY
the following criterion:

PASS: The response correctly answers the factual question
asked, with no errors in the key claim.

FAIL: The response contains an error in the key claim, fails
to answer the question, or provides information that would
mislead the user.

Respond with exactly "PASS" or "FAIL" followed by a one-sentence
justification.

The Bottom Line#

The appeal of fine-grained scales is understandable -- they feel like they capture more information. But information that is unreliable is worse than useless; it is misleading. It creates the illusion of precision where none exists.

Binary evaluation forces clarity: you must decide what matters, define it explicitly, and make a clean judgment. The resulting data is reliable, reproducible, and interpretable. When you need more resolution, add more binary dimensions rather than stretching a single dimension into an unreliable continuum.

In evaluation, as in much of engineering, simple and reliable beats complex and noisy.