Reasoning · Evaluator card
BBH
2022Suzgun et al.Paper (opens in a new tab)
BIG-Bench Hard is a curated subset of 23 tasks from the BIG-Bench suite where language models initially performed below average human raters. It tests diverse reasoning skills including logical deduction, causal reasoning, and algorithmic thinking.
Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.
Headline claim
BBH accuracy supports per-task comparisons of few-shot chain-of-thought performance on 23 short, formulaic reasoning tasks for mid-range models; its macro-average no longer separates frontier models.
At a glance
- Mechanism
- Deterministic checkerPrior: execution feedback, rung 2 of 4
- Item format
- Multiple choice
- Size
- 6,511 examples across 23 tasks
- Typical uses
- Measure
- Saturation
- Near saturation
- Contamination
- Medium risk
- Year
- 2022
1Claim
What property are we trying to infer, over what world, and for what decision?
- Construct
- Breadth of short-form reasoning: logical deduction, temporal and causal reasoning, algorithmic manipulation
- Population
- 23 BIG-Bench tasks on which 2022-era models scored below average human raters, each with a fixed answer format, evaluated with few-shot chain-of-thought prompts
- Measures
- Performance across a diverse set of reasoning tasks that were originally challenging for LLMs
- Logical deduction, temporal reasoning, causal judgment, and algorithmic manipulation abilities
- Effectiveness of chain-of-thought prompting on tasks that benefit from step-by-step reasoning
- Breadth of reasoning capability across qualitatively different task types
- Does not measure
- Deep reasoning — many tasks have become easy for modern models and no longer discriminate
- Whether a model reasons or merely pattern-matches, as the fixed task formats are susceptible to memorization
- Real-world reasoning under ambiguity, incomplete information, or conflicting evidence
- Sustained reasoning over long problems — most BBH tasks are short and self-contained
2Instrument
How is the judgment actually produced?
- Mechanism
Deterministic checker
Exact match of the extracted answer against the target.
Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.
- Item format
- Multiple choice
- Specification
- An example is correct when the final answer extracted after the chain of thought exactly matches the target: an option label or a constrained free-form string such as a number or a sorted word list.
- Anchor
- Fixed target answer per example from the BIG-Bench task files
- Evidence view
- The extracted final answer; the reasoning chain is not graded
- Signal
- Binary correct/incorrect per example
- Resolution
- Final answer per example
- Aggregation
- Accuracy per task, macro-averaged across the 23 tasks
- Metrics
- accuracy
- Size
- 6,511 examples across 23 tasks
- Methodology
- Selects 23 tasks from the larger BIG-Bench suite based on a criterion that models performed below human baselines at the time of curation. Each task has a specific format (multiple-choice or free-form with constrained answers). Evaluation uses exact-match accuracy per task, typically reported as a macro-average across all 23 tasks. Chain-of-thought prompting is the standard evaluation protocol.
3Regime
What causal role does the signal play, and what happens when a system adapts to it?
- Typical uses
- Measure · supports inference and decision
- Exposure
- Public since 2022 with fixed few-shot chain-of-thought exemplars. The BIG-bench task files carry a canary string meant to keep them out of training corpora, but fixed formats and small per-task samples keep contamination risk material. It was part of the Hugging Face Open LLM Leaderboard (v2).
4ValidationDraft
Why should we believe that the signal means what we claim it means — before and after the system adapts to it?
BBH's evidence supported its original purpose — identifying reasoning tasks that were hard for 2022-era models — and its spread of qualitatively different task types remains useful. Model progress has eroded discrimination on many of the 23 tasks, so the aggregate is increasingly driven by a shrinking set of still-hard tasks; per-task results support more specific claims than the headline number.
Evidence notes1 partially supported · 2 not supported
- Partially supported
The macro-average summarizes reasoning breadth
Averaging 23 very different tasks hides per-task variation, and the aggregate is now dominated by the few tasks that are not near ceiling.
- Not supported
Separates frontier models
Most tasks are near ceiling for frontier models.
- Not supported
Scores are stable across prompting strategies
Chain-of-thought and answer-only prompting give very different scores; that gap was the original paper's central result, so the prompting protocol is part of the measurement.
Source: Suzgun et al., 2022
5Lifecycle
Does the validity survive over time, and are we still evaluating the right thing?
- Saturation
- Near saturation
Whether top systems still differ, or crowd the ceiling.
- Contamination
- Medium risk
How likely test items, or close variants, have leaked into training data.
- Successors
- None listed
- Last reviewed
- Not yet reviewed · card updated October 12, 2026
Limitations
- The selection criterion (hard for 2022-era models) is outdated — many tasks are now easy
- Macro-averaging across 23 very different tasks obscures important per-task variation
- Fixed task formats and small per-task sample sizes increase variance and contamination risk
- Some tasks test niche abilities (e.g., tracking shuffled objects) with unclear real-world relevance
- Chain-of-thought dependence makes scores highly sensitive to prompting strategy
When to use
- Assessing breadth of reasoning across qualitatively different task types in a single evaluation
- Comparing chain-of-thought effectiveness across diverse reasoning challenges
- Evaluating mid-range models that have not yet saturated the harder BBH tasks
When not to use
- Differentiating frontier models, where most BBH tasks are near-ceiling
- Drawing conclusions about specific reasoning capabilities — the aggregate score obscures per-task performance
Related benchmarks
- GSM8KReasoning · 2021Grade School Math 8K is a dataset of 8,792 grade-school-level math word problems requiring multi-step arithmetic reasoning. Once a key reasoning benchmark, it is now fully saturated and largely serves as a historical reference point.
- MATHReasoning · 2021A dataset of 12,500 competition-level mathematics problems spanning seven subjects from prealgebra to number theory. It remains more discriminating than GSM8K but is approaching saturation for the strongest frontier models.
- HellaSwagReasoning · 2019A commonsense natural language inference benchmark where models must choose the most plausible continuation of an everyday scenario. It was designed using adversarial filtering to be hard for models at the time of creation.