# BBH

> BIG-Bench Hard is a curated subset of 23 tasks from the BIG-Bench suite where language models initially performed below average human raters. It tests diverse reasoning skills including logical deduction, causal reasoning, and algorithmic thinking.

## Claim

- **headline**: BBH accuracy supports per-task comparisons of few-shot chain-of-thought performance on 23 short, formulaic reasoning tasks for mid-range models; its macro-average no longer separates frontier models.
- **construct**: Breadth of short-form reasoning: logical deduction, temporal and causal reasoning, algorithmic manipulation
- **population**: 23 BIG-Bench tasks on which 2022-era models scored below average human raters, each with a fixed answer format, evaluated with few-shot chain-of-thought prompts
- **measures**: Performance across a diverse set of reasoning tasks that were originally challenging for LLMs; Logical deduction, temporal reasoning, causal judgment, and algorithmic manipulation abilities; Effectiveness of chain-of-thought prompting on tasks that benefit from step-by-step reasoning; Breadth of reasoning capability across qualitatively different task types
- **doesNotMeasure**: Deep reasoning — many tasks have become easy for modern models and no longer discriminate; Whether a model reasons or merely pattern-matches, as the fixed task formats are susceptible to memorization; Real-world reasoning under ambiguity, incomplete information, or conflicting evidence; Sustained reasoning over long problems — most BBH tasks are short and self-contained

## Instrument

- **itemFormat**: multiple-choice
- **specification**: An example is correct when the final answer extracted after the chain of thought exactly matches the target: an option label or a constrained free-form string such as a number or a sorted word list.
- **anchor**: Fixed target answer per example from the BIG-Bench task files
- **evidenceView**: The extracted final answer; the reasoning chain is not graded
- **mechanism**: deterministic
- **mechanismDetail**: Exact match of the extracted answer against the target.
- **signal**: Binary correct/incorrect per example
- **resolution**: Final answer per example
- **aggregation**: Accuracy per task, macro-averaged across the 23 tasks
- **metrics**: accuracy
- **size**: 6,511 examples across 23 tasks
- **methodology**: Selects 23 tasks from the larger BIG-Bench suite based on a criterion that models performed below human baselines at the time of curation. Each task has a specific format (multiple-choice or free-form with constrained answers). Evaluation uses exact-match accuracy per task, typically reported as a macro-average across all 23 tasks. Chain-of-thought prompting is the standard evaluation protocol.

## Regime

- **typicalUses**: measure
- **exposure**: Public since 2022 with fixed few-shot chain-of-thought exemplars. The BIG-bench task files carry a canary string meant to keep them out of training corpora, but fixed formats and small per-task samples keep contamination risk material. It was part of the Hugging Face Open LLM Leaderboard (v2).

## Validation

- **summary**: BBH's evidence supported its original purpose — identifying reasoning tasks that were hard for 2022-era models — and its spread of qualitatively different task types remains useful. Model progress has eroded discrimination on many of the 23 tasks, so the aggregate is increasingly driven by a shrinking set of still-hard tasks; per-task results support more specific claims than the headline number.
- **evidence**: The macro-average summarizes reasoning breadth; Separates frontier models; Scores are stable across prompting strategies
- **status**: draft

## Lifecycle

- **saturation**: near-saturated
- **contamination**: medium
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/bbh