# TruthfulQA

> A benchmark testing whether language models generate truthful answers to questions that are designed to elicit common misconceptions, superstitions, and falsehoods. It specifically targets failure modes where models confidently reproduce popular but incorrect beliefs.

## Claim

- **headline**: TruthfulQA supports a targeted check of whether a model repeats common misconceptions on 817 adversarially chosen English questions; it does not support broad claims about honesty or hallucination rates.
- **construct**: Avoiding imitative falsehoods: popular misconceptions a model may reproduce from human-written text
- **population**: 817 English questions across 38 categories (including health, law, finance, and common misconceptions), written to elicit falsehoods
- **measures**: Resistance to generating commonly believed but factually incorrect information; Calibration between confidence and accuracy on adversarially chosen topics; Ability to say 'I don't know' or provide accurate caveats rather than confidently stating falsehoods; Truthfulness across categories including health, law, finance, and common misconceptions
- **doesNotMeasure**: General honesty or truthfulness beyond the specific 817 adversarially curated questions; Whether a model will be truthful in real-world deployment on arbitrary topics; Susceptibility to more sophisticated adversarial attacks or jailbreaks; Hallucination rates on factual questions that are not specifically designed to elicit errors; Truthfulness in long-form generation where subtle inaccuracies matter most

## Instrument

- **itemFormat**: open-ended
- **specification**: A generated answer is truthful if it asserts no falsehood (declining to answer counts as truthful) and informative if it actually answers; the multiple-choice variants score the choice or probability assigned to true reference answers.
- **anchor**: Author-written sets of true and false reference answers per question; human truthfulness labels used to train and validate the judges
- **evidenceView**: The generated answer (generation task), or the model's choice or likelihoods over reference answers (multiple-choice variants)
- **mechanism**: hybrid
- **mechanismDetail**: Generations are scored by human raters or by fine-tuned language-model judges (GPT-judge for truthfulness, GPT-info for informativeness); the multiple-choice variants are scored deterministically against the reference answers. Judge implementations vary across harnesses.
- **signal**: Truthful and informative verdicts (or probabilities) per answer
- **resolution**: Final answer per question
- **aggregation**: Percentage truthful, percentage informative, and percentage both (truthful*informative); accuracy for the multiple-choice variants
- **metrics**: truthfulness; informativeness; truthful*informative
- **size**: 817 questions across 38 categories
- **methodology**: Contains 817 questions across 38 categories, each designed to trigger a specific common falsehood. The generation task asks models to answer questions, and responses are scored for both truthfulness and informativeness using either human evaluation or a fine-tuned GPT-judge model. The multiple-choice variant presents true and false answer options. Key insight: a model that always says 'I have no comment' scores highly on truthfulness but low on informativeness.

## Regime

- **typicalUses**: measure; optimize
- **exposure**: A small public question set since 2021, included in the Hugging Face Open LLM Leaderboard (v1) and in alignment-training reports. Targeted training can raise scores without broader truthfulness gains, so the set has been under direct optimization pressure, and contamination risk is high.

## Validation

- **summary**: TruthfulQA's evidence supports a narrow claim: it probes a real failure mode — confidently reproducing popular falsehoods — within specific misconception categories. It does not support claims about general truthfulness, because models can improve through targeted training without becoming broadly more truthful, and automated judging adds its own uncertainty.
- **evidence**: Targets a real failure mode: imitative falsehoods; Score gains indicate broader truthfulness; Automated judging matches human truthfulness labels; Evasive answers cannot score well
- **status**: draft

## Lifecycle

- **saturation**: active
- **contamination**: high
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/truthfulqa