# VQAv2

> Visual Question Answering v2 is a large-scale benchmark where models answer open-ended questions about natural images. It was the foundational VQA benchmark but is now saturated and limited by its simplistic question types.

## Claim

- **headline**: VQAv2 accuracy supports only a basic check of visual grounding on everyday COCO photographs and comparison with pre-LLM vision-language systems; it carries no information for ranking current multimodal models.
- **construct**: Basic visual question answering on natural images: objects, attributes, and simple relations
- **population**: Over 1.1 million open-ended questions about 200,000+ COCO images, each with 10 crowd-sourced answers, using complementary image pairs to reduce language bias
- **measures**: Basic visual question answering on natural photographs; Ability to identify objects, attributes, and simple spatial relationships in images; Short-answer generation grounded in visual content
- **doesNotMeasure**: Complex visual reasoning, multi-step inference, or deep scene understanding; Knowledge-intensive visual reasoning that requires domain expertise beyond common objects; Any capability that differentiates modern multimodal models, given full saturation; Ability to handle nuanced, ambiguous, or compositionally complex questions about images; Spatial reasoning, counting, or fine-grained attribute discrimination, despite containing such questions

## Instrument

- **itemFormat**: multimodal
- **specification**: An answer earns credit in proportion to how many of the 10 human annotators gave the same answer, with full credit at three or more matches.
- **anchor**: 10 crowd-sourced human answers per question (a consensus anchor)
- **evidenceView**: The normalized short answer string
- **mechanism**: deterministic
- **mechanismDetail**: String normalization and matching against the human answers; score = min(matches / 3, 1), averaged over subsets of the annotators.
- **signal**: Soft score between 0 and 1 per question
- **resolution**: Final answer per question
- **aggregation**: Mean VQA accuracy, overall and by answer type (yes/no, number, other)
- **metrics**: VQA accuracy
- **size**: 1.1M+ questions on 200K+ images
- **methodology**: Contains over 1.1 million questions about 200,000+ images from COCO. Questions are open-ended with short free-form answers. Each question is paired with a complementary image that gives a different answer, designed to reduce language bias. Evaluation uses a soft accuracy metric where an answer is considered correct if it matches at least 3 of 10 human annotator answers.

## Regime

- **typicalUses**: measure
- **exposure**: Public since 2017 on COCO images that are thoroughly represented in training data; test-split answers are withheld behind an evaluation server, but validation answers are public.

## Validation

- **summary**: VQAv2's evidence supported visual question answering claims for its era, but it no longer supports claims about modern multimodal models: the questions are too simple, the images too familiar, and the scores saturated. Its continued use reflects established practice rather than information; MMMU or MathVista support more meaningful multimodal claims.
- **evidence**: Separates current multimodal models; Answering requires looking at the image; The consensus metric credits correct answers precisely
- **status**: draft

## Lifecycle

- **saturation**: saturated
- **contamination**: high
- **successors**: mmmu; mathvista
- **lastReviewed**: 2026-10-12

---
Source: https://benchmaker.ai/benchmarks/vqav2