Skip to content
Benchmaker

Multimodal · Evaluator card

VQAv2

2017Yash Goyal et al.Paper (opens in a new tab)

Visual Question Answering v2 is a large-scale benchmark where models answer open-ended questions about natural images. It was the foundational VQA benchmark but is now saturated and limited by its simplistic question types.

View as Markdown

Evidence notes pending review. The verdicts on this card are drafts derived from the benchmark's documentation and published sources; no reviewer has checked them against those sources yet.

Headline claim

VQAv2 accuracy supports only a basic check of visual grounding on everyday COCO photographs and comparison with pre-LLM vision-language systems; it carries no information for ranking current multimodal models.

At a glance

Mechanism
Deterministic checkerPrior: execution feedback, rung 2 of 4
Item format
Multimodal
Size
1.1M+ questions on 200K+ images
Typical uses
Measure
Saturation
Saturated
Contamination
High risk
Year
2017

1Claim

What property are we trying to infer, over what world, and for what decision?

Construct
Basic visual question answering on natural images: objects, attributes, and simple relations
Population
Over 1.1 million open-ended questions about 200,000+ COCO images, each with 10 crowd-sourced answers, using complementary image pairs to reduce language bias
Measures
  • Basic visual question answering on natural photographs
  • Ability to identify objects, attributes, and simple spatial relationships in images
  • Short-answer generation grounded in visual content
Does not measure
  • Complex visual reasoning, multi-step inference, or deep scene understanding
  • Knowledge-intensive visual reasoning that requires domain expertise beyond common objects
  • Any capability that differentiates modern multimodal models, given full saturation
  • Ability to handle nuanced, ambiguous, or compositionally complex questions about images
  • Spatial reasoning, counting, or fine-grained attribute discrimination, despite containing such questions

2Instrument

How is the judgment actually produced?

Mechanism

Deterministic checker

String normalization and matching against the human answers; score = min(matches / 3, 1), averaged over subsets of the annotators.

Hierarchy prior: execution feedback, rung 2 of 4. A prior over expected grounding, not a verdict on validity: the evidence notes under Validation say what has been measured.

Item format
Multimodal
Specification
An answer earns credit in proportion to how many of the 10 human annotators gave the same answer, with full credit at three or more matches.
Anchor
10 crowd-sourced human answers per question (a consensus anchor)
Evidence view
The normalized short answer string
Signal
Soft score between 0 and 1 per question
Resolution
Final answer per question
Aggregation
Mean VQA accuracy, overall and by answer type (yes/no, number, other)
Metrics
VQA accuracy
Size
1.1M+ questions on 200K+ images
Methodology
Contains over 1.1 million questions about 200,000+ images from COCO. Questions are open-ended with short free-form answers. Each question is paired with a complementary image that gives a different answer, designed to reduce language bias. Evaluation uses a soft accuracy metric where an answer is considered correct if it matches at least 3 of 10 human annotator answers.

3Regime

What causal role does the signal play, and what happens when a system adapts to it?

Typical uses
  • Measure · supports inference and decision
Exposure
Public since 2017 on COCO images that are thoroughly represented in training data; test-split answers are withheld behind an evaluation server, but validation answers are public.

4ValidationDraft

Why should we believe that the signal means what we claim it means — before and after the system adapts to it?

VQAv2's evidence supported visual question answering claims for its era, but it no longer supports claims about modern multimodal models: the questions are too simple, the images too familiar, and the scores saturated. Its continued use reflects established practice rather than information; MMMU or MathVista support more meaningful multimodal claims.

Evidence notes2 partially supported · 1 not supported

  1. Not supported

    Separates current multimodal models

    Scores approach human performance and are saturated.

  2. Partially supported

    Answering requires looking at the image

    Complementary image pairs reduced but did not remove language priors; strong text-only baselines exist.

  3. Partially supported

    The consensus metric credits correct answers precisely

    Matching against 10 human answers tolerates annotator variation but is noisy, and approximate answers can receive full credit.

5Lifecycle

Does the validity survive over time, and are we still evaluating the right thing?

Saturation
Saturated

Whether top systems still differ, or crowd the ceiling.

Contamination
High risk

How likely test items, or close variants, have leaked into training data.

Successors
Last reviewed
Not yet reviewed · card updated October 12, 2026

Limitations

  • Saturated for modern vision-language models, with scores approaching human performance
  • Most questions are superficial (object identification, color, counting) and do not require deep reasoning
  • The soft accuracy metric is noisy and allows approximate answers to receive full credit
  • COCO images are thoroughly represented in training data, causing massive contamination
  • Language biases remain despite v2's complementary image pairing — strong text-only baselines exist

When to use

  • Historical comparison with pre-LLM era vision-language systems
  • Testing very basic visual grounding in early-stage multimodal model development

When not to use

  • Any modern multimodal model evaluation — scores are at ceiling and uninformative
  • Assessing complex visual reasoning, knowledge integration, or real-world visual understanding
  • Drawing any conclusions about multimodal capability differences between competitive models

Related benchmarks