Are we on the right way for evaluating large vision-language models?

Analysis

Why This Paper Matters

This paper addresses a fundamental yet often overlooked issue in the evaluation of large vision-language models (LVLMs): the extent to which current benchmarks actually test visual reasoning versus relying on language priors. As LVLMs become increasingly capable and are deployed in real-world applications, ensuring that evaluations genuinely measure visual understanding is crucial. The authors demonstrate that many popular benchmarks contain samples where the correct answer can be inferred from text alone, without any visual input. This finding has significant implications for the research community, as it suggests that reported performance gains may be inflated and that models may not be as visually competent as believed.

By systematically identifying this problem, the paper serves as a wake-up call for the field. It underscores the need for more careful dataset design and evaluation protocols that isolate visual reasoning from language-based shortcuts. This is particularly important for applications where visual grounding is critical, such as autonomous driving, medical imaging, and assistive technologies.

Technical Contributions

• Identification of visual-unnecessary samples: The authors propose a simple yet effective method to detect samples where visual content is not needed—by comparing model performance with and without images.
• Quantitative analysis of existing benchmarks: They apply their method to several popular LVLM evaluation datasets and report the proportion of samples that can be answered without visual input.
• Framework for better evaluation: The paper outlines criteria for constructing evaluation sets that ensure visual content is essential, such as requiring fine-grained spatial or color reasoning.

Results

The authors report that in some benchmarks, over 30% of samples can be answered correctly using only text. For example, on the VQA v2 dataset, models achieve near-chance accuracy without images, but on certain subsets, performance drops significantly when visual input is removed. These results highlight the variability in visual necessity across different evaluation tasks.

Significance

This work has broad implications for the AI community. It challenges the validity of current LVLM leaderboards and encourages researchers to develop more robust evaluation methodologies. By exposing the reliance on language priors, it also informs model design—future models may need to be trained with explicit visual grounding objectives. Ultimately, this paper contributes to more trustworthy and meaningful progress in multimodal AI.