A survey on evaluation of multimodal large language models

Analysis

Why This Paper Matters

Multimodal large language models (MLLMs) have become a central focus in AI research, integrating text, image, audio, and video understanding. However, the rapid proliferation of these models has outpaced the development of robust evaluation frameworks. This survey addresses that gap by providing a comprehensive overview of MLLM architectures, training strategies, and evaluation methodologies. It is particularly timely as practitioners struggle to compare models that are evaluated on disparate benchmarks with inconsistent metrics.

The paper's significance lies in its systematic categorization of evaluation approaches, which helps researchers understand what aspects of MLLM performance are being measured and what is being overlooked. By synthesizing current practices, it lays the groundwork for more standardized and meaningful evaluation, which is essential for progress in the field.

Technical Contributions

The survey makes several key contributions:

• Framework taxonomy: It categorizes MLLM architectures, such as encoder-decoder and decoder-only models, and discusses how different modalities are fused.
• Training strategy overview: It reviews pretraining and fine-tuning strategies, including instruction tuning and reinforcement learning from human feedback (RLHF), and their impact on model capabilities.
• Evaluation taxonomy: It organizes evaluation methods into categories like benchmark-based, human-based, and model-based evaluation, and discusses metrics such as accuracy, F1, and BLEU, as well as more nuanced measures like hallucination rate and robustness.
• Gap analysis: It identifies limitations in current evaluation, such as lack of coverage for reasoning, safety, and real-world deployment.

Results

As a survey, the paper does not present new experimental results. Instead, it aggregates findings from existing studies, highlighting trends such as the dominance of benchmark-based evaluation and the underrepresentation of human evaluation. It notes that many MLLMs are evaluated on tasks like visual question answering and image captioning, but there is less focus on multimodal reasoning and alignment with human values. The survey also points out inconsistencies in evaluation setups, making cross-model comparisons difficult.

Significance

The broader impact of this survey is to encourage the AI community to develop more rigorous and standardized evaluation protocols for MLLMs. By clarifying the landscape, it helps researchers identify areas needing innovation, such as dynamic benchmarks and evaluation of emergent abilities. For practitioners, it offers a practical guide to selecting evaluation methods that align with their use cases. Ultimately, this work contributes to the responsible development of MLLMs by emphasizing the need for evaluation that goes beyond simple accuracy to include safety, fairness, and robustness.