Preprint
Large Language Models

Multi-Agent Collaboration for Multimodal LLMs

James Y. Huang, Sheng Zhang, Qianchu Liu, Guanghui Qin, Tinghui Zhu, Tristan Naumann, Muhao Chen, H. Poon
November 24, 2025arXiv.org6 citations
arXivPDFDOI

6

Citations

0

Influential Citations

arXiv.org

Venue

2025

Year

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in challenging, knowledge-intensive reasoning tasks. However, extending LLMs to perceive and reason over a new modality (e.g., vision), often requires costly development of large-scale vision language models (VLMs) with LLMs as backbones. Smaller VLMs are more efficient and adaptable but often lack the broad knowledge and reasoning capabilities of frontier LLMs. In this work, we propose BeMyEyes, a modular, multi-agent framework for extending LLMs to multimodal reasoning by orchestrating collaboration between efficient, adaptable VLMs as perceivers and powerful LLMs as reasoners through conversations. We then introduce a data synthesis and supervised fine-tuning pipeline to train the perceiver agent to effectively collaborate with the reasoner agent. By combining the complementary strengths of perception and reasoning agents, BeMyEyes avoids the need for training large-scale multimodal models, preserves the generalization and reasoning capabilities of LLMs, and allows flexible extension to new domains and modalities. Experiments show that our framework unlocks the multimodal reasoning capabilities for LLMs, enabling a lightweight and fully open-source solution, i.e. equipping text-only DeepSeek-R1 with Qwen2.5-VL-7B perceiver, to outperform large-scale proprietary VLMs such as GPT-4o on a wide range of knowledge-intensive multimodal tasks. These results demonstrate the effectiveness, modularity, and scalability of our multi-agent approach for building future multimodal reasoning systems.

Analysis

Why This Paper Matters

This paper addresses a critical bottleneck in multimodal AI: the prohibitive cost of training large-scale vision-language models (VLMs) from scratch. While frontier LLMs like GPT-4 and DeepSeek-R1 excel at reasoning, extending them to vision typically requires building massive VLMs with LLM backbones, which is expensive and resource-intensive. Smaller VLMs are efficient but lack the broad knowledge of their larger counterparts. BeMyEyes proposes a pragmatic middle ground: a multi-agent framework that lets a small, efficient VLM act as a "perceiver" and a powerful text-only LLM as a "reasoner," collaborating through natural language conversations. This modular design is significant because it decouples perception from reasoning, allowing each component to be independently upgraded or swapped without retraining the entire system. The paper also introduces a data synthesis and fine-tuning pipeline to train the perceiver to produce descriptions that are maximally useful for the reasoner, effectively bridging the modality gap.

Technical Contributions

  • Modular Multi-Agent Framework: BeMyEyes separates perception (handled by a small VLM) and reasoning (handled by a text-only LLM), enabling flexible composition of different models.
  • Collaborative Conversation Protocol: The perceiver and reasoner engage in iterative dialogue, where the perceiver provides visual descriptions and the reasoner asks clarifying questions or requests additional details.
  • Data Synthesis and Fine-Tuning Pipeline: The authors generate training data by simulating the multi-agent interaction, then fine-tune the perceiver via supervised learning to produce descriptions that align with the reasoner's needs.
  • Open-Source Lightweight System: The final system uses DeepSeek-R1 (text-only) as the reasoner and Qwen2.5-VL-7B as the perceiver, both fully open-source and lightweight (7B parameters for the VLM).

Results

The paper reports that the BeMyEyes system, combining DeepSeek-R1 with a Qwen2.5-VL-7B perceiver, outperforms GPT-4o on a wide range of knowledge-intensive multimodal tasks. This is a striking result because GPT-4o is a massive proprietary model, while BeMyEyes uses a 7B-parameter VLM and a text-only LLM. The experiments cover multiple benchmarks, demonstrating that the multi-agent approach not only matches but exceeds the performance of large-scale VLMs. The authors attribute this to the complementary strengths: the small VLM provides focused visual perception, while the LLM brings broad world knowledge and reasoning ability.

Significance

This work has broad implications for the AI field. It provides a practical, cost-effective pathway to extend any text-only LLM to multimodal reasoning, which could democratize access to multimodal AI capabilities. The modular design means that as better perceivers or reasoners become available, they can be plugged in without retraining the whole system. This aligns with the trend toward composable AI systems. Additionally, the open-source nature of the solution lowers the barrier for researchers and practitioners to build multimodal applications. The paper also highlights the potential of multi-agent collaboration as a paradigm for integrating different AI capabilities, which could inspire similar approaches for other modalities (e.g., audio, video) or tasks.

multi-agentmultimodallarge-language-modelsvision-language-modelsreasoningcollaborationsupervised-fine-tuning
Large Language ModelsFine TuningMultimodalReasoningComputer Vision

Related Papers