Eagle

Analysis

Why This Paper Matters

Multimodal Large Language Models (MLLMs) have made remarkable progress by integrating vision encoders with LLMs, but the design space for choosing and combining these encoders remains underexplored. Most prior work relies on a single vision encoder (e.g., CLIP) or ad-hoc combinations without systematic justification. Eagle fills this gap by conducting a thorough exploration of the mixture-of-vision-encoders paradigm, revealing that simple channel concatenation of multiple diverse encoders, combined with a pre-alignment training strategy, yields substantial and consistent improvements across a wide range of benchmarks. This matters because it provides a clear, reproducible recipe for building better MLLMs without requiring massive computational resources or proprietary data.

How It Works

Eagle builds on the LLaVA-1.5 architecture, which consists of a vision encoder, a projection layer, and a language model (Vicuna-7B or Llama3-8B/13B). The key innovation is the systematic exploration of three design dimensions: vision encoder selection, resolution adaptation, and fusion strategy.

Vision Encoder Selection: The authors experiment with six diverse vision encoders: CLIP (contrastive learning on image-text pairs), ConvNeXt (large-scale supervised classification), SAM (semantic segmentation), DINOv2 (self-supervised learning), Pix2Struct (document understanding), and EVA-02 (object-centric tasks). Each encoder brings unique strengths—for example, Pix2Struct excels at text recognition while EVA-02 performs best on object hallucination benchmarks.

Resolution Adaptation: To handle high-resolution inputs, Eagle compares two strategies: tiling (dividing the image into patches) and direct interpolation of position embeddings. A critical finding is that unfreezing the vision encoder during training is essential when interpolating to resolutions different from pre-training. The best configuration uses direct interpolation to 448×448 with an unfrozen CLIP encoder, achieving performance close to InternVL despite using a much smaller model.

Fusion Strategy: Eagle evaluates five fusion methods: sequence append (concatenating token sequences), channel concatenation (merging along channel dimension), LLaVA-HR (injecting high-res features via adapter), Mini-Gemini (cross-attention with CLIP queries), and deformable attention. Surprisingly, the simplest method—channel concatenation—achieves the best average performance while maintaining high throughput. Complex injection-based methods underperform on text-heavy tasks like TextVQA, likely because CLIP features dominate the visual tokens.

Pre-Alignment Training: A key contribution is the three-stage training pipeline. First, each vision expert is individually aligned with a frozen language model using next-token prediction on SFT data. Second, all experts are combined and only the projector is trained on image-text pairs. Finally, the entire model is fine-tuned on SFT data. This pre-alignment step stabilizes training and mitigates the inherent biases of each expert, leading to better overall performance compared to simply unfreezing all encoders during SFT.

Results

Eagle achieves state-of-the-art results across multiple benchmarks. With the Vicuna-7B backbone and five vision encoders (Eagle-X5), it scores 64.7 on GQA, 80.6 on VQAv2, 77.3 on TextVQA, 87.1 on POPE, 76.4 on SEED, and 2298 on MME. These results often surpass larger models like InternVL (6B parameters) and Cambrian-1 (13B parameters). On OCR and document understanding tasks, Eagle shows particularly strong performance, benefiting from the inclusion of Pix2Struct and high-resolution adaptation.

When trained on the same data as Cambrian-1, Eagle-X5 outperforms Cambrian-1 on OCR and chart categories by a significant margin, with consistent improvements across general knowledge and vision-centric categories. The normalized average performance across all benchmarks shows a clear upward trend as more vision encoders are added, confirming that diversity in visual features is beneficial.

Significance

Eagle provides a principled and practical framework for designing high-performance MLLMs. The key takeaway is that more is not always better—but carefully selected diversity, combined with simple fusion and proper training alignment, can unlock substantial gains. The finding that channel concatenation outperforms sophisticated attention-based fusion methods challenges the prevailing trend toward architectural complexity. This work offers clear guidelines for practitioners: use multiple complementary vision encoders, unfreeze them during training, apply channel concatenation, and adopt the pre-alignment strategy. The code and models are publicly available, enabling reproducible research and further innovation in multimodal AI.