Eagle 2

Analysis

Why This Paper Matters

Vision-language models (VLMs) have made remarkable progress, but much of their success hinges on the quality and quantity of post-training data. The open-source community often lacks a systematic methodology for building such data from scratch. Eagle 2 addresses this gap by presenting a complete, transparent pipeline for data collection, filtering, selection, augmentation, and formatting. This paper is particularly valuable because it shows that a 9B-parameter model can rival or surpass much larger models (including GPT-4V and GPT-4o on several benchmarks) through careful data engineering and a vision-centric architecture. The emphasis on reproducibility and detailed ablation studies makes this a practical guide for practitioners.

How It Works

Eagle 2 builds on the Cambrian dataset and LLaVA's two-stage training recipe, but introduces several key innovations. The data strategy is the core contribution. The authors employ both passive gathering (monitoring arXiv and HuggingFace) and proactive searching (error analysis to identify model weaknesses). They also convert non-QA data (e.g., Google Landmark) into VQA format using rules or auto-labeling tools. A diversity metric is defined to quantify the relevance of new data sources to the existing pool, using SSCD image embeddings and all-mpnet-base-v2 text embeddings.

Data filtering removes mismatched QA pairs, irrelevant image-question pairs, repeated texts, and numeric formatting issues. Subset selection uses K-means clustering on SSCD embeddings to ensure balanced representation across data types (e.g., chart types). Data augmentation includes generating fine-grained descriptions via third-party VLMs, adding chain-of-thought explanations, rule-based QA generation, and expanding short answers.

A critical architectural innovation is the tiled mixture of vision encoders. Following Eagle, SigLIP (448x448 input) and ConvNeXt-XXLarge (512x512 input) are used. Image tiling handles arbitrarily high-resolution images. PixelShuffle downsamples SigLIP features by 2x to match ConvNeXt's output shape (16x16), then features are concatenated along the channel dimension and aligned with the LLM via an MLP layer.

The training recipe introduces a three-stage approach. Stage-1 trains the MLP connector. Stage-1.5 is a new pre-training stage on a larger dataset to reduce dependency on SFT data. Stage-2 is the full fine-tuning with SFT data. The authors also propose a balance-aware greedy knapsack algorithm for data packing, which creates packs with uniform length distribution, balancing loss weights between long and short samples.

Results

Eagle2-9B achieves competitive results across 14 benchmarks. It outperforms InternVL2-8B and MiniCPM-v2.6 on all benchmarks, and beats Qwen2-VL-7B on 9 out of 14. It also performs competitively against much larger models like InternVL2-26B, LLaVa-OneVision-72B, and LLaMa-3.2-90B-Vision. Notably, it comprehensively surpasses GPT-4V except on MMVet and MMMU, and surpasses GPT-4o on ChartQA, OCRBench, and MathVista.

Ablation studies show that Stage-1.5 improves performance by an average of 3.9%, data formatting and filtering yield a 45-point gain on OCRBench, and the mixture of vision encoders improves performance on 12 out of 14 benchmarks, especially for documents, charts, and OCR.

Significance

Eagle 2 provides a practical, data-centric roadmap for building high-performing VLMs without relying on proprietary data or massive model sizes. Its detailed ablation studies offer actionable insights for practitioners: careful data selection and formatting can yield substantial gains, and a three-stage training recipe with a balanced packing algorithm improves training efficiency and model quality. The tiled mixture of vision encoders demonstrates that combining complementary visual features can significantly enhance understanding of complex visual spaces. This work sets a new standard for open-source VLM development and underscores the importance of systematic data engineering in achieving frontier-level performance.