BLIP-3 (xGen-MM)

Analysis

Why This Paper Matters

Large multimodal models (LMMs) that can understand images, text, and their interleaved combinations are becoming central to AI applications. However, most state-of-the-art LMMs are closed-source or lack detailed documentation of their training data and recipes. BLIP-3 (xGen-MM) addresses this gap by releasing a complete open-source framework: curated datasets, model architectures, training procedures, and pre-trained weights. This transparency enables reproducibility and further innovation, especially for in-context learning across multiple images—a capability critical for real-world tasks like document analysis, visual reasoning, and interactive assistants.

How It Works

The xGen-MM framework builds on the Salesforce xGen initiative. Its architecture combines a Vision Transformer (ViT) with a perceiver resampler that compresses image embeddings into a fixed number of tokens, feeding them into a pre-trained LLM (phi3-mini). A key innovation is the dynamic high-resolution encoding strategy used during fine-tuning and post-training: instead of resizing images to a fixed low resolution, the model splits an image into patches, encodes each patch separately, and concatenates them with a downsampled global view. This preserves fine-grained details, crucial for OCR and document understanding.

The training pipeline consists of four stages:

1. Pre-training on 100B multimodal tokens from a mixture of interleaved datasets (MINT-1T, OBELICS) and curated caption datasets (BLIP3-KALE, BLIP3-OCR-200M, BLIP3-GROUNDING-50M). The pre-training objective is next-token prediction.
2. Supervised fine-tuning (SFT) on 1M instruction-following examples from public datasets covering captioning, VQA, chart/document understanding, and pure text.
3. Interleaved multi-image SFT to enhance multi-image reasoning and in-context learning, mixing multi-image and single-image data to prevent catastrophic forgetting.
4. Post-training with two stages: Direct Preference Optimization (DPO) using the VLFeedback dataset to improve truthfulness, and safety fine-tuning using the VLGuard dataset to reduce harmful outputs.

Results

The pre-trained base model shows strong few-shot learning, outperforming comparably sized models like MM1-3B and Idefics-9B on OCR tasks (TextCaps, TextVQA) and VQA-v2. Performance improves with more shots, demonstrating effective in-context learning.

The SFT model (xGen-MM-instruct) surpasses prior baselines on general VQA and visual perception benchmarks. The interleave version achieves the highest overall scores across all single-image benchmarks.

On multi-image benchmarks, the single-image SFT model performs poorly, while the interleave fine-tuned model shows dramatic improvement, confirming the necessity of multi-image training for such tasks.

DPO reduces hallucination rates (HallusionBench, POPE) and safety fine-tuning lowers the attack success rate on VLGuard, both with slight gains in helpfulness.

Significance

BLIP-3 provides a complete, open-source recipe for building competitive multimodal models. Its modular design—separating pre-training, instruction tuning, multi-image tuning, and post-training—allows practitioners to adapt each stage to their needs. The release of curated datasets (BLIP3-KALE, BLIP3-OCR-200M, BLIP3-GROUNDING-50M) addresses data scarcity in OCR and grounding tasks. The two-stage post-training offers a practical template for improving truthfulness and safety without sacrificing performance. This work democratizes access to state-of-the-art LMM technology and sets a new standard for reproducibility in the field.