Teaching MLLMs to Think with Images

Analysis

Why This Paper Matters

This paper addresses a critical gap in multimodal large language models (MLLMs): the lack of explicit visual grounding in their reasoning chains. While recent advances in reinforcement learning (RL) have enabled models to generate chain-of-thought reasoning, existing open-source visual reasoning models typically reason in pure text, ignoring the visual context that is essential for tasks like visual question answering. GRIT introduces a grounded reasoning paradigm where models interleave natural language with bounding box coordinates, making the reasoning process transparent and visually anchored. This is a significant step toward more interpretable and reliable multimodal AI.

The data efficiency claim is particularly striking. Requiring as few as 20 image-question-answer triplets to train effective grounded reasoning is a paradigm shift from the typical large-scale supervised fine-tuning approaches. This opens up possibilities for rapid adaptation to new domains with minimal annotated data, which is crucial for real-world applications where data is scarce or expensive to obtain.

Technical Contributions

• Grounded Reasoning Paradigm: GRIT generates reasoning chains that interleave text and bounding box coordinates, explicitly referencing image regions during reasoning. This is a novel output format that combines language and spatial grounding.
• GRPO-GR Algorithm: An extension of the GRPO RL algorithm, designed to optimize the grounded reasoning format. It uses robust rewards that only depend on final answer accuracy and format compliance, eliminating the need for expensive annotations of reasoning chains or bounding boxes.
• Data Efficiency: The method achieves strong performance with as few as 20 training triplets, showcasing the power of RL with well-designed rewards over supervised learning.
• Unification of Reasoning and Grounding: GRIT demonstrates that reasoning and grounding can be jointly learned, rather than treated as separate tasks, leading to more coherent and visually grounded outputs.

Results

The paper reports that GRIT effectively trains MLLMs to produce coherent and visually grounded reasoning chains. While specific quantitative metrics are not provided in the abstract, the claim of successful unification of reasoning and grounding is supported by comprehensive evaluations. The data efficiency result—requiring only 20 triplets—is a standout achievement, suggesting that the method can achieve strong performance with minimal data. However, the lack of concrete numbers (e.g., accuracy comparisons with baselines) makes it difficult to assess the magnitude of improvement over existing methods.

Significance

GRIT has the potential to influence how MLLMs are trained for vision-language tasks, shifting from annotation-heavy supervised learning to RL-based approaches that leverage robust rewards. The ability to generate grounded reasoning chains could improve interpretability and trust in AI systems, especially in applications like medical imaging, autonomous driving, and assistive technologies. Moreover, the data efficiency could democratize access to advanced multimodal reasoning, enabling smaller teams and organizations to build specialized models without massive datasets. Future work may extend GRIT to other modalities (e.g., audio, video) and explore more complex grounding formats beyond bounding boxes.