MolmoAct: Generating Spatial Action Maps for Robots to Plan and Execute Text-Based Instructions

## The Challenge of Long-Horizon Robotic Tasks Robotic systems have made impressive strides in handling short, simple instructions like 'pick up the block.' However, they often falter with complex, multi-step directives such as 'clear the table by stacking dishes on the shelf.' Traditional approaches rely on reactive policies that act immediately on visual inputs and text, lacking deliberate planning. This leads to errors in spatial reasoning, where robots misjudge object positions or interaction points. Enter MolmoAct, a breakthrough from the Allen Institute for AI (AI2), which introduces proactive spatial planning through interpretable action maps. MolmoAct addresses this by decoupling perception from action: it first generates visual heatmaps highlighting key regions for attention, movement, and interaction, then translates these into precise robot controls. This mimics human-like foresight, allowing robots to 'plot their course' before moving. ## Core Architecture of MolmoAct At its heart, MolmoAct is built on the [Molmo family of vision-language models](https://github.com/allenai/molmo), which excel at multimodal understanding. Available in sizes 1B, 7B, and 72B parameters, these models process RGB images and text instructions to output low-level actions: continuous pixel displacements (Δx, Δy) and binary gripper states (open/close). ### Step-by-Step Breakdown 1. **Input Processing**: A single RGB frame pairs with a natural language goal, e.g., "Grasp the red apple on the conveyor belt." 2. **Spatial Reasoning Phase**: Instead of direct action prediction, MolmoAct outputs three probabilistic heatmaps: - **Attention Map**: Highlights regions to focus on (e.g., the apple). - **Movement Map**: Indicates where the robot should navigate its end-effector. - **Interaction Map**: Pinpoints grasp locations. These heatmaps are derived from the model's latent representations, providing human-interpretable visualizations. For instance, in a cluttered kitchen scene, the attention map might glow brightly over utensils, guiding subsequent steps. 3. **Action Inference**: Expected values from the heatmaps compute the final Δx, Δy, and gripper command. This argmax-like operation ensures precise, grounded actions. This two-stage process—reasoning then acting—sets MolmoAct apart from end-to-end policies like RT-2 or OpenVLA, which blend everything into a black box. By making intermediate spatial predictions explicit, MolmoAct offers better controllability and debugging. #### Visual Example Imagine a robot arm facing a table with scattered objects. Instruction: "Move the blue cup to the tray." - Attention heatmap: Peaks at the blue cup. - Movement heatmap: Path from current position to cup. - Interaction heatmap: Grasp point on cup handle. Resulting action: Smooth Δx/Δy shift to the cup, gripper closes. Demo videos on the project's page showcase this in real-time. ## Training and Data Pipeline MolmoAct leverages the massive Open X-Embodiment dataset, filtering 187,000 trajectories from diverse robots (e.g., RT-1, Bridge, LIBERO). Key augmentations include: - **Frame Sampling**: Dense sequences for fine-grained motion. - **Language Annotation**: GPT-4o generates 20+ task descriptions per trajectory, boosting generalization. Training uses a supervised fine-tune on Molmo backbones: ```python # Pseudocode for training loop (inspired by repo) import torch from molmoact.model import MolmoAct model = MolmoAct.from_pretrained('molmo-7b') optimizer = torch.optim.AdamW(model.parameters()) for batch in dataloader: images, texts, actions = batch['image'], batch['text'], batch['action'] heatmaps_pred = model.reason(images, texts) # Outputs spatial maps actions_pred = heatmaps_to_actions(heatmaps_pred) loss = mse_loss(actions_pred, actions) + kl_div(heatmaps_pred, gt_heatmaps) loss.backward() ``` This heatmap-supervised objective (MSE on actions + KL on maps) enforces spatial accuracy. No RLHF or imitation learning tweaks—pure supervised scaling works wonders. ## Performance Comparisons MolmoAct shines on standardized benchmarks, emphasizing long-horizon tasks: | Benchmark | Metric | MolmoAct-72B | RT-2-X (72B) | OpenVLA (7B) | PaliGemma (3B) | |-----------|--------|--------------|--------------|--------------|-----------------| | LIBERO (10 tasks) | Success Rate | 54.3% | 42.1% | - | 28.5% | | Bridge V2 (Roboturk) | SR | 68.2% | 55.4% | 62.1% | - | | Bridge V2 (Bridge) | SR | 72.1% | - | 65.3% | 51.2% | | CALVIN (seen) | SR | 47.8% | - | 39.2% | - | The 72B variant leads by 10-20% on average, with heatmaps explaining successes (e.g., correctly localizing occluded objects). Smaller 1B/7B models still beat 7B baselines, showing efficient scaling. ### Ablation Insights - **No Heatmaps**: Direct action prediction drops 15% SR—proving spatial planning's value. - **Molmo Backbone**: Outperforms Qwen2-VL or Llama-3.2 by 8-12% due to superior vision grounding. ## Real-World Applications and Deployment MolmoAct's zero-shot capabilities extend to unseen robots/manipulators via pixel-space actions—no calibration needed. Practical uses: - **Warehouse Automation**: 'Sort packages by color' on dynamic conveyors. - **Home Assistants**: 'Fold laundry from basket' in varied lighting. - **Surgical Robots**: 'Position tool at incision site' from scans. Integration is straightforward via the [MolmoAct GitHub repository](https://github.com/allenai/molmoact), offering inference code, eval suites, and model weights (Apache 2.0 licensed). Run locally: ```bash git clone https://github.com/allenai/molmoact cd molmoact pip install -r requirements.txt python demo.py --model molmoact-7b --image kitchen.jpg --prompt "Pick up the fork" ``` Outputs heatmaps and actions instantly on consumer GPUs. ## Comparisons to Prior Work - **Vs. End-to-End VLMs (e.g., RT-2, Octo)**: MolmoAct's maps add interpretability; failures are traceable ("missed interaction zone"). - **Vs. Diffusion Policies**: No iterative sampling—single-pass speed (50Hz+). - **Vs. Hierarchical Planners**: Simpler, no high-level subgoals needed. Strengths: Scalable training, open-source, broad generalization. Limitations: Single-frame input (future work: video); assumes static scenes. ## Future Directions and Broader Impact AI2 plans video-conditioned variants and RL integration for exploration. By open-sourcing via [Molmo](https://github.com/allenai/molmo), they democratize embodied AI—researchers can fine-tune for custom robots in days. This isn't just incremental; MolmoAct shifts paradigms toward reasoned robotics, paving the way for reliable household/ industrial deployment. Experiment yourself to see spatial intelligence in action. --- <div style="text-align: center; margin-top: 2rem;"> <a href="https://www.deeplearning.ai/the-batch/molmoact-creates-spatial-maps-for-robots-to-plot-their-actions-before-executing-text-directions/" target="_blank" rel="noopener noreferrer" class="view-full-resource-btn" style="display: inline-block; background-color: #f97316; color: white; padding: 12px 24px; border-radius: 8px; text-decoration: none; font-weight: 600; transition: background-color 0.2s;">View Full Resource</a> </div>