PaLM-E: An Embodied Multimodal Language Model

Free

Embodied Multimodal Language Model for Robotics and VQA

FreeFree tier

Inputs: text

Type

Open Source

Visit Website GitHub (38 stars)

About PaLM-E: An Embodied Multimodal Language Model

PaLM-E is an embodied multimodal language model that integrates continuous sensor modalities—such as images, state estimates, and other real-world observations—directly into the embedding space of a pre-trained large language model (PaLM). It is a decoder-only LLM that processes multi-modal sentences interleaving visual, state, and textual input encodings, trained end-to-end for multiple embodied tasks including sequential robotic manipulation planning, visual question answering, and captioning. PaLM-E demonstrates positive transfer across diverse tasks and domains, with its largest variant (PaLM-E-562B, 562B parameters) achieving state-of-the-art performance on OK-VQA while retaining generalist language capabilities. The model is open-source and available for research.

Key Features

Injects continuous sensor modalities (images, state estimates, etc.) into language model embeddings

End-to-end training with pre-trained PaLM for embodied tasks

Supports multi-modal sentences with interleaved visual, state, and textual encodings

Handles multiple embodied reasoning tasks: robotic manipulation planning, VQA, captioning

Exhibits positive transfer from joint training across internet-scale language, vision, and visual-language domains

Largest model PaLM-E-562B (562B parameters) achieves SOTA on OK-VQA

Decoder-only autoregressive architecture generating textual completions

Pros & Cons

Pros

Integrates real-world continuous sensor data directly into language model reasoning
Demonstrates positive transfer across tasks and domains when trained jointly
State-of-the-art performance on OK-VQA visual question answering benchmark
Capable of long-horizon planning with visual feedback
Open-source code and models available for research

Best For

Long-horizon robotic manipulation planning (e.g., 'bring me the rice chips from the drawer')Visual question answeringImage/video captioningMulti-stage task sequencing with visual feedback (e.g., sorting blocks by color)General embodied reasoning across different robots and observation modalities

FAQ

What is PaLM-E?

PaLM-E is an embodied multimodal language model that connects real-world sensor modalities (images, state estimates, etc.) to a large language model (PaLM) for reasoning and planning in robotics and visual-language tasks.

How does PaLM-E work?

PaLM-E encodes continuous observations (e.g., images, state estimates) into vectors of the same dimension as language token embeddings, then injects them into a decoder-only LLM (PaLM) which generates textual completions autoregressively.

What tasks can PaLM-E perform?

PaLM-E can perform sequential robotic manipulation planning, visual question answering, and captioning, including long-horizon tasks that incorporate visual feedback.

Is PaLM-E open source?

Yes, PaLM-E is open-source. The code and model are available via the project page and GitHub.

What is the largest PaLM-E model?

The largest model is PaLM-E-562B with 562 billion parameters, which also serves as a visual-language generalist with state-of-the-art performance on OK-VQA.