Transformers are sample-efficient world models

Analysis

Why This Paper Matters

Model-based reinforcement learning (RL) promises improved sample efficiency by learning a world model and training policies within it. However, learning accurate world models from high-dimensional pixel observations remains challenging. IRIS addresses this by leveraging the Transformer architecture, which has revolutionized sequence modeling in NLP and vision, to build a sample-efficient world model. This is significant because it demonstrates that Transformers can effectively model complex environment dynamics directly from pixels, achieving human-level performance on the Atari 100k benchmark with only 100k environment steps—a fraction of the data typically required.

The paper's importance is further underscored by its timing: as Transformers become ubiquitous in AI, exploring their potential in RL is a natural and impactful direction. IRIS shows that a single Transformer can jointly model observations, actions, and rewards, enabling accurate long-term predictions. This contrasts with prior model-based methods that often rely on recurrent networks or specialized architectures. By releasing code and models, the authors facilitate reproducibility and further research, potentially accelerating progress in sample-efficient RL.

Technical Contributions

• Discrete Autoencoder for Tokenization: IRIS first trains a discrete autoencoder (VQ-VAE) to compress each frame into a small set of tokens, reducing the complexity of raw pixel inputs.
• Transformer World Model: A GPT-like Transformer is trained to predict future tokens and rewards autoregressively, conditioned on past observations and actions.
• Imagination-based Policy Learning: An actor-critic policy is trained entirely within the latent space of the world model, using imagined rollouts, which avoids costly environment interactions.
• Sample Efficiency: The method achieves strong performance on Atari 100k, a benchmark designed to test sample efficiency, with only 100k steps (equivalent to 2 hours of gameplay).
• Open-source Release: Code and pretrained models are made available, promoting further research.

Results

On the Atari 100k benchmark, IRIS achieves a mean human-normalized score of 1.046, surpassing prior model-based methods such as SimPLe (0.443) and Dreamer (0.823). It also outperforms some model-free methods like Efficient Rainbow (0.973). Notably, IRIS reaches human-level performance on several games, including Pong and Breakout. The results demonstrate that Transformers can serve as effective world models, achieving high sample efficiency without task-specific inductive biases. Ablations show the importance of discrete tokenization and the Transformer's capacity for long-term predictions.

Significance

IRIS bridges two major AI trends: Transformers and model-based RL. It provides strong evidence that sequence models can learn compact, predictive representations of complex environments from pixels, enabling agents to learn efficiently. This could lead to more general and scalable RL systems, particularly in domains where data collection is expensive, such as robotics and healthcare. Furthermore, the success of IRIS suggests that the same architectures driving advances in language and vision can be leveraged for decision-making, potentially unifying representation learning across modalities. Future work may extend this approach to continuous control, multi-agent settings, and real-world applications, further validating the Transformer as a universal world model.