A Generalist Agent

A Generalist Agent, referred to as Gato, is a multi-modal, multi-task, multi-embodiment generalist policy developed by Google DeepMind. Inspired by progress in large-scale language modelling, Gato applies a similar approach beyond text outputs. The same neural network with the same weights can perform diverse tasks such as playing Atari games, captioning images, engaging in interactive dialogue, and controlling a real robot arm to stack blocks. During training, data from different tasks and modalities are serialized into a flat sequence of tokens and processed by a transformer neural network. When deployed, Gato samples actions autoregressively based on a context window of 1024 tokens. It is trained on numerous datasets including agent experience in simulated and real-world environments, as well as natural language and image datasets. Gato represents a step towards building a single generalist agent capable of handling a wide range of tasks with a single set of weights.