Reinforcement learning in robotics: A survey

Analysis

Why This Paper Matters

This survey is a landmark in the field of robot reinforcement learning, serving as a bridge between two communities that often speak different languages. Published in 2013, it arrived at a time when RL was gaining traction in robotics but lacked a unified framework for understanding the diverse approaches. The paper's importance lies in its systematic organization of the field, making it accessible to both roboticists and RL researchers. It emphasizes the mutual benefits: RL provides tools for complex behavior generation, while robotics offers real-world challenges that drive RL innovation. The analogy to physics and mathematics underscores the potential for deep synergy.

The paper's timing is also crucial. It captures the state of the art before the deep learning revolution, yet its insights remain relevant today. Many of the challenges it identifies—sample efficiency, safety, and the choice between model-based and model-free methods—are still central to modern RL research. By framing these challenges clearly, the survey has influenced how subsequent research is structured and communicated.

Technical Contributions

The paper's primary contribution is its taxonomy of RL methods for robotics. It distinguishes between:

• Model-based vs. model-free: Model-based methods learn a dynamics model and plan, while model-free methods learn policies directly from interaction.
• Value-function-based vs. policy-search: Value-function methods estimate the expected return of states/actions, while policy-search methods directly optimize the policy parameters.

The survey also discusses the role of prior knowledge, such as demonstrations or domain knowledge, in reducing the complexity of RL. It highlights the importance of representations, such as basis functions or features, in making learning tractable. The paper provides a detailed case study of a simple problem (e.g., pendulum swing-up) to illustrate how different methods can be applied and compared.

Results

As a survey, the paper does not present new experimental results. Instead, it synthesizes findings from numerous prior studies, noting notable successes in areas like robotic manipulation, locomotion, and control. It discusses how certain methods have been successfully applied in real-world scenarios, such as helicopter flight and robotic arm control. The paper does not provide quantitative metrics but rather qualitative assessments of when each approach is most suitable. For example, it notes that model-based methods can be more sample-efficient but are sensitive to model errors, while model-free methods are more robust but require more interaction.

Significance

The impact of this survey is profound. It has been cited over 3,800 times, indicating its role as a key reference for researchers entering the field. It has helped standardize terminology and frameworks, enabling clearer communication and comparison across studies. The survey also identified open questions that have driven subsequent research, such as how to incorporate prior knowledge effectively and how to scale RL to complex, high-dimensional tasks. Its influence extends beyond robotics, as many concepts discussed are applicable to general RL. The paper's emphasis on the interplay between algorithms, representations, and prior knowledge remains a guiding principle in modern AI research, especially in areas like deep RL and imitation learning.