Journal Article
Reinforcement Learning

Deep Reinforcement Learning: A Survey

Xu Wang(Xidian University), Sen Wang(Xidian University), Xingxing Liang(National University of Defense Technology), Dawei Zhao(National University of Defense Technology), Jincai Huang(National University of Defense Technology), Xin Xu(National University of Defense Technology), Bin Dai(National University of Defense Technology), Qiguang Miao(Xidian University)
September 28, 2022IEEE Transactions on Neural Networks and Learning Systems809 citations
DOI

809

Citations

0

Influential Citations

IEEE Transactions on Neural Networks and Learning Systems

Venue

2022

Year

Abstract

Deep reinforcement learning (DRL) integrates the feature representation ability of deep learning with the decision-making ability of reinforcement learning so that it can achieve powerful end-to-end learning control capabilities. In the past decade, DRL has made substantial advances in many tasks that require perceiving high-dimensional input and making optimal or near-optimal decisions. However, there are still many challenging problems in the theory and applications of DRL, especially in learning control tasks with limited samples, sparse rewards, and multiple agents. Researchers have proposed various solutions and new theories to solve these problems and promote the development of DRL. In addition, deep learning has stimulated the further development of many subfields of reinforcement learning, such as hierarchical reinforcement learning (HRL), multiagent reinforcement learning, and imitation learning. This article gives a comprehensive overview of the fundamental theories, key algorithms, and primary research domains of DRL. In addition to value-based and policy-based DRL algorithms, the advances in maximum entropy-based DRL are summarized. The future research topics of DRL are also analyzed and discussed.

Analysis

Why This Paper Matters

Deep reinforcement learning has become a cornerstone of modern AI, enabling breakthroughs in areas like game playing, robotics, and autonomous driving. This survey, published in a top-tier journal (IEEE TNNLS) with over 800 citations, provides a timely and comprehensive synthesis of the field's rapid evolution. It is particularly valuable for practitioners seeking to understand the landscape of DRL algorithms and identify promising research directions.

The paper's significance lies in its breadth and structure. It not only covers classic value-based (e.g., DQN) and policy-based (e.g., PPO) methods but also dedicates attention to maximum entropy-based approaches like SAC, which have become state-of-the-art in continuous control. By also addressing subfields such as hierarchical RL, multiagent RL, and imitation learning, the survey offers a holistic view that is rare in the literature.

Technical Contributions

The survey makes several key technical contributions:

  • Taxonomy of DRL algorithms: Clearly categorizes methods into value-based, policy-based, and maximum entropy-based, explaining their theoretical foundations and trade-offs.
  • Coverage of advanced topics: Includes hierarchical RL (e.g., options framework), multiagent RL (e.g., MADDPG), and imitation learning (e.g., GAIL), showing how deep learning extends classical RL.
  • Discussion of challenges: Identifies critical open problems such as sample efficiency, sparse rewards, and multi-agent coordination, which are active research areas.
  • Future directions: Provides a forward-looking analysis of topics like meta-learning, transfer learning, and safe RL.

Results

As a survey, the paper does not present new experimental results. However, it synthesizes findings from numerous prior works, noting that DRL has achieved superhuman performance in games (e.g., Atari, Go) and complex control tasks. The survey emphasizes that while value-based methods excel in discrete action spaces, policy gradient methods are preferred for continuous control, and maximum entropy methods offer improved exploration. The paper also highlights that sample efficiency remains a major bottleneck, with many algorithms requiring millions of interactions.

Significance

This survey has broad impact on the AI field by providing a structured reference that helps researchers and practitioners navigate the rapidly growing DRL literature. Its identification of key challenges and future directions can guide funding and research priorities. By covering both foundational and emerging topics, it serves as a useful educational resource for newcomers and a concise update for experts. The paper's high citation count reflects its utility as a go-to overview in the DRL community.

deep-reinforcement-learningsurveyreinforcement-learningdeep-learninghierarchical-reinforcement-learningmultiagent-reinforcement-learningimitation-learning
Reinforcement Learning

Related Papers