Multi-agent reinforcement learning: A selective overview of theories and algorithms

Analysis

Why This Paper Matters

Multi-agent reinforcement learning (MARL) is a rapidly growing area that extends RL to settings where multiple agents interact, making it crucial for applications like autonomous driving, robotics, and economics. This paper, with over 1700 citations, has become a key reference for both newcomers and experts. It provides a structured overview that helps demystify the complex landscape of MARL, which is often fragmented across different problem formulations and solution concepts.

The paper's significance lies in its clear categorization of MARL problems into cooperative, competitive, and mixed settings. This taxonomy is not just a pedagogical tool but also guides algorithm selection and theoretical analysis. By highlighting the theoretical foundations—such as Markov games and Nash equilibria—the authors bridge the gap between classical game theory and modern RL, offering a coherent framework that has influenced subsequent research.

Technical Contributions

The paper makes several key technical contributions:

• Unified Problem Formulation: It formalizes MARL as a stochastic game and discusses various objectives (e.g., Nash equilibrium, correlated equilibrium) depending on the setting.
• Theoretical Insights: It reviews convergence guarantees for algorithms in specific settings, such as Q-learning in zero-sum games and policy gradient methods in cooperative settings.
• Algorithmic Taxonomy: It categorizes algorithms into independent learners, joint-action learners, and centralized training with decentralized execution (CTDE), clarifying trade-offs.
• Mean-Field Approximation: It introduces mean-field games as a scalable approach for large-agent systems, which has become a popular research direction.
• Open Challenges: It identifies non-stationarity, scalability, and credit assignment as core challenges, setting a research agenda.

Results

As a survey, the paper does not present new experimental metrics. Instead, it synthesizes existing theoretical results, noting that convergence guarantees are often limited to two-player zero-sum games or mean-field limits. For example, it discusses how Q-learning converges in zero-sum stochastic games but not in general-sum games. The paper also highlights that policy gradient methods can achieve Nash equilibria in certain cooperative settings. These insights have guided the community toward more tractable problem formulations.

Significance

The broader impact of this paper is substantial. It has become a standard citation for MARL papers, providing a common language and framework. Its taxonomy has been widely adopted, and its discussion of open problems has inspired research on deep MARL, communication learning, and hierarchical methods. By clarifying the theoretical underpinnings, it has also encouraged more rigorous analysis in the field. As MARL continues to grow, this overview remains a vital entry point and a reference for understanding the field's evolution.