Deep Reinforcement Learning with Double Q-Learning

Analysis

Why This Paper Matters

This paper addresses a critical flaw in deep reinforcement learning: the overestimation bias of Q-learning. While Q-learning's tendency to overestimate action values was known in tabular settings, its practical impact in deep RL was unclear. The authors show that DQN, a landmark algorithm, suffers from substantial overestimations in many Atari games, which can degrade performance. By adapting Double Q-learning to deep networks, they provide a simple, effective fix that became a standard component in subsequent RL algorithms.

The significance lies in both the diagnosis and the cure. The paper empirically demonstrates that overestimation is not just a theoretical curiosity but a practical problem that harms performance. The proposed Double DQN algorithm is straightforward to implement (requiring only a minor change to DQN) and yields consistent improvements. This work helped shift the field toward more stable and reliable value-based methods.

Technical Contributions

• Identification of overestimation in DQN: The authors measure the overestimation bias in DQN across multiple Atari games, showing it is substantial in games like Asterix, Breakout, and Seaquest.
• Generalization of Double Q-learning: They extend the tabular Double Q-learning idea to function approximation by using the online network for action selection and the target network for value evaluation, avoiding the need for two separate networks.
• Algorithmic simplicity: The modification to DQN is minimal—only the target value calculation changes from r + γ Q(s', argmax_a Q(s', a; θ); θ⁻) to r + γ Q(s', argmax_a Q(s', a; θ); θ⁻) where the selection uses the online network and evaluation uses the target network.
• Empirical validation: Extensive experiments on Atari 2600 games show reduced overestimation and improved scores, with particularly large gains on games where DQN overestimated most.

Results

• Double DQN achieves a 6x improvement over DQN on the game of Asterix (from ~600 to ~3600 points).
• On several games (e.g., Breakout, Seaquest), Double DQN reduces overestimation by 50% or more while improving final scores.
• The algorithm matches or exceeds DQN performance on all tested games, with no degradation.
• The paper reports that overestimation is correlated with poor performance: games where DQN overestimates most are those where Double DQN provides the largest gains.

Significance

Double DQN became a foundational component of the Rainbow DQN architecture and is widely used in deep RL research. It demonstrated that careful algorithmic design can fix known biases in deep learning systems, leading to more robust and sample-efficient learning. The paper also influenced later work on reducing bias in actor-critic methods (e.g., TD3). By providing a simple, principled improvement, it helped establish best practices for value-based RL that persist today.