Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding

Analysis

Why This Paper Matters

This paper is a landmark response to a critical challenge in reinforcement learning: the difficulty of combining function approximation with dynamic programming. In 1994, Boyan and Moore reported negative results on simple control tasks, suggesting that function approximation might be fundamentally incompatible with RL. Sutton's paper directly refutes this by showing that with the right function approximator (CMAC) and learning paradigm (online), all those tasks can be solved. This shifted the focus from whether function approximation can work to how it should be designed.

The paper's emphasis on sparse coarse coding (CMACs) versus global approximators (like neural networks with sigmoidal units) was prescient. It highlighted that the locality and resolution of the representation are crucial for stability and convergence in RL. This insight remains relevant today, as modern RL often uses neural networks with distributed representations, but still struggles with catastrophic interference and non-stationarity.

Technical Contributions

• Sparse Coarse Coding (CMAC): The paper advocates for using CMACs, which are local, sparse, and coarse function approximators. This contrasts with global approximators that use all parameters to represent any input, leading to interference.
• Online Learning: The paper emphasizes learning online (updating after each transition) rather than offline (batch) learning. Online learning allows the agent to adapt quickly and avoid issues with non-stationary targets.
• Empirical Demonstration: The paper provides a systematic comparison across multiple control tasks, including a larger one, showing consistent success.
• Identification of Key Factors: It isolates the choice of function approximator and learning mode as critical factors, rather than blaming the RL algorithm itself.

Results

The paper reports successful learning on all tasks that Boyan and Moore attempted, including the cart-pole balancing and other continuous control benchmarks. It also demonstrates success on a significantly larger task, indicating scalability. The key comparison is that Boyan and Moore's methods (using global approximators and offline learning) failed, while Sutton's CMAC-based online approach succeeded. The paper does not provide quantitative metrics like learning curves or success rates, but the qualitative outcome is clear: all tasks were solved.

Significance

This paper had a lasting impact on the RL community. It reinforced the importance of representation and learning paradigm in RL, and it influenced later work on tile coding and other sparse representations. It also contributed to the understanding that online learning is often more stable for RL than offline batch learning, a principle that still guides algorithm design. While modern RL often uses deep neural networks, the lessons about interference and the need for careful representation design remain relevant. The paper is a classic example of how empirical negative results can be overturned by thoughtful algorithmic choices.