Geometric-Mean Policy Optimization

Analysis

Why This Paper Matters

Group Relative Policy Optimization (GRPO) has become a cornerstone for enhancing reasoning in large language models, but its reliance on arithmetic mean of token rewards makes it vulnerable to outlier importance-weighted rewards. These outliers cause unstable policy updates, hindering performance. This paper identifies a critical flaw in a widely used method and offers an elegant fix: replacing the arithmetic mean with the geometric mean. The geometric mean is inherently less sensitive to extreme values, leading to more stable training dynamics. This is particularly important for mathematical reasoning tasks where token-level reward signals can vary dramatically.

The work is timely as the AI community increasingly focuses on reasoning capabilities of LLMs. By providing a theoretically grounded and empirically validated improvement, GMPO addresses a practical pain point for practitioners fine-tuning models with reinforcement learning. The plug-and-play nature means it can be adopted with minimal code changes, lowering the barrier for adoption.

Technical Contributions

• Geometric Mean Reward: GMPO replaces the arithmetic mean of token-level rewards in GRPO with the geometric mean, which is less sensitive to outliers.
• Stable Importance Sampling: The geometric mean maintains a more stable range of importance sampling ratios, preventing extreme updates.
• Theoretical Framework: The authors prove that both GMPO and GRPO are forms of weighted policy gradient, but GMPO's weights are inherently more stable, leading to better optimization.
• Plug-and-Play Design: GMPO requires only a single line change in the GRPO loss function, making it easy to integrate into existing pipelines.

Results

• GMPO-7B achieves up to 4.1% improvement in average Pass@1 over GRPO on multiple mathematical reasoning benchmarks.
• Outperforms several state-of-the-art approaches, demonstrating the effectiveness of the geometric mean approach.
• The improvement is consistent across different tasks, indicating robustness.

Significance

GMPO provides a simple yet powerful enhancement to GRPO that improves training stability and reasoning performance. Its theoretical grounding and empirical success suggest it could become a standard component in LLM fine-tuning pipelines. The approach is not limited to mathematical reasoning and could be applied to other domains where token-level rewards are used. By addressing a fundamental instability in policy optimization, GMPO contributes to more reliable and effective reinforcement learning for language models.