Group Sequence Policy Optimization
FreeStable and efficient RL for LLMs via sequence-level optimization
FreeFree tier
About Group Sequence Policy Optimization
Group Sequence Policy Optimization (GSPO) is a reinforcement learning algorithm designed for training large language models. It uses sequence-level importance ratios and clipping, rewarding, and optimization, offering stability and efficiency advantages over token-level methods. GSPO has been shown to outperform GRPO, stabilize Mixture-of-Experts (MoE) RL training, and simplify RL infrastructure design. It contributed to the improvements in Qwen3 models.
Key Features
Sequence-level importance ratio instead of token-level
Sequence-level clipping, rewarding, and optimization
Superior training efficiency and performance over GRPO
Stabilizes Mixture-of-Experts (MoE) RL training
Simplifies design of RL infrastructure
Contributed to improvements in Qwen3 models
Pros & Cons
Pros
- More stable training compared to token-level methods
- Higher training efficiency and performance than GRPO
- Simplifies RL infrastructure design
- Effective for MoE architectures
Cons
- May require sequence-level computation which could be memory intensive
- Limited to RL training scenarios
- No detailed analysis of failure cases provided
Best For
Reinforcement learning for large language modelsTraining Mixture-of-Experts models with RLImproving reasoning capabilities of LLMs
FAQ
What is Group Sequence Policy Optimization?
GSPO is a stable, efficient, and performant reinforcement learning algorithm for training large language models, using sequence-level importance ratios and optimization.