Constrained Generative Policy Optimization (Mixture of Judges)

Analysis

Why This Paper Matters

Reinforcement Learning from Human Feedback (RLHF) has become the dominant paradigm for aligning large language models (LLMs) with human preferences. However, standard RLHF suffers from two critical issues: reward hacking, where models exploit proxy reward signals to achieve high scores without genuine alignment, and difficulty in balancing multiple, often conflicting objectives (e.g., helpfulness vs. harmlessness). This paper addresses both challenges head-on by introducing a Mixture of Judges (MoJ) framework that combines rule-based and LLM-based evaluators to explicitly constrain undesirable generation patterns.

The significance lies in moving beyond single-reward optimization toward a constrained optimization view of alignment. By treating alignment as a multi-objective problem with explicit constraints, the method provides a more principled way to balance competing desiderata. This is particularly relevant as LLMs are deployed in increasingly sensitive applications where safety, factuality, and style must be simultaneously controlled.

Technical Contributions

• Mixture of Judges (MoJ): Combines fast, deterministic rule-based judges (e.g., length, toxicity classifiers) with slower, more nuanced LLM-based judges (e.g., for helpfulness, coherence) to provide a rich set of constraints.
• Constrained Policy Optimization with Stratification: Formulates RLHF as a constrained optimization problem where the policy maximizes a calibrated reward subject to constraints from the judges. Stratification ensures that constraints are applied at appropriate granularity (e.g., per task or per example).
• Per-Task Optimization Strategies: In multi-task settings, the method avoids conflicting gradients by tailoring optimization strategies (e.g., different constraint thresholds or reward weights) for each task, improving the overall Pareto frontier.
• Cost Efficiency: By using rule-based judges for cheap, frequent checks and LLM judges for sparse, high-stakes evaluations, the approach maintains computational feasibility.

Results

The abstract reports that the method improves the Pareto frontier for multi-objective alignment, meaning it achieves better trade-offs between competing objectives (e.g., higher reward with lower constraint violations). It specifically addresses reward hacking and conflicting objectives that plague standard RLHF. However, the abstract does not provide concrete numerical results (e.g., win rates, BLEU scores, or human evaluation metrics), nor does it compare against baselines like PPO or DPO on standard benchmarks. This lack of quantitative evidence limits the ability to assess the magnitude of improvement.

Significance

This work contributes a structured framework for multi-objective LLM alignment that could influence how future RLHF systems are designed. By explicitly modeling constraints and using a mixture of judges, it offers a path toward more controllable and safer LLMs. The approach is particularly relevant for production systems where multiple quality dimensions must be balanced. However, without empirical validation on widely used benchmarks, the practical impact remains uncertain. Future work should provide clear metrics and comparisons to establish the method's effectiveness relative to existing approaches.