OpenRLHF

Free

An Easy-to-use, Scalable and High-performance RLHF Framework (70B+ PPO Full Tuning & Iterative DPO & LoRA & RingAttention & RFT).

FreeFree tier

Inputs: text, imageOutputs: text

Type

Open Source

Visit Website GitHub (38 stars)

About OpenRLHF

OpenRLHF is an open-source, production-ready framework for reinforcement learning from human feedback (RLHF) that combines Ray and vLLM distributed architecture with a unified agent-based design paradigm. It supports a wide range of RL algorithms including PPO, REINFORCE++, GRPO, RLOO, and DAPO, and scales to hundreds of billions of parameters. The framework offers a complete RLHF pipeline from supervised fine-tuning (SFT) and reward model training to full RL training, with support for LoRA, RingAttention, single-turn and multi-turn agents, custom reward functions, and vision-language model (VLM) RLHF. It is designed to be easy-to-use, scalable, and high-performance for both research and production deployment.

Key Features

Combines Ray + vLLM + DeepSpeed distributed infrastructure for scalable RL training

Supports multiple RL algorithms: PPO, REINFORCE++, GRPO, RLOO, DAPO

Unified agent-based execution pipeline for single-turn and multi-turn agents

Includes complete RLHF pipeline: SFT, reward model training, and RL training

Supports LoRA and RingAttention for efficient fine-tuning

Vision-Language Model (VLM) RLHF support with image inputs

Async RLHF training via train.async_enable flag

Custom reward functions and complex environment integration

Scalable to hundreds of billions of parameters with DeepSpeed backend

Production-ready with vLLM for high-throughput inference

Pros & Cons

Pros

Scalable and production-ready architecture combining Ray, vLLM, and DeepSpeed
Supports a wide range of state-of-the-art RL algorithms out of the box
Easy-to-use agent-based design paradigm simplifies RL pipeline setup
Actively maintained with frequent updates and new features (VLM RLHF, async RL)
Comprehensive documentation and examples for quick start
Open-source with permissive license, enabling customization and community contributions
Backed by real-world adoption (e.g., used in CMU course, various research papers)

Cons

Requires significant computational resources for full-scale training (e.g., 70B+ models)
Designed primarily for RLHF; may be overkill for simpler fine-tuning tasks
Relatively new framework; documentation for advanced features may still be evolving
Dependency on Ray, vLLM, and DeepSpeed adds complexity to environment setup

Best For

Training large language models with reinforcement learning from human feedbackDeveloping reasoning models (e.g., ProRL V2, Logic-RL, PRIME) using REINFORCE++Multi-agent system training with RL (MARTI fork)Vision-language model fine-tuning with RLHF including image inputsScaling RL training to 70B+ parameters with full PPO tuningAcademic and industry research on RLHF algorithms and agentic RL