openrlhf-training
Category: ml-training · License: MIT · Version: 1.0.0
Tags: Post-Training, OpenRLHF, RLHF, PPO, GRPO, RLOO, DPO, Ray, vLLM, Distributed Training, Large Models, ZeRO-3
When to use it
High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.Example requests
Ask your agent in plain language, for example:- “How do I run GRPO with OpenRLHF on 8 GPUs?”
- “I’m running PPO on a Llama-3 8B SFT checkpoint (./ckpt/llama3-8b-sft) with a reward model at ./ckpt/rm-8b, on 2 nodes of 8×A100 80GB with a Ray cluster. Using the hybrid engine I get CUDA OOM during the actor update, and DeepSpeed also throws ‘GPU index out of range’. How do I fix this?”
- “I want to align a 70B model with a learned reward signal, with rollouts generated by a fast inference engine and the training spread over a multi-node GPU cluster managed by a distributed scheduler. Which approach and setup would you recommend, and what hardware would I need?”
Install
/plugin marketplace add KalarisLabs/research-agent-skills and install the whole ml-training category with /plugin install ml-training@research-agent-skills.