fine-tuning-with-trl
Category: ml-training · License: MIT · Version: 1.0.0
Tags: Post-Training, TRL, Reinforcement Learning, Fine-Tuning, SFT, DPO, PPO, GRPO, RLHF, Preference Alignment, HuggingFace
When to use it
Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimization, and reward model training. Use when need RLHF, align model with preferences, or train from human feedback. Works with HuggingFace Transformers.Example requests
Ask your agent in plain language, for example:- “How do I run DPO on my preference pairs with TRL?”
- “I’m fine-tuning Qwen2.5-7B on an A100 40GB with my file pairs.jsonl (fields: question, good_answer, bad_answer). DPOTrainer crashes with ‘CUDA out of memory’ about 20 steps in. The reference model is eating memory I guess. What should I change?”
- “I have a base 8B model and thousands of human-ranked response pairs from annotators. I want a model that follows instructions and matches those human rankings, and I’d like a reward model along the way so I can score generations. Walk me through the stages and how to check it worked.”
Install
/plugin marketplace add KalarisLabs/research-agent-skills and install the whole ml-training category with /plugin install ml-training@research-agent-skills.
Related
constitutional-aigrpo-rl-trainingsimpo-trainingverl-rl-trainingllavaopenrlhf-training- Skills for machine learning research