implementing-llms-litgpt
Category: ml-training · License: MIT · Version: 1.0.0
Tags: Model Architecture, LitGPT, Lightning AI, LLM Implementation, LoRA, QLoRA, Fine-Tuning, Llama, Gemma, Phi, Mistral, Educational
When to use it
Implements and trains LLMs using Lightning AI’s LitGPT with 20+ pretrained architectures (Llama, Gemma, Phi, Qwen, Mistral). Use when need clean model implementations, educational understanding of architectures, or production fine-tuning with LoRA/QLoRA. Single-file implementations, no abstraction layers.Example requests
Ask your agent in plain language, for example:- “How do I LoRA fine-tune Phi-2 with litgpt on a 16GB GPU?”
- “I have 8k instruction/response pairs in data/lab_protocols.json in Alpaca format. I want to fine-tune Llama 3 1B using litgpt, but I get CUDA out of memory on my 12GB RTX 3060 when I run full finetuning. What should I change?”
- “I want to read the actual code of a Llama-style transformer, then train a small 1B model from scratch on my own tokenized domain corpus across 4 GPUs. I want clean single-file implementations, not a framework with many abstraction layers. What do you suggest, and how would I set it up?”
Install
/plugin marketplace add KalarisLabs/research-agent-skills and install the whole ml-training category with /plugin install ml-training@research-agent-skills.