quantizing-models-bitsandbytes
Category: ml-training · License: MIT · Version: 1.0.0
Tags: Optimization, Bitsandbytes, Quantization, 8-Bit, 4-Bit, Memory Optimization, QLoRA, NF4, INT8, HuggingFace, Efficient Inference
When to use it
Quantizes LLMs to 8-bit or 4-bit for 50-75% memory reduction with minimal accuracy loss. Use when GPU memory is limited, need to fit larger models, or want faster inference. Supports INT8, NF4, FP4 formats, QLoRA training, and 8-bit optimizers. Works with HuggingFace Transformers.Example requests
Ask your agent in plain language, for example:- “how do i load llama 2 13b in 4-bit on my 12GB 3060?”
- “I’m fine-tuning Mistral-7B on my instruction dataset
train.jsonlwith a single 24GB RTX 4090. Full-precision LoRA hitstorch.cuda.OutOfMemoryErrorat step 0 with batch size 8. Can you set it up so it fits? I’d like to use the HF Trainer and keep the loss comparable to what I got on the A100.” - “My lab only has a couple of older 16GB GPUs, and I need to run a 7B language model for inference and also cut the optimizer state memory while training a small adapter. What’s the least invasive way to shrink memory without changing my HuggingFace code much?”
Install
/plugin marketplace add KalarisLabs/research-agent-skills and install the whole ml-training category with /plugin install ml-training@research-agent-skills.
Related
awq-quantizationblip-2-vision-languagegptqhqq-quantizationmodel-pruning- Skills for machine learning research