PEFT Fine-Tuning with LoRA and QLoRA for Large Language Models

{ "TITLE": "PEFT and LoRA Fine-Tuning Reference", "BODY": "## Overview\n\nPEFT (Parameter-Efficient Fine-Tuning) enables fine-tuning large language models by training less than 1% of parameters. It uses LoRA, QLoRA, and 25+ adapter methods to reduce memory and storage requirements dramatically. This approach is ideal when you need to fine-tune 7B-70B models on consumer GPUs (RTX 4090, A100) and want to train less than 1% parameters, resulting in 6MB adapters versus 14GB full models. Use PEFT/LoRA for fast iteration with multiple task-specific adapters and when deploying multiple fine-tuned variants from one base model. QLoRA (PEFT + quantization) is the choice when fine-tuning 70B models on a single 24GB GPU, when memory is the primary constraint, and when you can accept approximately a 5% quality trade-off versus full fine-tuning. Full fine-tuning is recommended for training small models under 1B parameters, when maximum quality is needed and you have the compute budget, or when significant domain shift requires updating all weights.\n\n## Capabilities\n\nPEFT supports LoRA, QLoRA, AdaLoRA, IA3, Prefix Tuning, Prompt Tuning, and P-Tuning v2. It enables fine-tuning on consumer GPUs with a low memory footprint, allows saving and loading small adapter files (e.g., 6MB vs 16GB), supports multi-adapter serving with runtime switching, and can merge adapters into the base model for deployment. It integrates with TRL (SFTTrainer), Axolotl (YAML config), and vLLM (inference). It supports 4-bit quantization via bitsandbytes for QLoRA and provides automatic detection of all linear layers with target_modules='all-linear' (PEFT 0.6.0+).\n\n## Prerequisites\n\nInstall the required libraries. For basic usage:\n\npython\n# Basic installation\npip install peft\n\n# With quantization support (recommended)\npip install peft bitsandbytes\n\n# Full stack\npip install peft transformers accelerate bitsandbytes datasets\n\n\nFor quantization support, install bitsandbytes. For vLLM integration, install the vllm library. For TRL integration, install the trl library. You need access to the base model (e.g., from Hugging Face Hub) and sufficient GPU memory (see benchmarks).\n\n## Fine-Tuning Procedures\n\n### LoRA Fine-Tuning (Standard)\n\nThis procedure fine-tunes a model using LoRA adapters. The following example uses Llama 3.1 8B on the databricks-dolly-15k dataset.\n\npython\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments, Trainer\nfrom peft import get_peft_model, LoraConfig, TaskType\nfrom datasets import load_dataset\n\n# Load base model\nmodel_name = \"meta-llama/Llama-3.1-8B\"\nmodel = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=\"auto\", device_map=\"auto\")\ntokenizer = AutoTokenizer.from_pretrained(model_name)\ntokenizer.pad_token = tokenizer.eos_token\n\n# LoRA configuration\nlora_config = LoraConfig(\n task_type=TaskType.CAUSAL_LM,\n r=16, # Rank (8-64, higher = more capacity)\n lora_alpha=32, # Scaling factor (typically 2*r)\n lora_dropout=0.05, # Dropout for regularization\n target_modules=[\"q_proj\", \"v_proj\", \"k_proj\", \"o_proj\"], # Attention layers\n bias=\"none\" # Don't train biases\n)\n\n# Apply LoRA\nmodel = get_peft_model(model, lora_config)\nmodel.print_trainable_parameters()\n# Output: trainable params: 13,631,488 || all params: 8,043,307,008 || trainable%: 0.17%\n\n# Prepare dataset\ndataset = load_dataset(\"databricks/databricks-dolly-15k\", split=\"train\")\n\ndef tokenize(example):\n text = f\"### Instruction:\\n{example['instruction']}\\n\\n### Response:\\n{example['response']}\"\n return tokenizer(text, truncation=True, max_length=512, padding=\"max_length\")\n\ntokenized = dataset.map(tokenize, remove_columns=dataset.column_names)\n\n# Training\ntraining_args = TrainingArguments(\n output_dir=\"./lora-llama\",\n num_train_epochs=3,\n per_device_train_batch_size=4,\n gradient_accumulation_steps=4,\n learning_rate=2e-4,\n fp16=True,\n logging_steps=10,\n save_strategy=\"epoch\"\n)\n\ntrainer = Trainer(\n model=model,\n args=training_args,\n train_dataset=tokenized,\n data_collator=lambda data: {\"input_ids\": torch.stack([f[\"input_ids\"] for f in data]),\n \"attention_mask\": torch.stack([f[\"attention_mask\"] for f in data]),\n \"labels\": torch.stack([f[\"input_ids\"] for f in data])}\n)\n\ntrainer.train()\n\n# Save adapter only (6MB vs 16GB)\nmodel.save_pretrained(\"./lora-llama-adapter\")\n\n\nThe output shows that only 13,631,488 parameters are trainable out of 8,043,307,008 total, which is 0.17% of all parameters. This is a 100% reduction in trainable parameters compared to full fine-tuning.\n\n### QLoRA Fine-Tuning (Memory-Efficient)\n\nQLoRA combines LoRA with 4-bit quantization to fit very large models on limited hardware. This example fine-tunes Llama 3.1 70B on a single 24GB GPU.\n\npython\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nfrom peft import get_peft_model, LoraConfig, prepare_model_for_kbit_training\n\n# 4-bit quantization config\nbnb_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_quant_type=\"nf4\", # NormalFloat4 (best for LLMs)\n bnb_4bit_compute_dtype=\"bfloat16\", # Compute in bf16\n bnb_4bit_use_double_quant=True # Nested quantization\n)\n\n# Load quantized model\nmodel = AutoModelForCausalLM.from_pretrained(\n \"meta-llama/Llama-3.1-70B\",\n quantization_config=bnb_config,\n device_map=\"auto\"\n)\n\n# Prepare for training (enables gradient checkpointing)\nmodel = prepare_model_for_kbit_training(model)\n\n# LoRA config for QLoRA\nlora_config = LoraConfig(\n r=64, # Higher rank for 70B\n lora_alpha=128,\n lora_dropout=0.1,\n target_modules=[\"q_proj\", \"v_proj\", \"k_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\"],\n bias=\"none\",\n task_type=\"CAUSAL_LM\"\n)\n\nmodel = get_peft_model(model, lora_config)\n# 70B model now fits on single 24GB GPU!\n\n\nThe QLoRA approach has approximately a 5% quality trade-off versus full fine-tuning.\n\n### IA3 Fine-Tuning\n\nIA3 trains only 0.01% of parameters:\n\npython\nfrom peft import IA3Config\n\nia3_config = IA3Config(\n target_modules=[\"q_proj\", \"v_proj\", \"k_proj\", \"down_proj\"],\n feedforward_modules=[\"down_proj\"]\n)\nmodel = get_peft_model(model, ia3_config)\n# Trains only 0.01% of parameters!\n\n\n### Prefix Tuning\n\nPrefix Tuning uses virtual tokens prepended to input:\n\npython\nfrom peft import PrefixTuningConfig\n\nprefix_config = PrefixTuningConfig(\n task_type=\"CAUSAL_LM\",\n num_virtual_tokens=20, # Prepended tokens\n prefix_projection=True # Use MLP projection\n)\nmodel = get_peft_model(model, prefix_config)\n\n\n## Configuration Parameters\n\n### LoRA Parameters\n\n#### r (rank)\nControls capacity versus efficiency. Higher rank means more trainable parameters, more memory, and higher quality. Rank selection: r=4 minimal quality, r=8 recommended start, r=16 general, r=32 complex, r=64 domain adaptation or 70B models.\n\n#### lora_alpha\nScaling factor for LoRA updates. The rule of thumb is alpha = 2 * rank. Conservative: alpha=rank. Aggressive: alpha=4*rank.\n\npython\n# Rule of thumb: alpha = 2 * rank\nLoraConfig(r=16, lora_alpha=32) # Standard\nLoraConfig(r=16, lora_alpha=16) # Conservative (lower learning rate effect)\nLoraConfig(r=16, lora_alpha=64) # Aggressive (higher learning rate effect)\n\n\n#### lora_dropout\nDropout rate for regularization.\n\n#### target_modules\nList of module names to apply LoRA to. Can be 'all-linear' for all linear layers (PEFT 0.6.0+). Target modules vary by architecture:\n\npython\n# Llama / Mistral / Qwen\ntarget_modules = [\"q_proj\", \"v_proj\", \"k_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\"]\n\n# GPT-2 / GPT-Neo\ntarget_modules = [\"c_attn\", \"c_proj\", \"c_fc\"]\n\n# Falcon\ntarget_modules = [\"query_key_value\", \"dense\", \"dense_h_to_4h\", \"dense_4h_to_h\"]\n\n# BLOOM\ntarget_modules = [\"query_key_value\", \"dense\", \"dense_h_to_4h\", \"dense_4h_to_h\"]\n\n# Auto-detect all linear layers\ntarget_modules = \"all-linear\" # PEFT 0.6.0+\n\n\n#### bias\nWhether to train bias parameters. Typically 'none'.\n\n#### task_type\nType of task, e.g., 'CAUSAL_LM'.\n\n### QLoRA-specific parameters\n- load_in_4bit: Enable 4-bit quantization.\n- bnb_4bit_quant_type: Quantization type, e.g., 'nf4' (NormalFloat4, best for LLMs).\n- bnb_4bit_compute_dtype: Compute dtype during quantization, e.g., 'bfloat16'.\n- bnb_4bit_use_double_quant: Enable nested quantization for additional memory savings.\n\n### Prefix Tuning parameters\n- num_virtual_tokens: Number of prepended tokens.\n- prefix_projection: Whether to use MLP projection.\n\n### IA3 parameters\n- feedforward_modules: List of feedforward module names.\n\n## Loading and Merging Adapters\n\nAfter training, load the adapter using either method:\n\npython\nfrom peft import PeftModel, AutoPeftModelForCausalLM\nfrom transformers import AutoModelForCausalLM\n\n# Option 1: Load with PeftModel\nbase_model = AutoModelForCausalLM.from_pretrained(\"meta-llama/Llama-3.1-8B\")\nmodel = PeftModel.from_pretrained(base_model, \"./lora-llama-adapter\")\n\n# Option 2: Load directly (recommended)\nmodel = AutoPeftModelForCausalLM.from_pretrained(\n \"./lora-llama-adapter\",\n device_map=\"auto\"\n)\n\n\nTo merge the adapter into the base model for deployment:\n\npython\n# Merge for deployment (no adapter overhead)\nmerged_model = model.merge_and_unload()\n\n# Save merged model\nmerged_model.save_pretrained(\"./llama-merged\")\ntokenizer.save_pretrained(\"./llama-merged\")\n\n# Push to Hub\nmerged_model.push_to_hub(\"username/llama-finetuned\")\n\n\n## Multi-Adapter Serving\n\nLoad a base model with multiple adapters and switch between them at runtime:\n\npython\nfrom peft import PeftModel\n\n# Load base with first adapter\nmodel = AutoPeftModelForCausalLM.from_pretrained(\"./adapter-task1\")\n\n# Load additional adapters\nmodel.load_adapter(\"./adapter-task2\", adapter_name=\"task2\")\nmodel.load_adapter(\"./adapter-task3\", adapter_name=\"task3\")\n\n# Switch between adapters at runtime\nmodel.set_adapter(\"task1\") # Use task1 adapter\noutput1 = model.generate(**inputs)\n\nmodel.set_adapter(\"task2\") # Switch to task2\noutput2 = model.generate(**inputs)\n\n# Disable adapters (use base model)\nwith model.disable_adapter():\n base_output = model.generate(**inputs)\n\n\n## Integration with Other Tools\n\n### Integration with TRL (SFTTrainer)\n\nPass the LoRA config directly to SFTTrainer:\n\npython\nfrom trl import SFTTrainer, SFTConfig\nfrom peft import LoraConfig\n\nlora_config = LoraConfig(r=16, lora_alpha=32, target_modules=\"all-linear\")\n\ntrainer = SFTTrainer(\n model=model,\n args=SFTConfig(output_dir=\"./output\", max_seq_length=512),\n train_dataset=dataset,\n peft_config=lora_config, # Pass LoRA config directly\n)\ntrainer.train()\n\n\n### Integration with Axolotl (YAML Config)\n\nCreate a YAML configuration file:\n\nyaml\n# axolotl config.yaml\nadapter: lora\nlora_r: 16\nlora_alpha: 32\nlora_dropout: 0.05\nlora_target_modules:\n - q_proj\n - v_proj\n - k_proj\n - o_proj\nlora_target_linear: true # Target all linear layers\n\n\n### Integration with vLLM (Inference)\n\nServe models with LoRA adapters using vLLM:\n\npython\nfrom vllm import LLM\nfrom vllm.lora.request import LoRARequest\n\n# Load base model with LoRA support\nllm = LLM(model=\"meta-llama/Llama-3.1-8B\", enable_lora=True)\n\n# Serve with adapter\noutputs = llm.generate(\n prompts,\n lora_request=LoRARequest(\"adapter1\", 1, \"./lora-adapter\")\n)\n\n\n## Performance Benchmarks\n\n- LoRA r=16 uses 18 GB GPU memory for Llama 3.1 8B; QLoRA r=16 uses 6 GB; IA3 uses 16 GB. Full FT uses 60+ GB.\n- Training speed on A100 80GB: LoRA achieves 3200 tok/sec (1.3x full FT), QLoRA achieves 2100 tok/sec (0.84x full FT).\n- Quality (MMLU): LoRA is slightly lower than full FT; QLoRA is slightly lower than LoRA.\n\n## Failure Modes and Solutions\n\n### CUDA OOM during training\n\npython\n# Solution 1: Enable gradient checkpointing\nmodel.gradient_checkpointing_enable()\n\n# Solution 2: Reduce batch size + increase accumulation\nTrainingArguments(\n per_device_train_batch_size=1,\n gradient_accumulation_steps=16\n)\n\n# Solution 3: Use QLoRA\nfrom transformers import BitsAndBytesConfig\nbnb_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type=\"nf4\")\n\n\n### Adapter not applying\n\npython\n# Verify adapter is active\nprint(model.active_adapters) # Should show adapter name\n\n# Check trainable parameters\nmodel.print_trainable_parameters()\n\n# Ensure model in training mode\nmodel.train()\n\n\n### Quality degradation\n\npython\n# Increase rank\nLoraConfig(r=32, lora_alpha=64)\n\n# Target more modules\ntarget_modules = \"all-linear\"\n\n# Use more training data and epochs\nTrainingArguments(num_train_epochs=5)\n\n# Lower learning rate\nTrainingArguments(learning_rate=1e-4)\n\n\n## Constraints and Caveats\n\n- QLoRA has approximately a 5% quality trade-off versus full fine-tuning.\n- Full fine-tuning is recommended for models under 1B parameters or when maximum quality is needed.\n- Full fine-tuning is needed for significant domain shift.\n- LoRA rank selection: r=4 minimal quality, r=8 recommended start, r=16 general, r=32 complex, r=64 domain adaptation or 70B models.\n- Alpha rule of thumb: alpha = 2 * rank; conservative alpha=rank, aggressive alpha=4*rank.\n- Target modules vary by architecture (see target_modules section).\n- IA3 trains only 0.01% of parameters.\n- Prefix Tuning uses virtual tokens prepended to input.\n- Multi-adapter serving allows runtime switching between adapters and disabling adapters for base model inference.\n- Merging adapter into base model removes adapter overhead for deployment.\n- The adapter file size is typically 6MB for a 16GB model, representing a reduction of over 99.9% in storage.\n- When using QLoRA, the 4-bit quantization reduces memory usage by approximately 75% compared to 16-bit precision, but with a 0.1% quality loss on some benchmarks.\n- For very large models (70B+), the training time with QLoRA can be up to 2500 tokens per second on an A100, which is slower than full fine-tuning but enables training on limited hardware." }