Fine-Tuning LLMs with LoRA: Step-by-Step Python Guide (2026)

Fine-tuning massive language models used to require enormous GPU memory and weeks of training time. LoRA changes that by updating only a tiny fraction of parameters while preserving the pretrained knowledge. This article walks you through the intuition, mechanics, and a hands‑on example of LoRA‑based fine‑tuning.

Why It Matters

Full fine‑tuning updates every parameter of the model, which is computationally and memory intensive.

When you work with models that have billions of parameters, storing gradients and optimizer states for each weight quickly exceeds the memory of a single GPU. This makes experimentation slow and expensive, especially for researchers or small teams without access to large compute clusters.

LoRA (Low‑Rank Adaptation) is a Parameter‑Efficient Fine‑Tuning (PEFT) technique.

By learning only a small set of adapter weights, LoRA lets you specialize a model for a new task while keeping the original pretrained weights frozen. The result is a dramatic reduction in the number of trainable parameters, often down to less than one percent of the total.

Core Concepts

LoRA freezes the original pretrained model weights and injects trainable low‑rank adapter matrices A and B.

During LoRA training only the adapter parameters are updated, reducing trainable parameters to ~0.1% of the total model size.

The low‑rank decomposition approximates the weight update ΔW = B·A where B∈ℝ^{d×r} and A∈ℝ^{r×d}. This formulation allows the adapter to capture meaningful changes in the weight matrix while staying compact.

Architecture/How It Works

LoRA adapters are inserted into the combined query‑key‑value projection (c_attn) of each transformer block's attention layer. By focusing on the attention projections, the method efficiently adapts the model’s ability to focus on relevant tokens without altering the feed‑forward networks.

The PEFT library provides a LoraConfig class that lets users specify LoRA hyper‑parameters in a single call.

Typical hyper‑parameters include a rank r (e.g., 8), a scaling factor lora_alpha (e.g., 16), a dropout rate, and the list of target modules. The configuration used in the example is: r=8, lora_alpha=16, target_modules=["c_attn"], lora_dropout=0.1, bias="none", task_type="CAUSAL_LM" .

Only the adapter matrices are trained; the original weights remain unchanged, which makes it possible to merge the adapters back into the base model after training for inference without adding latency.

Implementation

The example workflow uses the Hugging Face libraries: transformers, datasets, peft, and accelerate.

First, we load a causal language model and its tokenizer, then we wrap the model with a LoRA adapter using the configuration described above.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
import torch

model_name = "gpt2"  # or any causal LM
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# LoRA configuration
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["c_attn"],
    lora_dropout=0.1,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

The call to print_trainable_parameters() shows how many weights are trainable. For a GPT‑2‑sized model, after applying LoRA the trainable parameters are 294,912 out of 124,734,720 total, i.e., 0.2364% of all weights.

Next we prepare the training data. The IMDB dataset is filtered to retain only positive reviews (label 1) for sentiment‑aligned fine‑tuning.

Each training example is formatted as the string "Review: TL;DR: Positive." to teach the model to append the summary.

We then set up the Trainer with arguments that enable mixed‑precision training and gradient accumulation. Training arguments include: output_dir=\"\/gpt2-imdb-finetune\", per_device_train_batch_size=2, gradient_accumulation_steps=2, learning_rate=2e-4, num_train_epochs=2, logging_steps=50, fp16=True.

Gradient accumulation with steps=2 yields an effective batch size of 4 despite a per‑device batch size of 2. (Derived from source_10)

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir=\"\/gpt2-imdb-finetune\",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=2,
    learning_rate=2e-4,
    num_train_epochs=2,
    logging_steps=50,
    fp16=True,
    save_strategy=\"epoch\",
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,  # assume pre‑processed
    tokenizer=tokenizer,
)

trainer.train()

Run the executable notebook directly in your browser.

QLoRA vs LoRA vs Full Fine-Tuning

Understanding which adaptation paradigm fits your compute constraints and data density is critical. Below is a comprehensive comparison of Full Fine-Tuning, Standard LoRA, and QLoRA (Quantized Low-Rank Adaptation).

GPU Memory (VRAM) & Resource Requirements

To successfully launch fine-tuning runs, matching model size to GPU VRAM is vital. Below is a structured benchmark estimation showing the VRAM requirements for different baseline models using different ranks ($r$ values).

Merging Adapters for Production Deployment

Once training completes, serving the model via separate base weights and adapter layers adds runtime latency. To optimize deployment, merge the adapter parameters back into the base model's weight matrices before exporting.

The code block below demonstrates how to load the base model, load the trained LoRA adapters, merge the weights, and export the unified standalone model to disk.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model_name = "gpt2"
adapter_dir = "./gpt2-imdb-finetune"
output_dir = "./merged-gpt2-imdb"

# 1. Load base model in FP16 for clean weight merging
print("Loading base model...")
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# 2. Load Peft wrapper containing trained adapter weights
print("Loading adapter weights...")
model = PeftModel.from_pretrained(base_model, adapter_dir)

# 3. Merge weights in-place and unload PEFT structures
print("Merging adapters...")
merged_model = model.merge_and_unload()

# 4. Export unified standalone weights to disk
print("Saving merged model...")
merged_model.save_pretrained(output_dir)
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
tokenizer.save_pretrained(output_dir)
print(f"Model saved successfully to {output_dir}")

High-Performance Alternatives: Unsloth & Axolotl

While Hugging Face's PEFT library is the industry standard reference, it has performance bottlenecks. In 2026, specialized training setups often replace PEFT with Unsloth or Axolotl for high-performance optimization.

• Unsloth: Hand-written GPU kernels specifically for LoRA. It speeds up training by 2x to 5x while using 60% less VRAM, making it possible to fine-tune Llama 3 8B on a standard 12GB VRAM consumer GPU.
• Axolotl: A configuration-driven orchestration wrapper supporting multi-GPU configurations, DeepSpeed, and advanced packing algorithms for maximum training speed.

Common Troubleshooting & Error Mitigation

When running LoRA adaptation cycles, developer teams frequently run into several technical bottlenecks. Here is how to mitigate the most common issues:

1. Out of Memory (OOM) Errors

If you encounter CUDA out of memory during training, lower the per_device_train_batch_size to 1, and raise gradient_accumulation_steps proportionally (e.g. from 2 to 4) to maintain the effective batch size. Additionally, verify gradient_checkpointing=True is set in your TrainingArguments to store only a subset of activations.

2. Tokenizer Parallelism Deadlocks

Hugging Face's default configuration sometimes deadlocks on multi-threaded CPUs. Suppress this warning by setting the environment variable export TOKENIZERS_PARALLELISM=false before launching python scripts.

3. Loss Instability (NaN / Inf values)

If your training loss spikes to NaN, switch from standard mixed-precision FP16 to BF16 (if your hardware supports Ampere or newer architecture). BF16 has a larger dynamic range, preventing underflow or overflow bugs in gradient calculation.

Examples

Example 1

Fine‑tuning GPT‑2 on the IMDB positive‑review subset so the model learns to output movie‑review‑style text ending with "TL;DR: Positive." After training, the model generates text matching the positive review style, e.g., appending "TL;DR: Positive."

Example 2

Adapting Llama 3 8B to a domain‑specific corpus (e.g., medical notes) using LoRA with r=8, achieving <0.3% trainable parameters. The base Llama 3 8B model contains approximately 8 billion parameters.

Example 3

Generating product reviews that automatically include a summary tagline after LoRA‑based fine‑tuning on an e‑commerce dataset. The approach works because LoRA freezes the original model parameters and injects small adapter matrices A and B.

FAQs

Conclusion

LoRA transforms the way we adapt large language models by making fine‑tuning accessible, fast, and resource‑friendly. By freezing the bulk of the pretrained weights and learning only compact adapter matrices, we can specialize models for sentiment analysis, domain‑specific language generation, or any other task without the prohibitive cost of full fine‑tuning. The hands‑on example with GPT‑2 and the IMDB dataset shows how just a few lines of code and a modest GPU can yield a model that reliably produces positive‑review style text. As LLMs continue to grow, techniques like LoRA will remain essential for bringing cutting‑edge AI to a broader audience.

Explore more technical guides and tutorials on our articles page.