Fine-Tuning LLMs with LoRA and QLoRA: A Practical Deep-Dive

Scenario context

LoRA freezes the base model and trains two tiny matrices per layer — 0.1 % of parameters, 70 % less GPU memory, near-identical quality. QLoRA adds 4-bit NF4 quantization of the frozen base, enabling 70B fine-tuning on 2× A100 80 GB instead of 8×. Use HuggingFace PEFT + TRL + bitsandbytes; always call merge_and_unload() before serving; monitor both task loss and a general-capability eval (e.g. MMLU) to catch catast…

This walkthrough focuses on The Memory Wall That Blocked Every LLM Fine-Tune Before 2022.

Before making a design choice, clarify:

• Before 2022, fine-tuning a large language model was an HPC problem.
• A 7-billion-parameter model stored in 32-bit floating point occupies 28 GB of GPU memory for the weights alone.
• Add the Adam optimizer and the situation explodes: Adam maintains a momentum estimate and a variance estimate for each parameter — that is four copies of the model in memory simultaneously (weights + gradients + two opt…

Concepts in play: deep-learning-foundations, design-patterns-gof, quantization-and-llm-inference, graph-traversal-algorithms

Stage 1 — LoRA in Plain English: Margin Notes on a Frozen Textbook

• Think of the pre-trained base model as a textbook that already contains enormous knowledge about language, reasoning, and the world.
• Full fine-tuning is like reprinting the entire textbook with a few changed chapters — enormously expensive and wasteful since most pages need no change at all.
• LoRA does something more elegant.
• It keeps the original textbook exactly as it is and writes margin notes — small, targeted annotations that modify how specific passages are interpreted.

Observable outcome: It keeps the original textbook exactly as it is and writes margin notes — small, targeted annotations that modify how specific passages are interpreted.

Stage 2 — How LoRA Adds a Parallel Path to Every Attention Layer

• LoRA does not modify the frozen weight matrix.
• Instead, it runs a lightweight parallel branch alongside the frozen forward pass.
• For every attention projection the forward computation becomes: where W frozen is completely frozen and B · A is the trainable low-rank adapter.
• At initialisation, B is set to all zeros and A is drawn from a small Gaussian.

y = W_frozen · x  +  (B · A) · x

graph TD
    Input[Input activation x] --> FrozenW[Frozen weight W - shape d x k]
    Input --> AdapterA[LoRA down-projection A - shape d x r]
    AdapterA --> AdapterB[LoRA up-projection B - shape r x k]
    FrozenW --> Adder[Add frozen and adapter outputs]
    AdapterB --> Adder
    Adder --> Output[Layer output y]

Observable outcome: At initialisation, B is set to all zeros and A is drawn from a small Gaussian.

Stage 3 — Deep Dive: The Math of Low-Rank Decomposition, NF4 Quantization, and QLoRA's Architecture

• Mathematical Model: The Low-Rank Decomposition Formula and Why Fine-Tuning Deltas Are Naturally Sparse The diagram below shows every stage, from the raw base model to a production-ready merged model served behind vLLM.
• Pay particular attention to the merge step: LoRA adapters loaded at inference time without merging incur a 2× compute overhead in the adapter projection, and the adapter checkpoint is architecturally fragile (it require…
• Each arrow in the diagram hides a potential failure mode.
• The quantisation step must use bnb 4bit quant type="nf4" and bnb 4bit compute dtype=torch.bfloat16 — using float16 as the compute dtype can produce NaN gradients on A100 and H100 GPUs.

graph TD
    BaseModel[Base model in fp16] --> Quantise[NF4 4-bit quantisation via bitsandbytes]
    Quantise --> FrozenBase[Frozen 4-bit base in memory]
    FrozenBase --> PrepKbit[prepare model for kbit training - gradient checkpointing]
    PrepKbit --> AttachAdapters[Attach LoRA adapters in bfloat16 via PEFT]
    TrainingData[Formatted training data] --> SFTLoop[SFTTrainer supervised fine-tuning loop]
    AttachAdapters --> SFTLoop
    SFTLoop --> SaveAdapter[Save LoRA adapter checkpoint]
    SaveAdapter --> MergeStep[Load base in fp16 and call merge and unload]
    MergeStep --> MergedModel[Merged fp16 model - no adapter dependency]
    MergedModel --> ServingLayer[vLLM or TGI serving endpoint]

Observable outcome: The quantisation step must use bnb 4bit quant type="nf4" and bnb 4bit compute dtype=torch.bfloat16 — using float16 as the compute dtype can produce NaN gradients on A100 and H100 GPUs.

Stage 4 — Complete Working Example: Fine-Tuning Mistral-7B with QLoRA on a Custom Support Dataset

• This section walks through a complete, runnable QLoRA fine-tune from data preparation to inference with the merged model.
• The scenario is a Mistral-7B-Instruct model adapted to summarise customer complaint tickets in a specific corporate format — a realistic domain-adaptation task that highlights every config decision you will encounter in…
• The code is split into two scripts that mirror the two operational stages: training (including adapter saving) and merging + inference.
• Both scripts are annotated inline with the reasoning behind each hyperparameter choice.

# qlora_train.py — fine-tune Mistral-7B-Instruct with QLoRA on custom data
import torch
from datasets import Dataset
from transformers import (
    AutoTokenizer,
    AutoModelForCausalLM,
    BitsAndBytesConfig,
    TrainingArguments,
)
from peft import LoraConfig, get_peft_model, TaskType, prepare_model_for_kbit_training
from trl import SFTTrainer

MODEL_ID = "mistralai/Mistral-7B-Instruct-v0.3"

# --- 1. Training data: instruction-response pairs ---
# In production, load from JSONL or Hugging Face dataset.
# Minimum viable size: ~200 examples. Recommended: 500–2000.
TRAINING_EXAMPLES = [
    {
        "instruction": "Summarize the following customer complaint in one sentence.",
        "input": "I've been waiting 3 weeks for my order and no one is responding to my emails.",
        "response": "Customer has been waiting 3 weeks for an undelivered order with no email response from support.",
    },
    # ... add 500-2000 more examples for real use cases
]

def format_for_mistral(ex: dict) -> str:
    """Apply Mistral's instruction template.

    Critically: use the model's own chat template, not a custom one.
    Mismatched templates are the #1 cause of fine-tuned models ignoring instructions.
    For other base models, use: tokenizer.apply_chat_template(messages, tokenize=False)
    """
    user_msg = ex["instruction"]
    if ex.get("input"):
        user_msg += f"\n\n{ex['input']}"
    return f"<s>[INST] {user_msg} [/INST] {ex['response']} </s>"

# --- 2. QLoRA quantization config ---
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",            # Normal Float 4 — optimal for normally distributed LLM weights
    bnb_4bit_compute_dtype=torch.bfloat16, # bfloat16 avoids NaN on A100/H100; do NOT use float16
    bnb_4bit_use_double_quant=True,        # quantise the quantisation constants for ~0.5 GB additional saving
)

# --- 3. Load model + tokenizer ---
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token  # Mistral has no dedicated pad token
tokenizer.padding_side = "right"           # right-padding avoids position embedding shifts

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",          # fills GPU first, spills to CPU if needed
    trust_remote_code=True,
)
# Enables gradient checkpointing for the 4-bit base — required before get_peft_model
model = prepare_model_for_kbit_training(model)

# --- 4. LoRA adapter config ---
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,               # rank — 16 balances adapter capacity vs parameter count
    lora_alpha=32,      # effective scale = alpha / r = 2.0; amplifies adapter contribution
    lora_dropout=0.05,  # light regularisation; 0.0 is acceptable for larger datasets
    bias="none",        # don't train biases — adds minimal expressiveness for significant overhead
    target_modules=[    # target all attention + MLP projections for maximal adaptation
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Expected output: trainable params: 41,943,040 || all params: 7,283,965,952 || trainable%: 0.5759

# --- 5. Dataset ---
dataset = Dataset.from_list([
    {"text": format_for_mistral(ex)} for ex in TRAINING_EXAMPLES
])

# --- 6. Training ---
trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048,       # set to your median sequence length; longer = more GPU memory
    args=TrainingArguments(
        output_dir="./mistral-7b-qlora",
        num_train_epochs=3,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,   # effective batch size = 2 × 8 = 16
        warmup_ratio=0.05,               # warm up over first 5% of steps; prevents early LR spikes
        learning_rate=2e-4,              # standard QLoRA LR; lower to 1e-4 if loss is unstable
        fp16=False,
        bf16=True,                       # must match bnb_4bit_compute_dtype
        logging_steps=10,
        save_strategy="epoch",
        evaluation_strategy="no",        # add eval_dataset + strategy="steps" for early stopping
        report_to="none",                # swap to "wandb" for production monitoring
    ),
)
trainer.train()

# --- 7. Save adapter only (not the full merged model) ---
model.save_pretrained("./mistral-7b-qlora-adapter")
tokenizer.save_pretrained("./mistral-7b-qlora-adapter")
print("Adapter saved. Run qlora_merge_and_infer.py to merge before serving.")

# qlora_merge_and_infer.py — merge adapter into base model weights, then run inference.
# This script runs AFTER training is complete.
# Always merge before deploying to production — adapter-only serving is 2x slower.
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

BASE_MODEL   = "mistralai/Mistral-7B-Instruct-v0.3"
ADAPTER_PATH = "./mistral-7b-qlora-adapter"
MERGED_PATH  = "./mistral-7b-merged"

def merge_adapter():
    """Fuse the LoRA adapter matrices (B × A) into the frozen base weights (W).

    Why: After merge_and_unload(), W_new = W_frozen + B × A for each layer.
    The adapter is removed, removing the extra projection overhead at inference.
    Load the base in fp16 for merging — NOT 4-bit.
    Merging requires full-precision matrix addition; NF4 cannot represent the result accurately.
    """
    print("Loading base model in fp16 for merging...")
    tokenizer = AutoTokenizer.from_pretrained(ADAPTER_PATH)
    base = AutoModelForCausalLM.from_pretrained(
        BASE_MODEL,
        torch_dtype=torch.float16,  # full precision needed for merge arithmetic
        device_map="auto"
    )
    print("Attaching LoRA adapter...")
    model = PeftModel.from_pretrained(base, ADAPTER_PATH)

    print("Merging and unloading...")
    merged = model.merge_and_unload()  # fuses B×A into W, removes PEFT wrapper

    merged.save_pretrained(MERGED_PATH)
    tokenizer.save_pretrained(MERGED_PATH)
    print(f"Merged model saved to {MERGED_PATH}")

def infer(prompt: str) -> str:
    """Run inference on the merged model.

    The merged model is a standard HuggingFace CausalLM — no PEFT dependency.
    In production, replace this with vLLM for batched serving with 3–5x higher throughput.
    """
    tokenizer = AutoTokenizer.from_pretrained(MERGED_PATH)
    model = AutoModelForCausalLM.from_pretrained(
        MERGED_PATH,
        torch_dtype=torch.float16,
        device_map="auto"
    )
    inputs = tokenizer(
        f"<s>[INST] {prompt} [/INST]",
        return_tensors="pt"
    ).to(model.device)

    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=256,
            temperature=0.1,   # low temperature for deterministic summarisation
            do_sample=True,
        )
    # Slice off the input tokens; decode only the generated response
    response_tokens = output[0][inputs["input_ids"].shape[1]:]
    return tokenizer.decode(response_tokens, skip_special_tokens=True)

if __name__ == "__main__":
    merge_adapter()
    test_prompt = (
        "Summarize the following customer complaint in one sentence.\n\n"
        "I ordered a laptop stand three weeks ago. It never arrived, tracking hasn't updated "
        "in two weeks, and your customer support keeps closing my tickets without responding."
    )
    print("\nModel response:")
    print(infer(test_prompt))

Observable outcome: Both scripts are annotated inline with the reasoning behind each hyperparameter choice.

Stage 5 — Evaluating Your Fine-Tune: Loss Curves, Perplexity, and the MMLU Canary

• Training loss alone is a dangerously incomplete signal.
• A model can reach near-zero training loss on 800 examples while simultaneously catastrophically forgetting how to perform multi-step reasoning on topics outside the training set.
• Production-safe evaluation requires three distinct signals monitored together.
• Training loss tells you whether the model is learning the target task format.

Observable outcome: Training loss tells you whether the model is learning the target task format.

Stage 6 — Advanced Deployment Patterns: Adapter Composition, Continued Pre-Training, and Multi-Tenant Serving

• Once you have mastered the basic QLoRA fine-tune loop, three advanced patterns become relevant as you move from single-model prototypes to production serving systems.
• Stacking Multiple LoRA Adapters with LoraHub LoraHub (Huang et al.
• 2023) demonstrates that LoRA adapters trained on different tasks can be combined by weighted interpolation.
• If you have an adapter fine-tuned on code generation and a separate adapter fine-tuned on structured data extraction, LoraHub searches for a coefficient vector [w1, w2] such that ΔW = w1 × ΔW code + w2 × ΔW extraction m…

Observable outcome: If you have an adapter fine-tuned on code generation and a separate adapter fine-tuned on structured data extraction, LoraHub searches for a coefficient vector [w1, w2] such that ΔW = w1 × ΔW code + w2 × ΔW extraction m…

End-to-end outcome

After working through the stages, you should be able to explain:

• merge and unload() is a deployment gate, not an optimisation.
• Adapters left attached at inference time add a full extra matrix multiplication to every forward pass.

Alternatives considered

Choosing Your Fine-Tuning Strategy: LoRA vs QLoRA vs Full Fine-Tune vs DPO

The decision between LoRA and QLoRA is almost entirely a VRAM question.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.