Fine-Tuning LLMs: The Complete Engineer's Guide to SFT, LoRA, and RLHF

Scenario context

A pretrained LLM is a generalist. Fine-tuning makes it a specialist. Supervised Fine-Tuning (SFT) teaches it your domain's language through labeled examples. LoRA does the same with 99% fewer trainable parameters. RLHF shapes its behavior using human preference signals. Picking the right approach depends on your data budget, GPU budget, and whether you need format adaptation or behavioral alignment.

Your…

This walkthrough focuses on The Legal Document Assistant That Answered Like a Freshman.

Before making a design choice, clarify:

• Your team is three weeks into building an AI assistant for a mid-size law firm.
• The brief is clear: help paralegals review contracts, flag non-standard clauses, and suggest revisions in the firm's house style.
• You wire up GPT-4 with a system prompt describing the firm's preferred citation format, their standard indemnification language, and the three clause patterns their partners watch for.

Concepts in play: deep-learning-foundations, vector-databases-rag, llm-prompt-engineering, design-patterns-gof

Stage 1 — What Fine-Tuning Actually Is — and Three Things It Isn't

• Fine-tuning takes a pretrained language model — which has already learned the structure of language, world knowledge, and general reasoning from hundreds of billions of tokens — and updates its weights on a smaller, dom…
• After fine-tuning, the model has changed.
• Its parameters now encode both the broad pretraining knowledge and the specialized patterns from your domain.
• Here's what fine-tuning is not : It's not prompting.

Observable outcome: Here's what fine-tuning is not : It's not prompting.

Stage 2 — Three Ways to Fine-Tune: SFT, LoRA, and RLHF at a Glance

• Before diving into each technique, here's the high-level landscape.
• The three dominant fine-tuning approaches differ fundamentally in what they optimize, how much they cost, and what problem they solve best.
• The decision tree below shows how to navigate between them before you write a single line of training code.
• It is a practical guide, not an exhaustive taxonomy — most production systems combine multiple approaches (base pretraining → SFT → RLHF is exactly what OpenAI used for ChatGPT).

Observable outcome: It is a practical guide, not an exhaustive taxonomy — most production systems combine multiple approaches (base pretraining → SFT → RLHF is exactly what OpenAI used for ChatGPT).

Stage 3 — Supervised Fine-Tuning: Teaching an LLM Your Domain's Language Step by Step

• Supervised Fine-Tuning is the conceptually simplest approach.
• You provide the model with a dataset of (instruction, response) pairs — the outputs you want the model to produce — and train it using the same cross-entropy next-token-prediction loss used during pretraining.
• The difference is that pretraining runs on raw, unsupervised internet text; SFT runs on your curated, labeled examples.
• Dataset Format: Instruction-Response Pairs The standard format for SFT is an instruction template.

Observable outcome: Dataset Format: Instruction-Response Pairs The standard format for SFT is an instruction template.

Stage 4 — LoRA: Fine-Tuning 7B Parameters by Updating Only 0.1% of Them

• Supervised fine-tuning of a 7B-parameter model requires storing the full model in FP16 (~14GB), plus optimizer states in FP32 for Adam (~56GB for momentum and variance).
• Total: roughly 70GB of GPU memory for training — far beyond what a single A100 80GB can comfortably handle once gradients and activations are included.
• LoRA (Low-Rank Adaptation, Hu et al.
• 2021) sidesteps this entirely through a clever algebraic insight: the weight updates from fine-tuning tend to be low-rank.

Observable outcome: 2021) sidesteps this entirely through a clever algebraic insight: the weight updates from fine-tuning tend to be low-rank.

Stage 5 — RLHF: The Three-Stage Training Pipeline That Shaped ChatGPT

• Supervised fine-tuning and LoRA are both fundamentally about imitation learning — the model learns to replicate examples you labeled.
• This works well for format and domain adaptation.
• It fails for one important class of problem: teaching the model to be helpful , harmless , and honest in ways that are hard to express as labeled examples.
• When InstructGPT (the direct predecessor to ChatGPT) was SFT-trained on human-written responses, the model learned to produce text that looked like what a helpful assistant would say.

Observable outcome: When InstructGPT (the direct predecessor to ChatGPT) was SFT-trained on human-written responses, the model learned to produce text that looked like what a helpful assistant would say.

Stage 6 — Real-World Fine-Tuning: How GitHub Copilot, ChatGPT, and MedPaLM Were Built

• Fine-tuning is not an academic exercise.
• The most commercially successful AI products of the past three years are almost universally the result of disciplined fine-tuning pipelines.
• ChatGPT (OpenAI, 2022): The canonical RLHF success story.
• GPT-3.5 (a strong SFT-trained base) was aligned using the InstructGPT RLHF recipe: human-written SFT data → reward model trained on human comparisons → PPO optimization.

Observable outcome: GPT-3.5 (a strong SFT-trained base) was aligned using the InstructGPT RLHF recipe: human-written SFT data → reward model trained on human comparisons → PPO optimization.

End-to-end outcome

After working through the stages, you should be able to explain:

• These are the errors that appear most frequently in fine-tuning projects — not in the tutorial stage, but after teams move to production data.
• Too-small SFT datasets creating confident-but-wrong specialists.

Alternatives considered

Choosing Your Fine-Tuning Strategy: A Decision Guide for Engineers

The decision tree below distills the trade-off analysis into a practical flowchart for engineering teams.

Failure and degraded modes

Trade-offs and Failure Modes: Where Each Approach Breaks Down

• Detection: Every fine-tuning approach has a characteristic failure mode.
• User or operator impact: Every fine-tuning approach has a characteristic failure mode.