SFT for LLMs: A Practical Guide to Supervised Fine-Tuning

Scenario context

Supervised fine-tuning (SFT) is the stage where a pretrained model learns task-specific response behavior from curated input-output examples. It is usually the first alignment step after pretraining and often the foundation for later RLHF. Good SFT depends more on data quality and format consistency than on exotic training tricks.

---

LLaMA-2 was released as a capable base model — but it wouldn't follow instructio…

This walkthrough focuses on Why SFT Is the First Practical Alignment Layer.

Before making a design choice, clarify:

• LLaMA-2 was released as a capable base model — but it wouldn't follow instructions.
• Meta released LLaMA-2-Chat two weeks later after SFT on 27,540 instruction-response pairs.
• The same 7B weights, entirely different behaviour.

Concepts in play: deep-learning-foundations, consistency-replication, design-patterns-gof, event-driven-architecture

Stage 1 — Building SFT Data That Actually Improves Behavior

• Most SFT failures are data failures.
• Core design rules - Keep prompt format consistent across the dataset.
• • Make expected outputs unambiguous.
• • Include edge cases, not only happy-path examples.

flowchart TD
    Raw[Raw Prompt-Response Pairs]
    Clean["Deduplicate + Normalize (schema, formatting)"]
    Format["Format as Chat Messages (system / user / assistant)"]
    Split[Train / Val / Holdout Split]
    Tokenize["Tokenize + Mask (loss on assistant only)"]
    Train["Train SFT Model (cross-entropy loss)"]
    Eval["Evaluate on Held-Out Set (task + behavioral metrics)"]
    Done{Quality Gate Pass?}
    Deploy[Publish Model Version]
    Fix[Improve Data Balance and Retrain]

    Raw --> Clean --> Format --> Split --> Tokenize --> Train --> Eval --> Done
    Done -->|Yes| Deploy
    Done -->|No| Fix --> Train

sequenceDiagram
    participant D as DataLoader
    participant T as Tokenizer
    participant M as Model (+ LoRA)
    participant L as Loss Function
    participant O as Optimizer

    D->>T: instruction + response batch
    T->>M: Token IDs + attention mask
    T->>M: Labels (assistant tokens only)
    M->>M: Forward pass
    M->>L: Predicted logits
    L->>L: Cross-entropy on assistant span
    L->>O: Backpropagate gradients
    O->>M: Update LoRA A & B weights

Observable outcome: - Include edge cases, not only happy-path examples.

Stage 2 — What SFT Optimizes in the Model

• If your target deployment expects chat format, train in chat format.
• SFT should mirror inference conditions as much as possible.

{
  "messages": [
    {"role": "system", "content": "You are a concise cloud architecture assistant."},
    {"role": "user", "content": "Compare event-driven and request-response systems."},
    {"role": "assistant", "content": "Event-driven systems react to events asynchronously ..."}
  ]
}

Observable outcome: SFT should mirror inference conditions as much as possible.

Stage 3 — What SFT Optimizes in the Model

• SFT typically uses next-token prediction loss on labeled assistant responses, conditioned on prior context.
• y T , loss is: [ \mathcal{L} {SFT} = - \sum {t=1}^{T} \log P(y t \mid x, y {<t}) ] Where x includes system and user context.
• Important practical point You usually mask loss on user/system tokens and compute loss only on assistant target spans.
• If you compute loss on everything, the model may waste capacity modeling prompt boilerplate instead of response quality.

Observable outcome: If you compute loss on everything, the model may waste capacity modeling prompt boilerplate instead of response quality.

Stage 4 — Deep Dive: Distribution Shift, Forgetting, and Evaluation

• Internals: why catastrophic forgetting happens If your SFT dataset is narrow, the model can over-specialize and lose general capabilities from pretraining.
• This is catastrophic forgetting.
• You will observe: - strong performance on narrow in-domain prompts, - degraded general QA or reasoning behavior, - brittle outputs when user phrasing changes.
• A practical mitigation is mixing: - domain-specific instruction data, - general instruction-following exemplars, - format-control examples.

Observable outcome: A practical mitigation is mixing: - domain-specific instruction data, - general instruction-following exemplars, - format-control examples.

Stage 5 — SFT Pipeline from Dataset to Deployment

• Treat SFT as a data-centric iteration loop.
• Most gains come from better datasets and evaluations, not from endlessly changing optimizer settings.

flowchart TD
    A[Collect prompts and gold responses] --> B[Clean and normalize schema]
    B --> C[Split train, validation, holdout]
    C --> D[Train SFT model or adapter]
    D --> E[Run task and behavioral evals]
    E --> F{Pass quality gate?}
    F -- No --> G[Fix data balance and retrain]
    G --> D
    F -- Yes --> H[Publish model version]
    H --> I[Monitor drift and regression]

Observable outcome: Most gains come from better datasets and evaluations, not from endlessly changing optimizer settings.

Stage 6 — Real-World Applications of SFT

• Customer support copilots SFT teaches: - policy-compliant tone, - escalation patterns, - concise troubleshooting sequences.
• Developer assistants SFT improves: - structured explanations, - code-style consistency, - safer command recommendations.
• Internal enterprise knowledge bots SFT aligns: - response templates, - references to approved documents, - role-specific answer depth.

Observable outcome: Internal enterprise knowledge bots SFT aligns: - response templates, - references to approved documents, - role-specific answer depth.

End-to-end outcome

LLaMA-2 was released as a capable base model — but it wouldn't follow instructions.

Alternatives considered

Decision Guide: When SFT Is Enough and When It Is Not

SFT is foundational, but it is one layer in a larger alignment and product architecture stack.

Failure and degraded modes

Trade-offs & Failure Modes: Trade-offs and Failure Modes in SFT

• Detection: SFT does not magically "fix" model truthfulness.
• User or operator impact: SFT does not magically "fix" model truthfulness.