RLHF Explained: How We Teach AI to Be Nice

Scenario context

A raw LLM is a super-smart parrot that read the entire internet — including its worst parts. RLHF (Reinforcement Learning from Human Feedback) is the training pipeline that transforms it from a pattern-matching engine into an assistant that is helpful, harmless, and honest.

Left unconstrained, PPO finds the response the Reward Model scores highest — and it may be nonsensical text that superficially satisfies t…

This walkthrough focuses on The KL Divergence Constraint: Why the Model Doesn't Collapse.

Before making a design choice, clarify:

• Left unconstrained, PPO finds the response the Reward Model scores highest — and it may be nonsensical text that superficially satisfies the reward function (Goodhart's Law).
• The KL divergence penalty prevents this: $$\text{Maximize: } \mathbb{E}\left[ R(x, y) - \beta \cdot \log \frac{\pi {RL}(y x)}{\pi {SFT}(y x)} \right]$$ Plain English: - $R(x, y)$: reward score — higher is better.
• • $\beta \cdot \log(\pi {RL}/\pi {SFT})$: how much the RL policy has drifted from the SFT baseline.

Concepts in play: deep-learning-foundations, api-gateway, llm-agents-and-langgraph

Stage 1 — The Parrot Who Read Everything

• Imagine a parrot that has read every book, forum post, Reddit thread, and dark corner of the web.
• Ask it anything and it can produce text.
• But it might: - Answer in the style of a conspiracy forum.
• • Generate offensive content because that's statistically common in its training data.

Observable outcome: - Generate offensive content because that's statistically common in its training data.

Stage 2 — RLHF Key Vocabulary

• Before diving into the pipeline, here is a quick reference for the terms that appear throughout this post.
• If you have seen these words before but never formally defined them, this table is your anchor.
• These nine terms are the full vocabulary of modern RLHF.
• By the end of this post, you will have seen all of them in context.

sequenceDiagram
    participant H as Human Labelers
    participant SFT as SFT Model
    participant RM as Reward Model
    participant PPO as PPO Trainer
    participant P as RLHF Policy

    SFT->>H: Generate candidate responses
    H->>H: Rank responses (A > B)
    H->>RM: Train on preference pairs
    RM->>RM: Learn scalar reward score
    SFT->>PPO: Initialize policy from SFT
    PPO->>RM: Score generated responses
    RM-->>PPO: Reward signal
    PPO->>PPO: Update policy (KL-constrained)
    PPO->>P: RLHF-aligned model

stateDiagram-v2
    [*] --> PreTraining
    PreTraining --> SFT : Fine-tune on human demos
    SFT --> RewardModel : Train on preference pairs
    RewardModel --> PPO : RL optimization with KL penalty
    PPO --> RLHF_Tuned : Aligned policy
    RLHF_Tuned --> [*]

Observable outcome: By the end of this post, you will have seen all of them in context.

Stage 3 — The Three Stages of RLHF

• This flowchart compresses the entire RLHF pipeline into a three-node summary.
• Stage 1 (SFT) produces a behaviorally reasonable model by imitating human-written responses.
• Stage 2 (Reward Model) learns a scalar preference signal from pairwise human rankings.
• Stage 3 (PPO) uses that signal to shift the policy toward higher-scoring outputs while the KL penalty prevents collapse.

flowchart LR
    SFT[Stage 1: SFT Supervised Fine-Tuning Human writes ideal answers  imitation learning]
    RM[Stage 2: Reward Model Humans rank A vs B  train a preference predictor]
    PPO[Stage 3: PPO RL with KL penalty  optimize policy to maximize reward]

    SFT --> RM --> PPO

Observable outcome: Stage 3 (PPO) uses that signal to shift the policy toward higher-scoring outputs while the KL penalty prevents collapse.

Stage 4 — Deep Dive: Human Preference Data and Annotation

• Labelers evaluate output pairs using a rubric: The preference signal is a ranking (A B), not a rating (A = 8/10, B = 6/10).
• Rankings are more reliable and faster to collect than absolute scores.

Observable outcome: Rankings are more reliable and faster to collect than absolute scores.

Stage 5 — Real-World Applications of RLHF

• RLHF is not an academic curiosity — it is the alignment backbone of the most widely deployed AI products today.
• Why does the product landscape matter for a beginner?
• Because it shows that RLHF is not one fixed recipe.
• OpenAI pioneered it, Anthropic scaled it with AI feedback, and Meta simplified it with DPO.

Observable outcome: OpenAI pioneered it, Anthropic scaled it with AI feedback, and Meta simplified it with DPO.

Stage 6 — Practical: What RLHF Looks Like in Code

• Most researchers use the trl library (Transformer Reinforcement Learning) from Hugging Face to implement RLHF.
• Below is a minimal sketch of the three stages.
• These are not production-ready snippets, but they illustrate the shape of the pipeline.
• Stage 1 — SFT is standard fine-tuning: Stage 2 — Reward Model training on preference pairs: Stage 3 — PPO optimization loop: The kl penalty and init kl coef are the tunable β from the KL divergence formula above.

from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTTrainer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3-8B")

trainer = SFTTrainer(
    model=model,
    train_dataset=sft_dataset,   # rows: {"prompt": ..., "response": ...}
    dataset_text_field="response",
)
trainer.train()
sft_model = trainer.model

from trl import RewardTrainer, RewardConfig

# Preference dataset rows: {"chosen": ..., "rejected": ...}
# Both are full prompt+response strings.
reward_trainer = RewardTrainer(
    model=AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B"),
    args=RewardConfig(output_dir="./reward_model"),
    train_dataset=preference_dataset,
)
reward_trainer.train()
reward_model = reward_trainer.model

Observable outcome: Stage 1 — SFT is standard fine-tuning: Stage 2 — Reward Model training on preference pairs: Stage 3 — PPO optimization loop: The kl penalty and init kl coef are the tunable β from the KL divergence formula above.

End-to-end outcome

Left unconstrained, PPO finds the response the Reward Model scores highest — and it may be nonsensical text that superficially satisfies the reward function (Goodhart's Law).

Alternatives considered

Decision Guide: RLHF vs DPO vs RLAIF

Decision Guide: RLHF vs DPO vs RLAIF

Failure and degraded modes

Trade-offs & Failure Modes: RLHF Trade-offs and Failure Modes

• Detection: Alternatives and successors: - DPO (Direct Preference Optimization): Skips the RM and PPO entirely — optimizes preference directly.
• User or operator impact: Alternatives and successors: - DPO (Direct Preference Optimization): Skips the RM and PPO entirely — optimizes preference directly.