LLM Hyperparameters Guide: Temperature, Top-P, and Top-K Explained

Scenario context

Temperature, Top-p, and Top-k are three sampling controls that determine how "creative" or "deterministic" an LLM's output is. Temperature rescales the probability distribution; Top-k limits the candidate pool by count; Top-p limits it by cumulative probability. Understanding all three — and when to combine them — is essential for production LLM work.

Think of sampling controls as three dials on a mixing board…

This walkthrough focuses on The Three Knobs Behind Every LLM Response.

Before making a design choice, clarify:

• Think of sampling controls as three dials on a mixing board: - Temperature — the creativity dial.
• Turn it up for surprising outputs; turn it down for safe, predictable ones.
• It reshapes the entire probability distribution before any other control runs.

Concepts in play: llm-prompt-engineering, vector-embeddings-and-similarity, quantization-and-llm-inference, deep-learning-foundations

Stage 1 — The Basics: What Sampling Parameters Control

• After an LLM finishes processing your prompt, it outputs a probability distribution over its entire vocabulary — potentially 50,000+ tokens.
• Sampling parameters control how you draw a single token from that distribution.
• The three parameters and their roles: - Temperature — scales the probability distribution.
• Low values concentrate probability on the top tokens; high values spread it out.

Observable outcome: Low values concentrate probability on the top tokens; high values spread it out.

Stage 2 — Temperature: Sharpening or Flattening the Probability Curve

• After the model computes raw scores (logits) for each candidate token, it applies a softmax to get probabilities.
• Temperature $T$ scales the logits before softmax: $$P i = rac{\exp(z i / T)}{\sum j \exp(z j / T)}$$ Concrete example — logits for three tokens: Mat=2.0 , Rug=1.5 , Moon=0.2 At T=0.5, "Mat" wins 73% of the time — focuse…
• At T=2.0, "Moon" steals 20% — surprising.

Observable outcome: At T=2.0, "Moon" steals 20% — surprising.

Stage 3 — Top-k: Hard Cap on the Candidate Pool

• Top-k restricts sampling to only the k highest-probability next tokens.
• After Temperature scaling, only the top k tokens are kept; all others are set to zero probability (and renormalized).
• Trade-off: Top-k is a static threshold.
• If the distribution is wide (many roughly equal options), k=10 may still allow incoherent tokens.

All tokens: [Mat=0.52, Rug=0.35, Floor=0.09, Moon=0.04, ...]
Top-k = 2:  [Mat=0.52, Rug=0.35] → renormalized → [Mat=0.60, Rug=0.40]
Moon is impossible.

Observable outcome: If the distribution is wide (many roughly equal options), k=10 may still allow incoherent tokens.

Stage 4 — Top-p (Nucleus Sampling): Dynamic Candidate Pool by Mass

• Top-p is more adaptive — it includes the smallest set of tokens whose cumulative probability reaches p .
• The key word is cumulative : the candidate set size changes dynamically.
• If the model is very confident, nucleus may be just 1–2 tokens.
• If uncertain, it may include 50+.

Sorted: [Mat=0.50, Rug=0.30, Floor=0.10, Moon=0.05, Star=0.03, ...]
Top-p = 0.9:
  Cumulative after Mat:   0.50  (< 0.9, continue)
  Cumulative after Rug:   0.80  (< 0.9, continue)
  Cumulative after Floor: 0.90  (= 0.9, stop)
Nucleus: {Mat, Rug, Floor}

graph LR
    A[Logits] --> B[Apply Temperature]
    B --> C[Sort by probability]
    C --> D{Top-k cutoff}
    D --> E{Top-p cumulative check}
    E --> F[Renormalize remaining tokens]
    F --> G[Sample final token]

Observable outcome: If uncertain, it may include 50+.

Stage 5 — How Token Sampling Flows: The Complete Pipeline

• Key observation: Temperature, Top-k, and Top-p are applied in sequence, not simultaneously.
• Temperature reshapes the entire distribution first; Top-k then hard-caps the candidates; Top-p further filters by cumulative probability.
• Each step makes the sampling more constrained.
• When controls are most impactful:

flowchart TD
    A[LLM produces raw logits for all tokens] --> B[Apply Temperature scaling]
    B --> C{Top-k enabled?}
    C -->|Yes| D[Keep top k tokens, zero rest]
    C -->|No| E[Keep all tokens]
    D --> F{Top-p enabled?}
    E --> F
    F -->|Yes| G[Keep smallest set with cumulative prob >= p]
    F -->|No| H[Keep current token set]
    G --> I[Renormalize probabilities to sum to 1]
    H --> I
    I --> J[Sample one token from renormalized distribution]
    J --> K[Append token to output]
    K --> L{End of sequence?}
    L -->|No| A
    L -->|Yes| M[Return complete response]

Observable outcome: When controls are most impactful:

Stage 6 — How Temperature, Top-k, and Top-p Interact

• These three controls are typically applied in sequence: Temperature → Top-k → Top-p.
• Interaction gotcha: Setting both Top-k and Top-p is redundant in most cases — whichever is more restrictive wins.
• Most practitioners use either temperature + top-p OR temperature alone (at T=0 for deterministic tasks).
• 📊 Sampling Strategy Comparison

# OpenAI API — all three at once
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Write a haiku about databases."}],
    temperature=0.8,
    top_p=0.95,
    # Note: OpenAI API doesn't expose top_k directly; Anthropic and local models do
)

flowchart LR
    LG[Logits] --> TS[Temperature Scale]
    TS --> TK[Top-K Filter]
    TK --> TP[Top-P Nucleus]
    TP --> SM[Softmax]
    SM --> SMP[Sample Token]

Observable outcome: 📊 Sampling Strategy Comparison

End-to-end outcome

Think of sampling controls as three dials on a mixing board: - Temperature — the creativity dial.

Alternatives considered

Decision Guide: Choosing Sampling Settings by Task

Use the table below as a quick reference when you first approach a new task and don't yet know your ideal settings.

Failure and degraded modes

Trade-offs & Failure Modes: Failure Modes and Mitigation

• Detection: Trade-offs & Failure Modes: Failure Modes and Mitigation
• User or operator impact: Trade-offs & Failure Modes: Failure Modes and Mitigation