LLM Terms You Should Know: A Helpful Glossary

Scenario context

The world of LLMs has its own dense vocabulary. This post is your decoder ring — covering foundation terms (tokens, context window), generation settings (temperature, top-p), safety concepts (hallucination, grounding), and architecture terms (attention, fine-tuning, RAG).

Observable outcome: Context Window The maximum number of tokens an LLM can process in a single interaction — both the prompt and the response combined.

Stage 2 — Generation Controls: Temperature, Top-p, and Top-k

• These settings control how "creative" vs "deterministic" the model's output is.
• Temperature Scales the probability distribution before sampling.
• Higher = more random, more creative; lower = more focused, more predictable.
• • temperature = 0.0 → always picks the most likely token (deterministic) - temperature = 1.0 → samples according to the raw probability distribution - temperature 1.0 → more random; occasional incoherence Top-p (nucleus…

graph LR
    A[Logits from model] --> B[Apply Temperature scaling]
    B --> C{Sampling method}
    C -->|Top-k| D[Keep top k tokens]
    C -->|Top-p| E[Keep tokens until cumulative prob = p]
    D --> F[Sample final token]
    E --> F

Observable outcome: - temperature = 0.0 → always picks the most likely token (deterministic) - temperature = 1.0 → samples according to the raw probability distribution - temperature 1.0 → more random; occasional incoherence Top-p (nucleus…

Stage 3 — Safety and Quality Terms: Hallucination, Grounding, and Bias

• Hallucination When an LLM confidently generates false or nonsensical information.
• The model predicts probable-sounding tokens without checking facts.
• • Example: Ask "Who was the 12th US President in 1750?" and the model invents a name.
• • Mitigation: Grounding, RAG, source attribution, user education.

Observable outcome: - Mitigation: Grounding, RAG, source attribution, user education.

Stage 4 — Deep Dive: Architecture Terms: Attention, Fine-Tuning, and RAG

• Attention (Self-Attention) The core mechanism of Transformer models.
• Each token attends to every other token in the context window, weighted by relevance.
• This is what allows LLMs to handle long-range dependencies.
• Fine-Tuning Additional training of a pre-trained model on a smaller, task-specific dataset to improve performance for a specific domain or task format.

flowchart LR
    NW[New Knowledge] --> D{Update method?}
    D -- Static --> FT[Fine-tune model]
    D -- Dynamic --> RAG[RAG retrieval]
    FT --> MO[Model weights updated]
    RAG --> CT[Context injected]

Observable outcome: Fine-Tuning Additional training of a pre-trained model on a smaller, task-specific dataset to improve performance for a specific domain or task format.

Stage 5 — How an LLM Processes a Request

• Understanding the data flow from your prompt to the model's response helps demystify every term in this glossary.
• Each step maps to one or more of the concepts defined above.
• This loop — tokenize, attend, sample, repeat — is the heartbeat of every LLM interaction.
• Parameters like temperature and top-p only affect step F; everything else is deterministic given the same weights.

graph TD
    A[User types a Prompt] --> B[Tokenizer splits prompt into Tokens]
    B --> C[Token IDs fed into LLM within Context Window]
    C --> D[Self-Attention layers compute relationships between all tokens]
    D --> E[Logits output  raw scores for every possible next token]
    E --> F{Sampling settings applied}
    F -->|Temperature + Top-p/Top-k| G[Next token sampled]
    G --> H{End of sequence?}
    H -->|No| C
    H -->|Yes| I[Completion returned to user]

Observable outcome: Parameters like temperature and top-p only affect step F; everything else is deterministic given the same weights.

Stage 6 — Real-World Applications: Deployment Terms: Inference, Quantization, and Latency

• Inference The process of generating output from a trained model given an input.
• This is what happens at serving time (as opposed to training time).
• Quantization Reducing model precision (e.g., from 32-bit floats to 8-bit integers) to shrink model size and speed up inference, at a small accuracy cost.
• Essential for running large models on constrained hardware.

Observable outcome: Essential for running large models on constrained hardware.

End-to-end outcome

You're reading a paper about GPT-4 and hit "temperature", "top-p", "logits" in the same paragraph.

Alternatives considered

Decision Guide: Quick Reference: Term → Concept in One Line

Decision Guide: Quick Reference: Term → Concept in One Line

Failure and degraded modes

Trade-offs & Failure Modes: LLM Generation Settings

• Detection: Hallucination is the dominant failure mode across all LLM deployments.
• User or operator impact: Hallucination is the dominant failure mode across all LLM deployments.