How GPT (LLM) Works: The Next Word Predictor

Scenario context

At its core, GPT asks one question, repeated: "Given everything so far, what is the most likely next token?" Tokens are not words — they're subword units. The Transformer architecture uses self-attention to weigh how much each token should influence the prediction. Sampling strategies (temperature, top-p) control how creative or deterministic the output is.

flowchart TD
    A[User Prompt]…

This walkthrough focuses on **Real-World Applications of GPT-Style LLMs**.

Before making a design choice, clarify:

- GPT-style large language models power a remarkable breadth of products across industries.
- Understanding the underlying mechanism helps you appreciate why they excel at some tasks and fail at others.
- Code Generation and Completion Tools like GitHub Copilot, Cursor, and Amazon CodeWhisperer use fine-tuned LLMs to autocomplete code, generate functions from comments, explain snippets, and write tests.

**Concepts in play:** deep-learning-foundations, api-gateway, llm-prompt-engineering, vector-embeddings-and-similarity

## Stage 1 — Practical: Using GPT via API

- The OpenAI API exposes GPT models over a simple HTTP interface.
- Here is a minimal Python example using the openai library: Parameter breakdown: Best practices: - Use a low temperature ( 0.0 – 0.3 ) for factual tasks (summarization, extraction, classification).
- - Use a higher temperature ( 0.7 – 1.0 ) for creative tasks (brainstorming, story writing).
- - Always set max tokens explicitly to avoid unexpected cost spikes.

*Applies [**deep-learning-foundations**](/concept/deep-learning-foundations) to the decision above.*

```python
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY from environment

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain how softmax turns logits into probabilities."}
    ],
    temperature=0.7,      # 0 = deterministic, 1 = high creativity
    max_tokens=512,       # cap the response length
    top_p=0.95,           # nucleus sampling threshold
)

print(response.choices[0].message.content)

Observable outcome: - Always set max tokens explicitly to avoid unexpected cost spikes.

Stage 2 — Deep Dive: The Transformer Under the Hood: Attention in Plain Language

• GPT is built from stacked Transformer decoder blocks.
• Each block runs multi-head self-attention .
• Self-attention answers: "For each position in the sequence, how much should I attend to every other position?" Example: "The animal didn't cross the street because it was too tired." - Self-attention figures out that "i…
• This diagram shows how all five input tokens flow into the Self-Attention layer simultaneously, which computes contextual representations for each token based on every other token in the sequence.

flowchart LR
    Token1[The] --> Attn[Self-Attention Layer]
    Token2[animal] --> Attn
    Token3[didn't] --> Attn
    Token4[cross] --> Attn
    Token5[it] --> Attn
    Attn --> Rep[Contextual Representation of each token]
    Rep --> FF[Feed-Forward Layer]
    FF --> Next[Next token distribution]

flowchart LR
    IN[Input Tokens [T1, T2, ... Tn]] --> ME[Token + Positional Embedding]
    ME --> MSA["Masked Self-Attention (causal: no future tokens)"]
    MSA --> AN1[Add & Norm]
    AN1 --> FFN[Feed-Forward Network GELU activation]
    FFN --> AN2[Add & Norm]
    AN2 --> OUT[Output Hidden State  next Decoder block]

Observable outcome: This diagram shows how all five input tokens flow into the Self-Attention layer simultaneously, which computes contextual representations for each token based on every other token in the sequence.

Stage 3 — HuggingFace Transformers & vLLM: Running GPT-Style Inference in Python

• HuggingFace Transformers is the standard Python library for loading and running open-weight language models (GPT-2, Mistral, LLaMA).
• Its pipeline("text-generation") abstraction handles tokenization, the autoregressive decode loop, and sampling — mirroring every step in the GPT Inference Pipeline diagram from this post, from token embeddings through s…
• vLLM is a high-throughput inference server that wraps HuggingFace-compatible models behind an OpenAI-compatible REST API using PagedAttention and continuous batching — allowing you to swap a local model in place of gpt-…
• HuggingFace Transformers vs vLLM — which to use: For a full deep-dive on HuggingFace Transformers inference and vLLM deployment, a dedicated follow-up post is planned.

# pip install transformers torch

# ── HuggingFace pipeline: GPT-2 text generation ───────────────────────────────
from transformers import pipeline, set_seed

# pipeline() wraps: tokenizer → model forward pass → sampling → decode
generator = pipeline(
    "text-generation",
    model="gpt2",           # 117 M params; runs on CPU for demos
)
set_seed(42)               # reproducibility

outputs = generator(
    "The best way to learn distributed systems is",
    max_new_tokens=60,      # cap output length (maps to max_tokens in OpenAI API)
    temperature=0.7,        # < 1 → more focused; > 1 → more creative
    top_p=0.9,              # nucleus sampling — sample from top 90% of mass
    do_sample=True,
    num_return_sequences=1,
)
print(outputs[0]["generated_text"])

# ── Manual decode loop (mirrors the flowchart in this post step-by-step) ─────
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("gpt2")
model     = AutoModelForCausalLM.from_pretrained("gpt2")

input_ids = tokenizer.encode("Distributed systems fail when", return_tensors="pt")

with torch.no_grad():
    for _ in range(20):                              # generate 20 tokens
        logits   = model(input_ids).logits           # → [1, seq_len, 50257]
        next_id  = logits[:, -1, :].argmax(-1)       # greedy: highest logit
        input_ids = torch.cat([input_ids,
                                next_id.unsqueeze(0)], dim=-1)  # append

print(tokenizer.decode(input_ids[0]))

Observable outcome: HuggingFace Transformers vs vLLM — which to use: For a full deep-dive on HuggingFace Transformers inference and vLLM deployment, a dedicated follow-up post is planned.

Stage 4 — Key Lessons About How GPT Works

• After walking through every component — tokenization, embeddings, self-attention, logits, and sampling — here are the five lessons that matter most: 1.
• Tokens, not words, are the unit of computation.
• It sees integer IDs mapped to subword pieces.
• This has practical consequences: long words, rare words, and non-English text cost more tokens and may be represented less reliably.

Observable outcome: This has practical consequences: long words, rare words, and non-English text cost more tokens and may be represented less reliably.

Stage 5 — Test Your Understanding

• GPT outputs a logit of 8.3 for "Paris" and 6.1 for "London." Which is more probable after softmax?
• What would temperature=0 mean operationally?
• In the self-attention formula, what do Q, K, and V represent in intuitive terms?
• Why doesn't GPT know about an event from last week?

Observable outcome: Why doesn't GPT know about an event from last week?

End-to-end outcome

GPT-style large language models power a remarkable breadth of products across industries.

Alternatives considered

Decision Guide: When to Use GPT

Use GPT when you need flexible language generation: summarization, Q&A, code completion, or open-ended chat.

Failure and degraded modes

Trade-offs & Failure Modes: GPT's Known Limitations

• Detection: Trade-offs & Failure Modes: GPT's Known Limitations
• User or operator impact: Trade-offs & Failure Modes: GPT's Known Limitations