Tokenization Explained: How LLMs Understand Text

Scenario context

LLMs don't read words — they read tokens. A token is roughly 4 characters. Byte Pair Encoding (BPE) builds an efficient subword vocabulary by iteratively merging frequent character pairs. Tokenization choices directly affect cost, context limits, and why LLMs struggle with counting and spelling.

Computers only work with numbers. Before an LLM can process the sentence "Hello world", it must first map that text…

This walkthrough focuses on Why LLMs Can't Just Read Words.

Before making a design choice, clarify:

• Computers only work with numbers.
• Before an LLM can process the sentence "Hello world", it must first map that text to a sequence of integers.
• That mapping is tokenization .

Concepts in play: api-gateway, vector-embeddings-and-similarity, deep-learning-foundations, graph-traversal-algorithms

Stage 1 — How Byte Pair Encoding Builds a Vocabulary

• BPE starts with individual characters and iteratively merges the most frequent adjacent pair, growing the vocabulary until a target size is reached.
• Step-by-step trace on "low lower lowest": Result: Common words stay whole ( the , in , is ).
• Rare or new words are split into known subword pieces ( tokenization → token , ization ).

Initial characters: ['l','o','w',' ','l','o','w','e','r',' ','l','o','w','e','s','t']

Round 1: Most frequent pair = ('l','o') → merge to 'lo'
→ ['lo','w',' ','lo','w','e','r',' ','lo','w','e','s','t']

Round 2: Most frequent pair = ('lo','w') → merge to 'low'
→ ['low',' ','low','e','r',' ','low','e','s','t']

Round 3: Most frequent pair = ('low','e') → merge to 'lowe'
→ ['low',' ','lowe','r',' ','lowe','s','t']

...continues until vocabulary size target is reached

graph TD
    A[All characters in training corpus] --> B[Find most frequent adjacent pair]
    B --> C[Merge pair into a new token]
    C --> D[Update frequency counts]
    D --> E{Vocabulary size reached?}
    E -->|No| B
    E -->|Yes| F[Final vocabulary]

Observable outcome: Rare or new words are split into known subword pieces ( tokenization → token , ization ).

Stage 2 — What a Token Actually Looks Like in Practice

• Key observations: - tokenization → 2 tokens - The space before is is part of the is token - !
• is its own token - Cost = number of tokens , not number of words Token counting rule of thumb (English): 1 token ≈ 4 characters ≈ 0.75 words.

import tiktoken                    # OpenAI's tokenizer library

enc = tiktoken.encoding_for_model("gpt-4o")

text = "Tokenization is fascinating!"
tokens = enc.encode(text)
print(tokens)
# → [8908, 2065, 374, 41353, 0]   (5 tokens)

decoded = [enc.decode([t]) for t in tokens]
print(decoded)
# → ['Token', 'ization', ' is', ' fascinating', '!']

sequenceDiagram
    participant T as Text Input
    participant TZ as Tokenizer
    participant M as LLM
    participant D as Detokenizer
    T->>TZ: raw string
    TZ->>M: token ID sequence
    M->>D: output token IDs
    D-->>T: decoded text output

Observable outcome: is its own token - Cost = number of tokens , not number of words Token counting rule of thumb (English): 1 token ≈ 4 characters ≈ 0.75 words.

Stage 3 — BPE Training and Encoding Flow

• The BPE algorithm has two distinct phases: a one-time vocabulary training phase (offline) and a per-text encoding phase (online at inference time).
• The critical insight is that the merge rules learned during training are applied in the same order during encoding.
• Rule 1 (first merge) is always applied before Rule 2 (second merge).
• This determinism ensures that the same text always tokenizes to the same integer sequence — a requirement for reproducible model behavior.

graph TD
    subgraph Training Phase offline
        A[Large text corpus] --> B[Initialize: each character is a token]
        B --> C[Count all adjacent token pairs]
        C --> D[Merge the most frequent pair into a new token]
        D --> E[Update pair counts]
        E --> F{Vocab size target reached?}
        F -->|No| C
        F -->|Yes| G[Final vocabulary saved]
    end
    subgraph Encoding Phase online
        H[Input text] --> I[Split into characters]
        I --> J[Apply learned merge rules in order]
        J --> K[Token sequence produced]
        K --> L[Convert tokens to integer IDs]
        L --> M[Integer sequence fed to LLM]
    end
    G --> J

Observable outcome: This determinism ensures that the same text always tokenizes to the same integer sequence — a requirement for reproducible model behavior.

Stage 4 — What to Learn Next

• • LLM Terms: A Helpful Glossary - LLM Hyperparameters: Temperature, Top-p, and Top-k - RAG Explained: How to Give Your LLM a Brain Upgrade

Observable outcome: - LLM Terms: A Helpful Glossary - LLM Hyperparameters: Temperature, Top-p, and Top-k - RAG Explained: How to Give Your LLM a Brain Upgrade

Stage 5 — Hands-On: Explore Tokenization with tiktoken

• The best way to build intuition for tokenization is to run experiments yourself.
• The tiktoken library (used by OpenAI's GPT models) is available as a Python package.
• Install and import: Experiment 1 — See how your prompt is tokenized: Experiment 2 — Compare English vs.
• a non-Latin script: The same semantic content uses 2.4× more tokens in Japanese — directly increasing API cost and consuming more of the context window.

pip install tiktoken

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

prompt = "The quick brown fox jumps over the lazy dog."
tokens = enc.encode(prompt)
print(f"Token count: {len(tokens)}")
print([enc.decode([t]) for t in tokens])
# Token count: 10
# ['The', ' quick', ' brown', ' fox', ' jumps', ' over', ' the', ' lazy', ' dog', '.']

Observable outcome: a non-Latin script: The same semantic content uses 2.4× more tokens in Japanese — directly increasing API cost and consuming more of the context window.

Stage 6 — HuggingFace Tokenizers: BPE and WordPiece in Two Lines of Code

• HuggingFace Tokenizers is the open-source Python library that powers tokenization for GPT-2, BERT, LLaMA, Mistral, and virtually every modern LLM — it exposes both the tokenizers (Rust-backed, fast) and transformers (hi…
• The library makes every concept in this post directly observable: you can load real BPE (GPT-2) and WordPiece (BERT) tokenizers, inspect split boundaries, count tokens, and measure the cost difference between languages…
• The prefix in BERT WordPiece tokens marks a continuation sub-word (part of a longer word), while GPT-2 BPE uses the Ġ prefix to mark a token that follows a space.
• Both are subword algorithms but produce different split boundaries — a subtle source of model-specific behavior differences.

# pip install transformers tokenizers

from transformers import AutoTokenizer

# ── BPE tokenizer (GPT-2 style) ───────────────────────────────────────────────
gpt2_tok = AutoTokenizer.from_pretrained("gpt2")

text = "Tokenization is fascinating!"
gpt2_ids   = gpt2_tok.encode(text)
gpt2_split = gpt2_tok.convert_ids_to_tokens(gpt2_ids)
print("GPT-2 BPE:")
print("  IDs:    ", gpt2_ids)       # → [30642, 1634, 318, 17049, 0]
print("  Tokens: ", gpt2_split)     # → ['Token', 'ization', 'Ġis', 'Ġfascinating', '!']
print("  Count:  ", len(gpt2_ids))  # → 5

# ── WordPiece tokenizer (BERT style) ──────────────────────────────────────────
bert_tok = AutoTokenizer.from_pretrained("bert-base-uncased")

bert_ids   = bert_tok.encode(text, add_special_tokens=False)
bert_split = bert_tok.convert_ids_to_tokens(bert_ids)
print("\nBERT WordPiece:")
print("  IDs:    ", bert_ids)       # → [19204, 3989, 2003, 18895, 999]
print("  Tokens: ", bert_split)     # → ['token', '##ization', 'is', 'fascinating', '!']
print("  Count:  ", len(bert_ids))  # → 5
# Note: '##' prefix marks continuation sub-words in WordPiece

# ── Token cost comparison: English vs. Japanese ───────────────────────────────
english  = "Artificial intelligence is transforming software."
japanese = "人工知能はソフトウェアを変革しています。"

print("\nToken counts (GPT-2 BPE):")
print(f"  English:  {len(gpt2_tok.encode(english))} tokens")   # → ~8
print(f"  Japanese: {len(gpt2_tok.encode(japanese))} tokens")  # → ~25+ (2-3× more)

# ── Reproduce the "strawberry" counting problem ───────────────────────────────
word = "strawberry"
pieces = gpt2_tok.convert_ids_to_tokens(gpt2_tok.encode(word))
print(f"\n'strawberry' splits to: {pieces}")
# → ['straw', 'berry']  — model sees 2 tokens, not 10 characters

# ── Count prompt tokens before sending to API ────────────────────────────────
# Use the model-specific tokenizer to get an accurate count
system_prompt = "You are a helpful assistant. Always answer concisely."
user_message  = "Explain the difference between SQL and NoSQL databases."
full_prompt   = system_prompt + "\n" + user_message

token_count = len(gpt2_tok.encode(full_prompt))
cost_usd    = token_count / 1_000_000 * 5.00  # GPT-4o input pricing
print(f"\nPrompt token count: {token_count}")
print(f"Cost per 1 call: ${cost_usd:.6f}")

Observable outcome: Both are subword algorithms but produce different split boundaries — a subtle source of model-specific behavior differences.

End-to-end outcome

Computers only work with numbers.

Alternatives considered

Decision Guide: Practical Tokenization Guidelines

Decision Guide: Practical Tokenization Guidelines

Failure and degraded modes

Trade-offs & Failure Modes: Comparing Tokenization Algorithms

• Detection: All modern subword algorithms solve the same core problem differently: balance vocabulary size, sequence length, and handling of unseen (OOV) words.
• User or operator impact: All modern subword algorithms solve the same core problem differently: balance vocabulary size, sequence length, and handling of unseen (OOV) words.