Tokenization Explained: How LLMs Understand Text

Deep Dive: Stage 1 — How Byte Pair Encoding Builds a Vocabulary

• BPE starts with individual characters and iteratively merges the most frequent adjacent pair, growing the vocabulary until a target size is reached.
• Step-by-step trace on "low lower lowest": Result: Common words stay whole ( the , in , is ).
• Rare or new words are split into known subword pieces ( tokenization → token , ization ).

Initial characters: ['l','o','w',' ','l','o','w','e','r',' ','l','o','w','e','s','t']

Round 1: Most frequent pair = ('l','o') → merge to 'lo'
→ ['lo','w',' ','lo','w','e','r',' ','lo','w','e','s','t']

Round 2: Most frequent pair = ('lo','w') → merge to 'low'
→ ['low',' ','low','e','r',' ','low','e','s','t']

Round 3: Most frequent pair = ('low','e') → merge to 'lowe'
→ ['low',' ','lowe','r',' ','lowe','s','t']

...continues until vocabulary size target is reached

graph TD
    A[All characters in training corpus] --> B[Find most frequent adjacent pair]
    B --> C[Merge pair into a new token]
    C --> D[Update frequency counts]
    D --> E{Vocabulary size reached?}
    E -->|No| B
    E -->|Yes| F[Final vocabulary]

Observable outcome: Rare or new words are split into known subword pieces ( tokenization → token , ization ).

Deep Dive: Stage 2 — What a Token Actually Looks Like in Practice

• Key observations: - tokenization → 2 tokens - The space before is is part of the is token - !
• is its own token - Cost = number of tokens , not number of words Token counting rule of thumb (English): 1 token ≈ 4 characters ≈ 0.75 words.

import tiktoken                    # OpenAI's tokenizer library

enc = tiktoken.encoding_for_model("gpt-4o")

text = "Tokenization is fascinating!"
tokens = enc.encode(text)
print(tokens)
# → [8908, 2065, 374, 41353, 0]   (5 tokens)

decoded = [enc.decode([t]) for t in tokens]
print(decoded)
# → ['Token', 'ization', ' is', ' fascinating', '!']

sequenceDiagram
    participant T as Text Input
    participant TZ as Tokenizer
    participant M as LLM
    participant D as Detokenizer
    T->>TZ: raw string
    TZ->>M: token ID sequence
    M->>D: output token IDs
    D-->>T: decoded text output

Observable outcome: is its own token - Cost = number of tokens , not number of words Token counting rule of thumb (English): 1 token ≈ 4 characters ≈ 0.75 words.

Deep Dive: Stage 3 — BPE Training and Encoding Flow

• The BPE algorithm has two distinct phases: a one-time vocabulary training phase (offline) and a per-text encoding phase (online at inference time).
• The critical insight is that the merge rules learned during training are applied in the same order during encoding.
• Rule 1 (first merge) is always applied before Rule 2 (second merge).
• This determinism ensures that the same text always tokenizes to the same integer sequence — a requirement for reproducible model behavior.

graph TD
    subgraph Training Phase offline
        A[Large text corpus] --> B[Initialize: each character is a token]
        B --> C[Count all adjacent token pairs]
        C --> D[Merge the most frequent pair into a new token]
        D --> E[Update pair counts]
        E --> F{Vocab size target reached?}
        F -->|No| C
        F -->|Yes| G[Final vocabulary saved]
    end
    subgraph Encoding Phase online
        H[Input text] --> I[Split into characters]
        I --> J[Apply learned merge rules in order]
        J --> K[Token sequence produced]
        K --> L[Convert tokens to integer IDs]
        L --> M[Integer sequence fed to LLM]
    end
    G --> J

Observable outcome: This determinism ensures that the same text always tokenizes to the same integer sequence — a requirement for reproducible model behavior.

Deep Dive: Stage 4 — What to Learn Next

• • LLM Terms: A Helpful Glossary - LLM Hyperparameters: Temperature, Top-p, and Top-k - RAG Explained: How to Give Your LLM a Brain Upgrade

Observable outcome: - LLM Terms: A Helpful Glossary - LLM Hyperparameters: Temperature, Top-p, and Top-k - RAG Explained: How to Give Your LLM a Brain Upgrade

Deep Dive: Stage 5 — Hands-On: Explore Tokenization with tiktoken

• The best way to build intuition for tokenization is to run experiments yourself.
• The tiktoken library (used by OpenAI's GPT models) is available as a Python package.
• Install and import: Experiment 1 — See how your prompt is tokenized: Experiment 2 — Compare English vs.
• a non-Latin script: The same semantic content uses 2.4× more tokens in Japanese — directly increasing API cost and consuming more of the context window.

pip install tiktoken

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")

prompt = "The quick brown fox jumps over the lazy dog."
tokens = enc.encode(prompt)
print(f"Token count: {len(tokens)}")
print([enc.decode([t]) for t in tokens])
# Token count: 10
# ['The', ' quick', ' brown', ' fox', ' jumps', ' over', ' the', ' lazy', ' dog', '.']

Observable outcome: a non-Latin script: The same semantic content uses 2.4× more tokens in Japanese — directly increasing API cost and consuming more of the context window.

Deep Dive: Stage 6 — HuggingFace Tokenizers: BPE and WordPiece in Two Lines of Code

• HuggingFace Tokenizers is the open-source Python library that powers tokenization for GPT-2, BERT, LLaMA, Mistral, and virtually every modern LLM — it exposes both the tokenizers (Rust-backed, fast) and transformers (hi…
• The library makes every concept in this post directly observable: you can load real BPE (GPT-2) and WordPiece (BERT) tokenizers, inspect split boundaries, count tokens, and measure the cost difference between languages…
• The prefix in BERT WordPiece tokens marks a continuation sub-word (part of a longer word), while GPT-2 BPE uses the Ġ prefix to mark a token that follows a space.
• Both are subword algorithms but produce different split boundaries — a subtle source of model-specific behavior differences.

# pip install transformers tokenizers

from transformers import AutoTokenizer

# ── BPE tokenizer (GPT-2 style) ───────────────────────────────────────────────
gpt2_tok = AutoTokenizer.from_pretrained("gpt2")

text = "Tokenization is fascinating!"
gpt2_ids   = gpt2_tok.encode(text)
gpt2_split = gpt2_tok.convert_ids_to_tokens(gpt2_ids)
print("GPT-2 BPE:")
print("  IDs:    ", gpt2_ids)       # → [30642, 1634, 318, 17049, 0]
print("  Tokens: ", gpt2_split)     # → ['Token', 'ization', 'Ġis', 'Ġfascinating', '!']
print("  Count:  ", len(gpt2_ids))  # → 5

# ── WordPiece tokenizer (BERT style) ──────────────────────────────────────────
bert_tok = AutoTokenizer.from_pretrained("bert-base-uncased")

bert_ids   = bert_tok.encode(text, add_special_tokens=False)
bert_split = bert_tok.convert_ids_to_tokens(bert_ids)
print("\nBERT WordPiece:")
print("  IDs:    ", bert_ids)       # → [19204, 3989, 2003, 18895, 999]
print("  Tokens: ", bert_split)     # → ['token', '##ization', 'is', 'fascinating', '!']
print("  Count:  ", len(bert_ids))  # → 5
# Note: '##' prefix marks continuation sub-words in WordPiece

# ── Token cost comparison: English vs. Japanese ───────────────────────────────
english  = "Artificial intelligence is transforming software."
japanese = "人工知能はソフトウェアを変革しています。"

print("\nToken counts (GPT-2 BPE):")
print(f"  English:  {len(gpt2_tok.encode(english))} tokens")   # → ~8
print(f"  Japanese: {len(gpt2_tok.encode(japanese))} tokens")  # → ~25+ (2-3× more)

# ── Reproduce the "strawberry" counting problem ───────────────────────────────
word = "strawberry"
pieces = gpt2_tok.convert_ids_to_tokens(gpt2_tok.encode(word))
print(f"\n'strawberry' splits to: {pieces}")
# → ['straw', 'berry']  — model sees 2 tokens, not 10 characters

# ── Count prompt tokens before sending to API ────────────────────────────────
# Use the model-specific tokenizer to get an accurate count
system_prompt = "You are a helpful assistant. Always answer concisely."
user_message  = "Explain the difference between SQL and NoSQL databases."
full_prompt   = system_prompt + "\n" + user_message

token_count = len(gpt2_tok.encode(full_prompt))
cost_usd    = token_count / 1_000_000 * 5.00  # GPT-4o input pricing
print(f"\nPrompt token count: {token_count}")
print(f"Cost per 1 call: ${cost_usd:.6f}")

Observable outcome: Both are subword algorithms but produce different split boundaries — a subtle source of model-specific behavior differences.