Dense LLM Architecture: How Every Parameter Works on Every Token

Scenario context

In a dense LLM every single parameter is active for every token in every forward pass — no routing, no selection. A transformer block runs multi-head self-attention (Q, K, V) followed by a feed-forward network (FFN) with roughly 4× the hidden dimension. Stack 32–96 of those blocks and you have GPT-3, LLaMA-2, or Mistral. The compute cost scales with the product of parameters and tokens, which is why models like Mixt…

This walkthrough focuses on The Universal Participation Rule: What Makes a Model "Dense".

Before making a design choice, clarify:

• You are teaching a class of 175 billion students.
• Every time a question comes in, every student raises their hand.
• Not just the specialists, not just the ones who slept through yesterday's lecture — every single one.

Concepts in play: deep-learning-foundations, vector-embeddings-and-similarity, caching, core-algorithm-patterns

Stage 1 — The Transformer Block: Anatomy of a Single Dense Layer

• Every large language model is built by stacking identical transformer blocks.
• Within each block, every token passes through the same two major sub-layers in sequence: multi-head self-attention and a feed-forward network (FFN) .
• Residual connections and layer norms wrap each sub-layer to stabilize gradients during training.
• Here is the precise data flow through one transformer block: 1.

graph TD
    A[Input Token Embeddings] --> LN1[Layer Norm 1]
    LN1 --> MHA[Multi-Head Self-Attention Q K V]
    MHA --> ADD1[Residual Add]
    A --> ADD1
    ADD1 --> LN2[Layer Norm 2]
    LN2 --> FFN[Feed-Forward Network - two linear layers with activation]
    FFN --> ADD2[Residual Add]
    ADD1 --> ADD2
    ADD2 --> OUT[Output to Next Transformer Block]

Observable outcome: Here is the precise data flow through one transformer block: 1.

Stage 2 — How Attention Heads Divide and Conquer Token Relationships

• Self-attention allows every token to look at every other token in the sequence simultaneously.
• But running attention just once would force a single set of weights to capture every type of linguistic relationship — syntax, semantics, coreference, positional proximity — all at once.
• Multi-head attention solves this by running several independent attention operations in parallel.
• The model dimension d model is split evenly across h heads.

Observable outcome: The model dimension d model is split evenly across h heads.

Stage 3 — The Feed-Forward Network: Where the Transformer Stores World Knowledge

• If attention decides which tokens to look at , the FFN decides what to do with the information once it has been gathered.
• The FFN is often described as the model's long-term memory because it stores factual associations learned during pre-training — for example, that "Eiffel Tower" is associated with "Paris" and "iron lattice construction"…
• This is not a metaphor: probing experiments on frozen FFN weights can successfully decode factual associations without any attention context, confirming that knowledge is encoded directly in the FFN weight matrices.
• Internals: Structure, Activation Functions, and Parameter Distribution The canonical FFN is a two-layer structure: - Up-projection: A linear layer expands the d model -dimensional vector to 4 × d model (e.g., 4096 → 16,…

Observable outcome: Internals: Structure, Activation Functions, and Parameter Distribution The canonical FFN is a two-layer structure: - Up-projection: A linear layer expands the d model -dimensional vector to 4 × d model (e.g., 4096 → 16,…

Stage 4 — Stacking Layers: How Depth Transforms Syntax into Reasoning

• A single transformer block produces a context-enriched token representation — but a single block is not an LLM.
• The power of large models comes from stacking many blocks in sequence, with each block receiving the output of the previous one.
• Depth creates qualitative capability.
• The representation of a token changes character as it moves through the stack: - Early blocks (layers 1–8 in a 32-layer model): Representations reflect surface patterns — tokenization artifacts, punctuation roles, and b…

graph TD
    EMB[Input Embeddings - Surface token identity] --> B1[Block 1-8 - Syntax and Surface Structure]
    B1 --> B2[Block 9-16 - Local Semantic Relationships]
    B2 --> B3[Block 17-24 - Entity Types and Coreference]
    B3 --> B4[Block 25-32 - Task Reasoning and World Knowledge]
    B4 --> LM[Language Model Head - Next Token Logits]

Observable outcome: The representation of a token changes character as it moves through the stack: - Early blocks (layers 1–8 in a 32-layer model): Representations reflect surface patterns — tokenization artifacts, punctuation roles, and b…

Stage 5 — The Dense Scaling Wall: When More Parameters Becomes Prohibitively Expensive

• The dense architecture has a clean, predictable scaling story — and a brutal cost curve.
• Chinchilla scaling laws (Hoffmann et al., 2022) established that a compute-optimal dense model should train on approximately 20 tokens per parameter.
• A 7B model is optimally trained on ~140 billion tokens; a 70B model on ~1.4 trillion tokens.
• Training models that are too large for the token budget underperforms smaller models trained longer.

Observable outcome: Training models that are too large for the token budget underperforms smaller models trained longer.

Stage 6 — Real Dense Models: GPT-3, LLaMA-2, and Mistral as Case Studies

• All three of the most influential open and closed dense model families share the same core architecture but diverge in the engineering choices that determine efficiency: GPT-3 (175B, OpenAI, 2020): The canonical large d…
• 96 transformer blocks, 96 attention heads, 12,288 model dimension, learned positional embeddings, Pre-LN layout.
• All 96 heads maintain full KV pairs, meaning the KV cache at inference scales as num layers × sequence length × 2 × d model .
• At long context lengths, this cache alone can exceed available GPU memory.

graph LR
    subgraph GPT3[GPT-3 175B]
        A1[Learned Positional Encoding]
        A2[Full Multi-Head Attention - 96 heads - 96 KV heads]
        A3[LayerNorm - Pre-LN]
        A4[ReLU FFN - 4x expansion]
    end
    subgraph LLaMA2[LLaMA-2 70B]
        B1[RoPE Rotary Positional Encoding]
        B2[GQA - 64 Q heads - 8 KV heads]
        B3[RMSNorm]
        B4[SwiGLU FFN]
    end
    subgraph Mistral[Mistral 7B]
        C1[RoPE Rotary Positional Encoding]
        C2[Sliding Window Attention - 4096 token window]
        C3[RMSNorm]
        C4[SwiGLU FFN - dense - all params active]
    end

Observable outcome: At long context lengths, this cache alone can exceed available GPU memory.

End-to-end outcome

You are teaching a class of 175 billion students.

Alternatives considered

Choosing the Right Dense Model: A Decision Guide for Production and Research

The single most important variable when choosing a dense LLM is the GPU memory envelope.

Failure and degraded modes

Dense vs. Sparse: The Real Trade-offs Every LLM Engineer Must Weigh

• Detection: Dense and sparse (MoE) architectures represent different engineering bets.
• User or operator impact: Dense and sparse (MoE) architectures represent different engineering bets.