Types of LLM Quantization: By Timing, Scope, and Mapping

Scenario context

There is no single "best" LLM quantization. You classify and choose quantization along three axes: when you quantize (timing), what you quantize (scope), and how values are encoded (mapping). In practice, most teams start with weight quantization, then add activation quantization when needed.

The internals

At inference time, quantized LLMs run through three hidden mechanisms:

1. **Tensor repr…

This walkthrough focuses on Deep Dive: Inside the Runtime: Why Timing, Scope, and Mapping Interact.

Before making a design choice, clarify:

• The internals At inference time, quantized LLMs run through three hidden mechanisms: 1.
• Tensor representation changes: weights/activations stored in low-bit format with scales.
• Kernel path changes: runtime chooses quantized GEMM kernels if available.

Concepts in play: quantization-and-llm-inference, deep-learning-foundations, caching, design-patterns-gof

Stage 1 — Quantization Is a Design Space, Not One Switch

• Deploying LLaMA-3-70B at full fp16 precision requires approximately 140GB of VRAM — two A100 80GB GPUs at roughly $8–12K/month in cloud GPU rental.
• At Q4 K M quantization (4-bit weights), the same model fits on a single consumer GPU with 24GB VRAM.
• The measured quality difference on most standard benchmarks?
• That is the practical upside, but "just quantize it to 4-bit" is not a strategy — it is a gamble.

Observable outcome: That is the practical upside, but "just quantize it to 4-bit" is not a strategy — it is a gamble.

Stage 2 — Three Classification Axes You Can Apply to Any Quantized LLM

• Before selecting a library or hardware backend, use this compact classifier: Use timing for lifecycle decisions, scope for memory/bandwidth impact, and mapping for error behavior.
• 📊 PTQ vs QAT Taxonomy This flowchart compares the two primary quantization timing strategies side by side.
• PTQ starts from an already-trained FP16 model, runs a small calibration dataset to determine quantization parameters layer by layer, and produces a 4-bit or INT8 model ready for deployment — requiring no gradient update…
• QAT instead embeds simulated low-precision arithmetic into the fine-tuning loop so that weights adapt to quantization noise before deployment, yielding better quality retention at the same bit width.

flowchart LR
    subgraph PTQ["Post-Training Quantization (PTQ)"]
        P1[Trained FP16 Model]
        P2["Calibration Dataset (representative prompts)"]
        P3["Quantize Weights (layer by layer)"]
        P4[Deploy 4-bit / INT8 Model]
        P1 --> P2 --> P3 --> P4
    end

    subgraph QAT["Quantization-Aware Training (QAT)"]
        Q1[FP16 Model or Adapter]
        Q2[Simulate Low-Precision During Fine-Tuning]
        Q3[Weights Adapt to Quantization Noise]
        Q4[Deploy with Better Quality Retention]
        Q1 --> Q2 --> Q3 --> Q4
    end

    PTQ -->|"Quality drops?"| QAT

flowchart TD
    Quant[Quantization Scope Decision]
    WeightOnly["Weights-Only (most linear layers)"]
    WeightAct["Weights + Activations (higher throughput)"]
    KVCache["+ KV Cache (long-context memory)"]
    Mixed["Selective / Mixed (sensitive layers = BF16)"]

    Quant --> WeightOnly
    WeightOnly -->|"Need more memory saving"| WeightAct
    WeightAct -->|"Long-context prompts"| KVCache
    WeightOnly -->|"Quality sensitive layers"| Mixed

    subgraph Granularity[Quantization Granularity]
        PerTensor["Per-Tensor (one scale for whole tensor)"]
        PerChannel["Per-Channel (scale per output channel)"]
        PerToken["Per-Token (scale per token, activations)"]
    end

    WeightOnly --> Granularity

Observable outcome: QAT instead embeds simulated low-precision arithmetic into the fine-tuning loop so that weights adapt to quantization noise before deployment, yielding better quality retention at the same bit width.

Stage 3 — By Scope: Which Parts of the LLM Get Quantized

• Scope determines where quantization is applied in the model and runtime path.
• Common pattern: quantize most linear layer weights first, keep sensitive heads in higher precision, then add activations only after quality baselines pass.

Observable outcome: Common pattern: quantize most linear layer weights first, keep sensitive heads in higher precision, then add activations only after quality baselines pass.

Stage 4 — By Mapping: How Float Values Become Low-Bit Values

• Mapping defines the numeric transformation from float tensors to low-bit formats.
• Symmetric mapping Values are centered around zero, typically with a single scale.
• • Works well when tensor distributions are roughly zero-centered.
• Asymmetric mapping Uses scale plus zero-point, allowing shifted ranges.

Observable outcome: Asymmetric mapping Uses scale plus zero-point, allowing shifted ranges.

Stage 5 — Popular Approaches: Weight Quantization and Activation Quantization

• These are the two most widely discussed practical approaches.
• Weight quantization Weight quantization compresses model parameters (usually linear layers).
• Why it is popular: - Big memory savings with manageable quality impact.
• • Often enough to move from "cannot deploy" to "production feasible." Typical setup: - 8-bit (safer) or 4-bit (more aggressive) weights.

Observable outcome: - Often enough to move from "cannot deploy" to "production feasible." Typical setup: - 8-bit (safer) or 4-bit (more aggressive) weights.

Stage 6 — Visualizing a Quantization Strategy Flow

• This flowchart shows the complete quantization strategy selection process from a pre-trained model to production deployment.
• After choosing between PTQ — which flows immediately into scope and mapping selection — and QAT — which requires a fine-tuning pass with simulated quantization noise — both paths converge at a benchmarking gate covering…
• If targets are not met, the loop expands scope or adjusts the numerical mapping; once satisfied, a canary deployment validates stability before full rollout.
• The key takeaway is that quantization is iterative: commit to production only after the benchmark gate passes.

flowchart TD
    A[Start with FP16 or BF16 LLM] --> B[Choose Timing Axis]
    B --> C{PTQ or QAT?}
    C -->|PTQ| D[Select Scope: Weights Only]
    C -->|QAT| E[Train with Quantization Simulation]
    D --> F[Select Mapping: Symmetric Asymmetric NF4]
    E --> F
    F --> G[Benchmark Memory Latency Quality]
    G --> H{Targets met?}
    H -- No --> I[Expand Scope or Adjust Mapping]
    I --> G
    H -- Yes --> J[Canary Deploy + Fallback]
    J --> K[Production Rollout]

Observable outcome: The key takeaway is that quantization is iterative: commit to production only after the benchmark gate passes.

End-to-end outcome

After working through the stages, you should be able to explain:

• • Classify decisions by timing, scope, and mapping before choosing tools.
• • Weight quantization is usually the highest-ROI first step.

Alternatives considered

Timing answers when quantization appears during the model lifecycle.

Timing answers when quantization appears during the model lifecycle.

Post-Training Quantization (PTQ)

PTQ quantizes an already trained model.

Quantization-Aware Training (QAT)

QAT simulates low-precision behavior during fine-tuning/training so weights adapt to quantization noise.

Decision Guide: Choosing by Constraint

If deployment is blocked by memory, optimize scope first.

Failure and degraded modes

Trade-offs & Failure Modes: Trade-offs and Failure Modes You Should Expect

• Detection: Intermediate-level rule: do not ship quantization based on memory metrics alone.
• User or operator impact: Intermediate-level rule: do not ship quantization based on memory metrics alone.