GPTQ vs AWQ vs NF4: Choosing the Right LLM Quantization Pipeline

Scenario context

GPTQ, AWQ, and NF4 all shrink LLMs, but they optimize different constraints. GPTQ focuses on post-training reconstruction error, AWQ protects salient weights for better quality at low bits, and NF4 offers practical 4-bit compression through bitsandbytes-style pipelines. Choose by hardware path and quality budget.

The internals

Even at the same bit width, these methods change runtime behavior differentl…

This walkthrough focuses on Deep Dive: Why the Three Pipelines Behave Differently.

Before making a design choice, clarify:

• The internals Even at the same bit width, these methods change runtime behavior differently: - GPTQ/AWQ artifacts may trigger backend-specific packed kernels.
• • NF4 workflows rely on runtime dequantization behavior in bitsandbytes-compatible paths.
• • Layer sensitivity differs: attention projections, MLP projections, and output heads do not fail equally.

Concepts in play: quantization-and-llm-inference, deep-learning-foundations, stack-and-queue-data-structures, api-gateway

Stage 1 — GPTQ, AWQ, and NF4 in One Practical Snapshot

• First, one clarification: NF4 is a quantization data type/mapping choice, not a standalone algorithm like GPTQ or AWQ.
• In practice, teams still talk about an "NF4 pipeline" because the end-to-end workflow is distinct (commonly bitsandbytes + 4-bit loading).
• Mental model: GPTQ optimizes reconstruction error, AWQ protects salient weights, and NF4 changes the 4-bit value representation to better match weight distributions.
• 📊 Three-Method Comparison Flow This flow maps the three quantization pipelines side-by-side from a shared FP16 checkpoint to a 4-bit output artifact.

flowchart LR
    subgraph GPTQ[GPTQ Pipeline]
        G1[FP16 Checkpoint]
        G2[Calibration Dataset]
        G3[Layer-by-Layer Error Minimization]
        G4[4-bit Packed GPTQ Weights]
        G1 --> G2 --> G3 --> G4
    end

    subgraph AWQ[AWQ Pipeline]
        A1[FP16 Checkpoint]
        A2[Activation Saliency Analysis]
        A3[Protect Key Weights Quantize Rest]
        A4[AWQ 4-bit Artifacts]
        A1 --> A2 --> A3 --> A4
    end

    subgraph NF4[NF4 Pipeline]
        N1[FP16 Checkpoint]
        N2[load_in_4bit=nf4 bitsandbytes]
        N3[BF16 Compute Dtype]
        N4[Runtime 4-bit NF4 Model]
        N1 --> N2 --> N3 --> N4
    end

Observable outcome: 📊 Three-Method Comparison Flow This flow maps the three quantization pipelines side-by-side from a shared FP16 checkpoint to a 4-bit output artifact.

Stage 2 — GPTQ Pipeline: Error-Aware Post-Training Quantization

• GPTQ is usually run after model training using a calibration dataset.
• It quantizes layer by layer, solving for quantized weights that minimize output error.
• Start with FP16/BF16 checkpoint.
• Prepare representative calibration prompts.

Observable outcome: Prepare representative calibration prompts.

Stage 3 — AWQ Pipeline: Salient-Weight-Aware Quantization

• AWQ (Activation-aware Weight Quantization) uses activation signals to find important weights and preserve them more carefully during quantization.
• Run activation collection on representative prompts.
• Score or identify salient channels/weights.
• Apply quantization while protecting sensitive components.

Observable outcome: Apply quantization while protecting sensitive components.

Stage 4 — NF4 Pipeline: Non-Uniform 4-Bit in Practice

• NF4 (NormalFloat4) is commonly used through bitsandbytes-driven workflows.
• It is frequently paired with BF16 compute and optional double quantization for metadata compression.
• Load base model with load in 4bit=True .
• Set bnb 4bit quant type="nf4" .

Observable outcome: Set bnb 4bit quant type="nf4" .

Stage 5 — Real-World Applications: Which Pipeline Wins Where

• Case study 1: Support chatbot on shared GPU cluster Case study 2: Offline summarization batch jobs Case study 3: Domain-specific assistant (legal/finance) Scaling note: as traffic grows, kernel support and observability…
• 📊 GPTQ vs AWQ vs NF4 Method Comparison This diagram compares the three quantization methods across three decision dimensions: calibration approach, hardware target, and accuracy trade-off.
• Reading across the subgraphs, GPTQ relies on layer-by-layer error minimization with ExInt4 kernel support, AWQ uses activation-guided salient-weight protection with AutoAWQ fused kernels, and NF4 skips offline calibrati…
• The reader should evaluate hardware compatibility and quality retention together, not in isolation, before committing to a pipeline.

flowchart LR
    subgraph Calibration[Calibration Approach]
        G_cal[GPTQ: Layer-by-layer error minimization]
        A_cal[AWQ: Activation-guided salient weight protection]
        N_cal["NF4: No offline calibration (runtime 4-bit load)"]
    end

    subgraph HW[Hardware Target]
        G_hw[GPTQ: ExInt4 kernels AMD / NVIDIA]
        A_hw[AWQ: AutoAWQ fused kernels NVIDIA preferred]
        N_hw[NF4: bitsandbytes BF16 compute path]
    end

    subgraph Accuracy[Accuracy Trade-off]
        G_acc[GPTQ: Good with large calibration set]
        A_acc[AWQ: Best 4-bit quality on instruction tasks]
        N_acc[NF4: Slightly lower, fastest to deploy]
    end

Observable outcome: The reader should evaluate hardware compatibility and quality retention together, not in isolation, before committing to a pipeline.

Stage 6 — AutoGPTQ, AutoAWQ, and bitsandbytes: The Three Quantization Toolkits

• AutoGPTQ is a Python library that implements the GPTQ algorithm with a high-level API for post-training quantization and export.
• AutoAWQ is the reference Python implementation for AWQ's activation-aware quantization.
• bitsandbytes is the low-level CUDA library that powers the NF4 and INT8 loading path through HuggingFace Transformers' BitsAndBytesConfig — and is the engine behind the NF4 examples in this post.
• The choice of toolkit largely mirrors the algorithm choice from earlier sections: AutoGPTQ for controlled offline PTQ, AutoAWQ when preserving instruction-following quality is the priority, and bitsandbytes when the fas…

# --- AutoGPTQ: offline GPTQ quantization with calibration ---
# pip install auto-gptq
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer

model_name = "mistralai/Mistral-7B-Instruct-v0.2"
tokenizer  = AutoTokenizer.from_pretrained(model_name)

quant_config = BaseQuantizeConfig(
    bits=4,
    group_size=128,       # smaller group = better quality, more metadata overhead
    desc_act=False,       # set True for better quality on some backends (slower)
)

# Calibration prompts — use prompts representative of your production queries
calib_data = [
    "Explain the CAP theorem in distributed systems.",
    "What is eventual consistency and when should you use it?",
    "How does a token bucket rate limiter work?",
]

model = AutoGPTQForCausalLM.from_pretrained(model_name, quant_config)
model.quantize(calib_data)
model.save_quantized("./mistral-7b-gptq-4bit")

# --- AutoAWQ: offline AWQ quantization ---
# pip install autoawq
from awq import AutoAWQForCausalLM

awq_model = AutoAWQForCausalLM.from_pretrained(model_name)
awq_model.quantize(
    tokenizer,
    quant_config={"zero_point": True, "q_group_size": 128, "w_bit": 4},
)
awq_model.save_quantized("./mistral-7b-awq-4bit")
tokenizer.save_pretrained("./mistral-7b-awq-4bit")

Observable outcome: The choice of toolkit largely mirrors the algorithm choice from earlier sections: AutoGPTQ for controlled offline PTQ, AutoAWQ when preserving instruction-following quality is the priority, and bitsandbytes when the fas…

End-to-end outcome

After working through the stages, you should be able to explain:

• • The best method depends on your runtime constraints, not on benchmark headlines.
• • GPTQ, AWQ, and NF4 can all succeed when calibration and evaluation are realistic.

Alternatives considered

Why a Tool-Level Comparison Matters

See Types of LLM Quantization: By Timing, Scope, and Mapping for taxonomy context.

Visualizing the GPTQ vs AWQ vs NF4 Decision Flow

The key is not picking one method forever.

Decision Guide: GPTQ, AWQ, or NF4?

Benchmark AWQ and GPTQ on the same prompt/eval suite.

Example 1: Benchmark GPTQ and AWQ checkpoints with one script

These examples provide a reproducible side-by-side comparison harness for evaluating GPTQ, AWQ, and NF4 on the same prompt set, measuring latency and output character count per method.

Example 2: NF4 baseline with bitsandbytes

This is a strong baseline for comparison.

Failure and degraded modes

Trade-offs & Failure Modes: Trade-offs and Failure Modes

• Detection: Performance vs cost is never free: lower bits reduce infra cost, but only if you invest in evaluation and runtime compatibility work.
• User or operator impact: Performance vs cost is never free: lower bits reduce infra cost, but only if you invest in evaluation and runtime compatibility work.