Skip to main content
← PROBLEMS / AI-SYSTEMS
An end-to-end domain learning journey

Speculative Decoding for
Faster LLM Inference.

Design a low-latency LLM serving engine for a 70B-parameter BF16 target model (Llama-3-70B) on 8x H100 80GB GPUs (NVLink) that raises single-request decode throughput from 24 tokens/sec to >= 60 tokens/sec while guaranteeing mathematically identical output token distributions (lossless sampling).
Start the journey5 stages
Loading diagram…
Speculative Decoding for Faster LLM Inference architecture sketch diagram
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Design a low-latency LLM serving engine for a 70B-parameter BF16 target model (Llama-3-70B) on 8x H100 80GB GPUs (NVLink) that raises single-request decode throughput from 24 tokens/sec to >= 60 tokens/sec while guaranteeing mathematically identical output token distributions (lossless sampling).

From patterns to decisions.

Design a low-latency LLM serving engine for a 70B-parameter BF16 target model (Llama-3-70B) on 8x H100 80GB GPUs (NVLink) that raises single-request decode throughput from 24 tokens/sec to >= 60 tokens/sec while guaranteeing mathematically identical output token distributions (lossless sampling).

01
AI-SYSTEMS · STAGE 01L4 · FUNDAMENTALS

At batch=1, 70B decoding is.

At batch=1, 70B decoding is strictly HBM-bandwidth-bound (reading 140 GB of weights for 1 token's math). Adding 5 more tokens increases FLOPs 6x while keeping HBM weight reads constant, utilizing idle Tensor Cores.

ChoiceWhat it buysWhat it costs
Trade-off 1Only holds while batch_tokens * batch_size remains below the GPU compute ridge point (~295 tokens).

Compute HBM weight read time

17.5 GB/GPU / 3.35 TB/s ~= 5.2 ms + AllReduce overhead.

Compare arithmetic intensity

1 token = 2 FLOPs/byte vs 6 tokens = 12 FLOPs/byte (well below H100's ~295 FLOPs/byte compute ridge point).

Step 3

Conclude that K=5 adds < 10% latency to the target forward pass.

Alternative 1

INT4/FP8 weight quantization (reduces HBM reads 2x-4x, orthogonal and composable with speculative decoding).

Failure Mode 1

Running speculative decoding at batch=64 pushes batch_tokens=384 past the ridge point, causing compute-bound slowdown.

def estimate_target_step_ms(batch_tokens: int, weights_gb: float = 17.5, bw_tb_s: float = 3.35, tflops: float = 989.0) -> float:
    hbm_ms = (weights_gb / (bw_tb_s * 1000.0)) * 1000.0
    compute_ms = ((2.0 * weights_gb * 0.5e9 * batch_tokens) / (tflops * 1e12)) * 1000.0
    allreduce_ms = 35.0  # 80 layers * ~440 us NVLink AllReduce + kernel overhead
    return max(hbm_ms, compute_ms) + allreduce_ms
02
AI-SYSTEMS · STAGE 02L5 · APPLIED DESIGN

Python implementation of Leviathan et.

Python implementation of Leviathan et al. lossless speculative sampling, handling prefix acceptance, residual resampling norm(max(0, p - q)), and bonus token emission.

ChoiceWhat it buysWhat it costs
Trade-off 1Requires storing draft probability distributions q_probs[0..K-1] until verification completes.

Step 1

Iterate i from 0 to K-1 comparing target p[i, x_i] and draft q[i, x_i].

Step 2

Accept if uniform(0,1) < min(1, p/q); on first rejection, normalize max(0, p[i] - q[i]) and sample replacement.

Step 3

If all K tokens are accepted, sample bonus token from p[K].

Alternative 1

Tree-based verification (SpecInfer / Medusa) verifying multiple branching draft candidates in one pass using a tree attention mask.

Failure Mode 1

Floating-point underflow in sum(residual) when p and q are nearly identical; fallback to p_probs[i] prevents division by zero.

import random
from typing import List

def speculative_sample(
    draft_tokens: List[int],
    q_probs: List[List[float]],  # shape [K, V]
    p_probs: List[List[float]],  # shape [K+1, V]
) -> List[int]:
    accepted: List[int] = []
    for i, tok in enumerate(draft_tokens):
        p_tok = p_probs[i][tok]
        q_tok = max(q_probs[i][tok], 1e-12)
        if random.random() < min(1.0, p_tok / q_tok):
            accepted.append(tok)
        else:
            # Sample from residual distribution norm(max(0, p_i - q_i))
            residual = [max(0.0, p - q) for p, q in zip(p_probs[i], q_probs[i])]
            total = sum(residual)
            if total <= 0.0:
                residual = p_probs[i]
                total = sum(residual)
            r = random.random() * total
            cum = 0.0
            for vocab_id, prob in enumerate(residual):
                cum += prob
                if cum >= r:
                    accepted.append(vocab_id)
                    return accepted
            accepted.append(len(residual) - 1)
            return accepted
    # All K accepted -> sample bonus token from p_probs[K]
    r = random.random()
    cum = 0.0
    for vocab_id, prob in enumerate(p_probs[len(draft_tokens)]):
        cum += prob
        if cum >= r:
            accepted.append(vocab_id)
            return accepted
    accepted.append(len(p_probs[-1]) - 1)
    return accepted
03
AI-SYSTEMS · STAGE 03L6 · SCALE & TRADE-OFFS

Placing a standalone 1B draft.

Placing a standalone 1B draft model unsharded on GPU 0 avoids 8-way NVLink AllReduce overhead (saving ~15 ms per K=5 step), while EAGLE-2 heads offer higher acceptance rates if custom-trained for the target model.

ChoiceWhat it buysWhat it costs
Trade-off 1Replicating the 2 GB draft model on GPU 0 creates a minor 2 GB memory asymmetry across the 8 GPUs.

Step 1

Quantify NVLink AllReduce tax on small models: ~0.48 ms/token across 16 layers.

Step 2

Compare standalone 1B drafter (zero training, separate KV cache) vs EAGLE-2 (reuses target hidden states, higher alpha).

Step 3

Choose unsharded GPU 0 placement for standalone drafter, or EAGLE-2 when training budget is available.

Alternative 1

N-gram / prompt-lookup decoding for code editing and RAG where tokens repeat verbatim from the prompt.

Failure Mode 1

Sharding a 1B draft model across TP=8 makes draft latency 2x slower than running it on a single GPU.

# Dynamic K controller based on rolling acceptance rate alpha and active batch size
def choose_speculation_k(ema_alpha: float, active_batch_size: int) -> int:
    if active_batch_size > 24 or ema_alpha < 0.45:
        return 0
    if ema_alpha >= 0.85:
        return 7
    if ema_alpha >= 0.70:
        return 5
    return 3
04
AI-SYSTEMS · STAGE 04L6 · SCALE & TRADE-OFFS

Implementation & Guardrails (4).

PagedAttention tracks logical-to-physical block mappings per request; after verification, each request updates its accepted_len independently and frees only trailing physical blocks beyond accepted_len.

ChoiceWhat it buysWhat it costs
Trade-off 1Sub-block tail slots (up to 15 tokens) remain allocated in the last block and are simply overwritten on the next step.

Step 1

Store seq_len before drafting as base_len.

Step 2

After verification yields n_accepted tokens, set new_len = base_len + n_accepted.

Step 3

Free any physical KV blocks whose start offset >= new_len, and mask attention using ragged cu_seqlens.

Alternative 1

Contiguous pre-allocated KV buffers (wastes up to 60% HBM due to internal fragmentation).

Failure Mode 1

Failing to rewind the draft model's KV cache alongside the target KV cache causes draft quality to collapse on the next step.

def rollback_kv_cache(block_table: list[int], base_len: int, n_accepted: int, block_size: int = 16, free_pool: list[int] = None) -> int:
    new_len = base_len + n_accepted
    needed_blocks = (new_len + block_size - 1) // block_size
    while len(block_table) > needed_blocks:
        freed_block = block_table.pop()
        if free_pool is not None:
            free_pool.append(freed_block)
    return new_len
05
AI-SYSTEMS · STAGE 05L7 · STAFF / PRINCIPAL

Implementation & Guardrails (5).

Disaggregated prefill/decode isolates compute-heavy prefill nodes from latency-sensitive speculative decode nodes, while tree verification checks multiple draft branches in one target pass.

ChoiceWhat it buysWhat it costs
Trade-off 1Tree verification increases accepted tokens per step but requires custom FlashAttention tree-mask kernels.

Step 1

Route long-prompt prefill to dedicated compute-optimized nodes and transfer KV blocks over RDMA/InfiniBand.

Step 2

Replace linear K-token draft with a top-k token tree (e.g., 16-32 nodes) using a custom tree attention mask.

Step 3

Verify all tree paths in one target pass to raise expected accepted tokens per step from ~3.5 to ~5.0+.

Alternative 1

Lookahead decoding (Jacobi iteration) requiring zero draft models.

Failure Mode 1

RDMA KV-cache transfer latency exceeding 20 ms negates speculative speedup on short prompts.

# Tree attention mask: node i attends to node j iff j is an ancestor of i in the draft tree
def build_tree_mask(parent_indices: list[int]) -> list[list[int]]:
    n = len(parent_indices)
    mask = [[0] * n for _ in range(n)]
    for i in range(n):
        curr = i
        while curr != -1:
            mask[i][curr] = 1
            curr = parent_indices[curr]
    return mask

Theory & Tips and Tricks

Practical heuristics & interview traps

  • 01Interview anchor: Because autoregressive LLM decoding is HBM-bandwidth-bound rather than compute-bound, verifying K=5 draft tokens in a single batched target pass costs nearly the same latency as generating 1 target token.
  • 02Trade-off tip: Replicating the 2 GB draft model on GPU 0 creates a minor 2 GB memory asymmetry across the 8 GPUs.
  • 03Watch out for: Failing to rewind the draft model's KV cache alongside the target KV cache causes draft quality to collapse on the next step.
Behavioral Progression · Speculative Decoding Llm Inference

Speculative Decoding Llm Inference — Normal → Load / Fault → Recovery

Send Request
1. Normal Request Ingress
Traffic: 1x baseline · Latency (p99): 4 ms
Process State
2. Core Execution & State Lookup
Traffic: 1x baseline · Latency (p99): 9 ms
Inject 10x Spike
3. Scale Spike / Failure Injection
Traffic: 10x burst · Latency (p99): 185 ms

Because autoregressive LLM decoding is HBM-bandwidth-bound rather than compute-bound, verifying K=5 draft tokens in a single batched target pass costs nearly the same latency as generating 1 target token.

Synthesis / One connected design

Putting Speculative Decoding for Faster LLM Inference Together

Because autoregressive LLM decoding is HBM-bandwidth-bound rather than compute-bound, verifying K=5 draft tokens in a single batched target pass costs nearly the same latency as generating 1 target token.

A candidate architecture, not a universal answer

Putting Speculative Decoding for Faster LLM Inference Together

Step 1
Step 2
Step 3
  1. Iterate i from 0 to K-1 comparing target p[i, x_i] and draft q[i, x_i].
  2. Accept if uniform(0,1) < min(1, p/q); on first rejection, normalize max(0, p[i] - q[i]) and sample replacement.
  3. If all K tokens are accepted, sample bonus token from p[K].
PagedAttention tracks logical-to-physical block mappings per request; after verification, each request updates its `accepted_len` independently and frees only trailing physical blocks beyond `accepted_len`.
Interview Readiness Check

PagedAttention tracks logical-to-physical block mappings per request; after verification, each request updates its accepted_len independently and frees only trailing physical blocks beyond accepted_len.

Walk through bottlenecks, back-of-the-envelope math, and failure recovery.

Practice / Explain the consequences

Make the trade-off yourself.

Practice Check

Test your understanding

Question 1 of 3

When is ACID Transaction Fundamentals the right default for a production team?

Key takeaway

Speculative Decoding for Faster LLM Inference

Because autoregressive LLM decoding is HBM-bandwidth-bound rather than compute-bound, verifying K=5 draft tokens in a single batched target pass costs nearly the same latency as generating 1 target token.