Design a low-latency LLM serving engine for a 70B-parameter BF16 target model (Llama-3-70B) on 8x H100 80GB GPUs (NVLink) that raises single-request decode throughput from 24 tokens/sec to >= 60 tokens/sec while guaranteeing mathematically identical output token distributions (lossless sampling).
From patterns to decisions.
Design a low-latency LLM serving engine for a 70B-parameter BF16 target model (Llama-3-70B) on 8x H100 80GB GPUs (NVLink) that raises single-request decode throughput from 24 tokens/sec to >= 60 tokens/sec while guaranteeing mathematically identical output token distributions (lossless sampling).
At batch=1, 70B decoding is.
At batch=1, 70B decoding is strictly HBM-bandwidth-bound (reading 140 GB of weights for 1 token's math). Adding 5 more tokens increases FLOPs 6x while keeping HBM weight reads constant, utilizing idle Tensor Cores.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Only holds while batch_tokens * batch_size remains below the GPU compute ridge point (~295 tokens). |
Compute HBM weight read time
17.5 GB/GPU / 3.35 TB/s ~= 5.2 ms + AllReduce overhead.
Compare arithmetic intensity
1 token = 2 FLOPs/byte vs 6 tokens = 12 FLOPs/byte (well below H100's ~295 FLOPs/byte compute ridge point).
Step 3
Conclude that K=5 adds < 10% latency to the target forward pass.
Alternative 1
INT4/FP8 weight quantization (reduces HBM reads 2x-4x, orthogonal and composable with speculative decoding).
Failure Mode 1
Running speculative decoding at batch=64 pushes batch_tokens=384 past the ridge point, causing compute-bound slowdown.
def estimate_target_step_ms(batch_tokens: int, weights_gb: float = 17.5, bw_tb_s: float = 3.35, tflops: float = 989.0) -> float:
hbm_ms = (weights_gb / (bw_tb_s * 1000.0)) * 1000.0
compute_ms = ((2.0 * weights_gb * 0.5e9 * batch_tokens) / (tflops * 1e12)) * 1000.0
allreduce_ms = 35.0 # 80 layers * ~440 us NVLink AllReduce + kernel overhead
return max(hbm_ms, compute_ms) + allreduce_ms
Python implementation of Leviathan et.
Python implementation of Leviathan et al. lossless speculative sampling, handling prefix acceptance, residual resampling norm(max(0, p - q)), and bonus token emission.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Requires storing draft probability distributions q_probs[0..K-1] until verification completes. |
Step 1
Iterate i from 0 to K-1 comparing target p[i, x_i] and draft q[i, x_i].
Step 2
Accept if uniform(0,1) < min(1, p/q); on first rejection, normalize max(0, p[i] - q[i]) and sample replacement.
Step 3
If all K tokens are accepted, sample bonus token from p[K].
Alternative 1
Tree-based verification (SpecInfer / Medusa) verifying multiple branching draft candidates in one pass using a tree attention mask.
Failure Mode 1
Floating-point underflow in sum(residual) when p and q are nearly identical; fallback to p_probs[i] prevents division by zero.
import random
from typing import List
def speculative_sample(
draft_tokens: List[int],
q_probs: List[List[float]], # shape [K, V]
p_probs: List[List[float]], # shape [K+1, V]
) -> List[int]:
accepted: List[int] = []
for i, tok in enumerate(draft_tokens):
p_tok = p_probs[i][tok]
q_tok = max(q_probs[i][tok], 1e-12)
if random.random() < min(1.0, p_tok / q_tok):
accepted.append(tok)
else:
# Sample from residual distribution norm(max(0, p_i - q_i))
residual = [max(0.0, p - q) for p, q in zip(p_probs[i], q_probs[i])]
total = sum(residual)
if total <= 0.0:
residual = p_probs[i]
total = sum(residual)
r = random.random() * total
cum = 0.0
for vocab_id, prob in enumerate(residual):
cum += prob
if cum >= r:
accepted.append(vocab_id)
return accepted
accepted.append(len(residual) - 1)
return accepted
# All K accepted -> sample bonus token from p_probs[K]
r = random.random()
cum = 0.0
for vocab_id, prob in enumerate(p_probs[len(draft_tokens)]):
cum += prob
if cum >= r:
accepted.append(vocab_id)
return accepted
accepted.append(len(p_probs[-1]) - 1)
return accepted
Placing a standalone 1B draft.
Placing a standalone 1B draft model unsharded on GPU 0 avoids 8-way NVLink AllReduce overhead (saving ~15 ms per K=5 step), while EAGLE-2 heads offer higher acceptance rates if custom-trained for the target model.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Replicating the 2 GB draft model on GPU 0 creates a minor 2 GB memory asymmetry across the 8 GPUs. |
Step 1
Quantify NVLink AllReduce tax on small models: ~0.48 ms/token across 16 layers.
Step 2
Compare standalone 1B drafter (zero training, separate KV cache) vs EAGLE-2 (reuses target hidden states, higher alpha).
Step 3
Choose unsharded GPU 0 placement for standalone drafter, or EAGLE-2 when training budget is available.
Alternative 1
N-gram / prompt-lookup decoding for code editing and RAG where tokens repeat verbatim from the prompt.
Failure Mode 1
Sharding a 1B draft model across TP=8 makes draft latency 2x slower than running it on a single GPU.
# Dynamic K controller based on rolling acceptance rate alpha and active batch size
def choose_speculation_k(ema_alpha: float, active_batch_size: int) -> int:
if active_batch_size > 24 or ema_alpha < 0.45:
return 0
if ema_alpha >= 0.85:
return 7
if ema_alpha >= 0.70:
return 5
return 3
Implementation & Guardrails (4).
PagedAttention tracks logical-to-physical block mappings per request; after verification, each request updates its accepted_len independently and frees only trailing physical blocks beyond accepted_len.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Sub-block tail slots (up to 15 tokens) remain allocated in the last block and are simply overwritten on the next step. |
Step 1
Store seq_len before drafting as base_len.
Step 2
After verification yields n_accepted tokens, set new_len = base_len + n_accepted.
Step 3
Free any physical KV blocks whose start offset >= new_len, and mask attention using ragged cu_seqlens.
Alternative 1
Contiguous pre-allocated KV buffers (wastes up to 60% HBM due to internal fragmentation).
Failure Mode 1
Failing to rewind the draft model's KV cache alongside the target KV cache causes draft quality to collapse on the next step.
def rollback_kv_cache(block_table: list[int], base_len: int, n_accepted: int, block_size: int = 16, free_pool: list[int] = None) -> int:
new_len = base_len + n_accepted
needed_blocks = (new_len + block_size - 1) // block_size
while len(block_table) > needed_blocks:
freed_block = block_table.pop()
if free_pool is not None:
free_pool.append(freed_block)
return new_len
Implementation & Guardrails (5).
Disaggregated prefill/decode isolates compute-heavy prefill nodes from latency-sensitive speculative decode nodes, while tree verification checks multiple draft branches in one target pass.
| Choice | What it buys | What it costs |
|---|---|---|
| Trade-off 1 | Tree verification increases accepted tokens per step but requires custom FlashAttention tree-mask kernels. |
Step 1
Route long-prompt prefill to dedicated compute-optimized nodes and transfer KV blocks over RDMA/InfiniBand.
Step 2
Replace linear K-token draft with a top-k token tree (e.g., 16-32 nodes) using a custom tree attention mask.
Step 3
Verify all tree paths in one target pass to raise expected accepted tokens per step from ~3.5 to ~5.0+.
Alternative 1
Lookahead decoding (Jacobi iteration) requiring zero draft models.
Failure Mode 1
RDMA KV-cache transfer latency exceeding 20 ms negates speculative speedup on short prompts.
# Tree attention mask: node i attends to node j iff j is an ancestor of i in the draft tree
def build_tree_mask(parent_indices: list[int]) -> list[list[int]]:
n = len(parent_indices)
mask = [[0] * n for _ in range(n)]
for i in range(n):
curr = i
while curr != -1:
mask[i][curr] = 1
curr = parent_indices[curr]
return mask
Theory & Tips and Tricks
Practical heuristics & interview traps
- 01Interview anchor: Because autoregressive LLM decoding is HBM-bandwidth-bound rather than compute-bound, verifying K=5 draft tokens in a single batched target pass costs nearly the same latency as generating 1 target token.
- 02Trade-off tip: Replicating the 2 GB draft model on GPU 0 creates a minor 2 GB memory asymmetry across the 8 GPUs.
- 03Watch out for: Failing to rewind the draft model's KV cache alongside the target KV cache causes draft quality to collapse on the next step.
Speculative Decoding Llm Inference — Normal → Load / Fault → Recovery
Because autoregressive LLM decoding is HBM-bandwidth-bound rather than compute-bound, verifying K=5 draft tokens in a single batched target pass costs nearly the same latency as generating 1 target token.
Putting Speculative Decoding for Faster LLM Inference Together
Because autoregressive LLM decoding is HBM-bandwidth-bound rather than compute-bound, verifying K=5 draft tokens in a single batched target pass costs nearly the same latency as generating 1 target token.
Putting Speculative Decoding for Faster LLM Inference Together
- Iterate i from 0 to K-1 comparing target p[i, x_i] and draft q[i, x_i].
- Accept if uniform(0,1) < min(1, p/q); on first rejection, normalize max(0, p[i] - q[i]) and sample replacement.
- If all K tokens are accepted, sample bonus token from p[K].
Interview Readiness Check
PagedAttention tracks logical-to-physical block mappings per request; after verification, each request updates its accepted_len independently and frees only trailing physical blocks beyond accepted_len.
Walk through bottlenecks, back-of-the-envelope math, and failure recovery.
Make the trade-off yourself.
Practice Check
Test your understanding
When is ACID Transaction Fundamentals the right default for a production team?
Speculative Decoding for Faster LLM Inference
Because autoregressive LLM decoding is HBM-bandwidth-bound rather than compute-bound, verifying K=5 draft tokens in a single batched target pass costs nearly the same latency as generating 1 target token.
