Disaggregated LLM serving separates prompt prefill (compute-bound TTFT) and autoregressive token decode (HBM-bandwidth-bound ITL) onto independent GPU pools, transferring computed KV-cache blocks over RDMA or NVLink without stalling in-flight decode batches.
TL;DR30-second simplified summary▾
Core Mechanics: LLM inference splits into two physics-opposite phases: Prefill (compute-bound GEMMs over all prompt tokens, governing Time-to-First-Token TTFT) and Decode (HBM-bandwidth-bound single-token generation, governing Inter-Token Latency ITL). Colocating both on one GPU causes 16K-token prefills to freeze active decode streams.
Visual & Structural Flow: Depicts Prompt Queue -> Prefill Pool (Compute-Bound GPUs) -> RDMA / NVLink KV Transfer (KV Blocks transferred without stalling decode) -> Decode Pool (HBM-Bandwidth-Bound GPUs) running Continuous Batching (ITL < 25ms) -> SSE Client, achieving Zero Prefill-Decode Interference.
Trade-Offs & Production: Disaggregation requires high-bandwidth interconnects (InfiniBand RDMA or NVLink) so KV transfer time stays below TTFT budgets, and allows asymmetric tensor/pipeline parallelism (e.g., high TP on Prefill, high DP/batching on Decode).
Failure Modes & Application: Bound KV-transfer queues with admission backpressure and fall back to local chunked prefill if RDMA links degrade.
From patterns to decisions.
Eliminate prefill-decode interference so long RAG prompts do not spike inter-token latency (ITL) for active streaming users.
Prefill vs. Decode Physics & Disaggregated Pools.
In a standard colocated vLLM/TensorRT-LLM replica, a single scheduler interleaves prompt prefill and autoregressive decode on the same GPUs. Prefill processes N prompt tokens in parallel (O(N) matrix-matrix multiplications), saturating Tensor Core FLOPs. Decode generates 1 token per sequence per step, streaming model weights and KV-cache from HBM (HBM-bandwidth-bound).
Disaggregated serving (used in DistServe, Splitwise, and Mooncake) assigns dedicated Prefill GPUs to compute the initial KV-cache and first token, streams the KV-cache blocks asynchronously over RDMA / NVLink (mooncake / NIXL), and hands the sequence to Decode GPUs optimized for large continuous batches.
Prefill Pool -> RDMA KV Transfer -> Decode Pool
1.Zero Interference: Long 32K prefills never preempt or stall active decode iterations.
2.Asymmetric Tuning: Prefill uses high Tensor Parallelism (TP); Decode scales batch size + FP8 KV.
3.KV Transfer Math: Transfer bytes = 2 * layers * kv_heads * head_dim * seq_len * dtype_bytes.
Caption: Separating Prefill and Decode lets each pool optimize for its side of the GPU Roofline while RDMA transfers KV blocks asynchronously.
Why Colocation Causes Tail-Latency Collapse.
Even with chunked prefill, colocating 32K-token RAG prompts with 100-token chat streams forces a painful tug-of-war: smaller prefill chunks protect decode ITL (<25 ms) at the cost of multi-second TTFT, while larger prefill chunks violate ITL SLOs for every user currently streaming tokens on that GPU.
Goodput vs. Raw Throughput
Measuring requests/sec that satisfy BOTH TTFT and ITL SLOs reveals colocated GPUs lose up to 40% goodput under mixed prompt lengths.
Network Bandwidth Requirement
A 70B GQA-8 model in FP8 generates ~80 KB of KV-cache per token; a 10K-token prompt transfers ~800 MB (~2 ms on 400 Gbps RDMA, ~64 ms on 100 Gbps Ethernet).
When to Disaggregate (and When Colocated Chunked Prefill Wins).
Adopt disaggregated serving when serving long-context RAG, coding assistants, or agentic workloads across multi-node InfiniBand/RoCEv2 clusters with strict TTFT + ITL SLAs. For small single-node 8B deployments or <1K-token prompts, colocated chunked prefill avoids network KV-transfer complexity.
KV-Cache Transfer Budgeting, Layer-Wise Pipelining & Fallback.
Overlap layer-wise KV transfer (NIXL / GPUDirect RDMA) so Layer i's KV blocks begin transferring while Prefill computes Layer i+1. Pair with prefix caching on the Prefill pool so shared system prompts are neither recomputed nor re-transferred when the Decode pool already holds those KV block hashes.
def estimate_kv_transfer_ms(layers: int, kv_heads: int, head_dim: int, seq_len: int, bytes_per_elem: int, link_gbps: float) -> float:
"""Compute theoretical KV-cache transfer time between Prefill and Decode pools."""
if link_gbps <= 0:
raise ValueError("Interconnect bandwidth must be positive")
total_bytes = 2 * layers * kv_heads * head_dim * seq_len * bytes_per_elem
bytes_per_sec = (link_gbps * 1e9) / 8.0
return (total_bytes / bytes_per_sec) * 1000.0
Colocated Chunked Prefill vs. Disaggregated Serving.
Colocated chunked prefill minimizes hardware fragmentation on small clusters, whereas disaggregation maximizes SLA goodput and lets you autoscale Prefill GPUs separately from Decode GPUs.
| Choice | What it buys | What it costs |
|---|---|---|
| Colocated Chunked Prefill | Zero network KV-transfer overhead; works over standard Ethernet. | Prefill and decode compete for HBM bandwidth and SM compute; mixed workloads degrade goodput. |
| Disaggregated Prefill / Decode | Zero prefill-decode interference; independent TP/batching and autoscaling per phase. | Requires high-speed RDMA/NVLink and distributed KV-block registry coordination. |
KV-Aware Router + Mooncake/NIXL Disaggregated Pool.
A KV-aware gateway router hashes the prompt prefix to pick a Prefill worker with warm prefix blocks, streams newly computed KV blocks via GPUDirect RDMA to an available Decode worker with free PagedAttention slots, and fails over to local chunked prefill if RDMA congestion exceeds threshold.
Disaggregated Request & KV Handoff States
Caption: Once Prefill computes the initial KV blocks and first token, RDMA pins them on a Decode worker for uninterrupted token streaming.
Related Inference & KV-Cache Problems.
Dive deeper into virtual KV-cache paging and draft-verify speculative decoding.
Practice: Make the trade-off yourself.
Why does colocating a 32K-token RAG prompt prefill with active chat decode streams spike Inter-Token Latency (ITL)?