Design a multi-tenant LLM inference routing and serving architecture across 64 H100 GPUs handling 2,000 req/sec where 80% of requests carry 4K–16K token system prompts + RAG context, targeting < 250 ms p99 TTFT, > 85% prefix cache hit rate, strict tenant isolation, and graceful degradation under GPU pod overload.
From patterns to decisions.
Route requests via a Prefix-Aware Consistent-Hash Router keyed by `(tenant_id, prefix_block_hash)` to GPU Worker Pods running local RadixAttention + CPU-DRAM L2 KV offload.
Why Prefix Caching Is Mandatory at 2,000 Req/Sec.
Assume a 70B-parameter model (GQA, ~320 KB KV/token in FP16, or ~160 KB in FP8). An 8K-token prefix occupies 8,192 * 160 KB ≈ 1.3 GB per copy.
With 80% prefix sharing, RadixAttention stores that 1.3 GB once per active tenant/prompt instead of duplicating it per concurrent request, freeing 40+ GB HBM per GPU for higher batch sizes and cutting prefill compute by ~85%.
HBM & Compute Savings Math
1.Stage Objective: Quantify prefill FLOPs and HBM savings for 8K-token shared prefixes across 64 H100s.
2.Level (Foundation): Assume a 70B-parameter model (GQA, ~320 KB KV/token in FP16, or ~160 KB in FP8). An 8K-token prefix occupies `8,192 * 160 KB ≈ 1.3 GB` per c
3.Operational Invariant: Zero-copy prefix sharing cuts per-request KV footprint and prefill FLOPs by ~85%.
Caption: Zero-copy prefix sharing cuts per-request KV footprint and prefill FLOPs by ~85%.
Consistent Hashing with Bounded Loads on `(tenant_id, prefix_hash)`.
At the API gateway/router, tokenize (or hash normalized UTF-8 chunks) of the leading K=512 tokens combined with tenant_id. Look up the target GPU pod on a consistent-hash ring.
Before forwarding, check the pod's real-time active_prefill_tokens telemetry: if overloaded (> 1.25x cluster average), walk clockwise on the ring to the secondary replica.
Bounded-Load Prefix Ring
1.Stage Objective: Route matching prefixes to the same GPU pod without creating hot-spot queuing delays.
2.Level (Foundation): At the API gateway/router, tokenize (or hash normalized UTF-8 chunks) of the leading `K=512` tokens combined with `tenant_id`. Look up the t
3.Operational Invariant: Consistent hashing preserves cache affinity while bounded-load spillover prevents hot-GPU queuing.
Caption: Consistent hashing preserves cache affinity while bounded-load spillover prevents hot-GPU queuing.
Implementing Tenant-Isolated Prefix-Aware Routing.
import hashlib
class PrefixAwareRouter:
def __init__(self, ring, max_load_factor: float = 1.25) -> None:
self.ring = ring
self.max_load_factor = max_load_factor
def select_pod(self, tenant_id: str, prefix_bytes: bytes, pod_loads: dict[str, float], avg_load: float) -> str:
if not tenant_id:
raise ValueError('tenant_id is required for cache isolation')
salted_key = hashlib.blake2b(
tenant_id.encode('utf-8') + b':' + prefix_bytes[:2048],
digest_size=16,
).hexdigest()
candidates = self.ring.get_candidates(salted_key, count=3)
if not candidates:
raise RuntimeError('No healthy GPU worker pods available')
cap = max(1.0, avg_load * self.max_load_factor)
for pod_id in candidates:
if pod_loads.get(pod_id, 0.0) <= cap:
return pod_id
# All candidates busy: pick least-loaded among candidates to bound TTFT
return min(candidates, key=lambda p: pod_loads.get(p, 0.0))Tenant-Salted Candidate Selection
1.Stage Objective: Build the deterministic prefix-hash router with bounded-load fallback and safe error handling.
2.Level (Intermediate): import hashlib
3.Operational Invariant: Router hashes tenant_id + prefix bytes and spills to least-loaded replica when primary exceeds cap.
Caption: Router hashes tenant_id + prefix bytes and spills to least-loaded replica when primary exceeds cap.
GPU HBM Radix Tree + Async CPU DRAM L2 Cache.
Each GPU worker runs a local Radix Tree over PagedAttention blocks. When HBM utilization crosses 85%, leaf nodes with ref_count == 0 offload asynchronously over PCIe Gen5 (~64 GB/s) to a 512 GB CPU-DRAM L2 pool.
Reloading an 8K prefix (1.3 GB) from CPU DRAM takes ~20 ms—over 10x faster than cold GPU prefill.
2-Tier HBM + CPU-DRAM KV Pool
1.Stage Objective: Retain warm tenant prefixes across bursty traffic without exhausting 80 GB HBM.
2.Level (Intermediate): Each GPU worker runs a local Radix Tree over PagedAttention blocks. When HBM utilization crosses 85%, leaf nodes with `ref_count == 0` offlo
3.Operational Invariant: Async PCIe offload to 512 GB CPU DRAM retains warm tenant prefixes during HBM pressure.
Caption: Async PCIe offload to 512 GB CPU DRAM retains warm tenant prefixes during HBM pressure.
Integrating RadixAttention with Disaggregated Prefill/Decode Pods.
At 2,000 req/sec, mixing 16K-token prefills with active decode batches causes inter-token latency (ITL) spikes. Route cache misses and partial suffix prefills to dedicated Prefill Pods (which own the Radix Tree), then RDMA/NVLink-transfer only the final KV blocks to Decode Pods for smooth token streaming.
Prefill-Owned Radix + Decode RDMA
1.Stage Objective: Prevent long prefills from stalling streaming decode tokens on the same GPU.
2.Level (Advanced): At 2,000 req/sec, mixing 16K-token prefills with active decode batches causes inter-token latency (ITL) spikes. Route cache misses and parti
3.Operational Invariant: Separating Prefill (Radix Tree) from Decode (RDMA-fed) eliminates decode stalls during prefill bursts.
Caption: Separating Prefill (Radix Tree) from Decode (RDMA-fed) eliminates decode stalls during prefill bursts.
Enforcing Tenant Quotas & Zero Cross-Tenant Leakage.
Salt every Radix Tree root with (tenant_id, model_version, lora_id) so cross-tenant hits are impossible. Pair global LRU with per-tenant HBM soft caps (e.g., max 25% of HBM per tenant) so a single tenant's 16K-token sweep cannot thrash other tenants' warm caches.
Per-Tenant HBM Quota & Salting
1.Stage Objective: Prevent one noisy tenant from evicting all other tenants' prefixes or probing TTFT.
2.Level (Advanced): Salt every Radix Tree root with `(tenant_id, model_version, lora_id)` so cross-tenant hits are impossible. Pair global LRU with **per-tenant
3.Operational Invariant: Tenant-salted roots and per-tenant HBM caps block both timing leaks and noisy-neighbor eviction.
Caption: Tenant-salted roots and per-tenant HBM caps block both timing leaks and noisy-neighbor eviction.
Scaling on Prefix Hit Rate, TTFT & KV Eviction Churn.
Monitor prefix_cache_hit_ratio, ttft_p99_ms, hbm_to_dram_offload_rate, and bounded_load_spill_rate. Use consistent-hash virtual-node draining during scale-in so only 1/N keys remap, and pre-warm newly added pods from a shared NIXL/RDMA KV store when traffic ramps.
Locality-Preserving Autoscaling
1.Stage Objective: Autoscale GPU pods without shattering prefix cache locality.
2.Level (Advanced): Monitor `prefix_cache_hit_ratio`, `ttft_p99_ms`, `hbm_to_dram_offload_rate`, and `bounded_load_spill_rate`. Use **consistent-hash virtual-no
3.Operational Invariant: Consistent-hash virtual nodes and KV pre-warming keep hit rates > 85% across autoscaling events.
Caption: Consistent-hash virtual nodes and KV pre-warming keep hit rates > 85% across autoscaling events.
Practice: Make the trade-off yourself.
Why does standard round-robin load balancing ruin RadixAttention prefix cache hit rates across 64 GPUs?