Skip to main content
← PROBLEMS / AI-SYSTEMS
An end-to-end domain learning journey

Design a Multi-Tenant LLM Inference Router
with Zero-Copy RadixAttention Prefix Caching.

Route requests via a Prefix-Aware Consistent-Hash Router keyed by (tenant_id, prefix_block_hash) to GPU Worker Pods running local RadixAttention + CPU-DRAM L2 KV offload.
Start the journey7 stages
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Design a multi-tenant LLM inference routing and serving architecture across 64 H100 GPUs handling 2,000 req/sec where 80% of requests carry 4K–16K token system prompts + RAG context, targeting < 250 ms p99 TTFT, > 85% prefix cache hit rate, strict tenant isolation, and graceful degradation under GPU pod overload.

From patterns to decisions.

Route requests via a Prefix-Aware Consistent-Hash Router keyed by `(tenant_id, prefix_block_hash)` to GPU Worker Pods running local RadixAttention + CPU-DRAM L2 KV offload.

Full walkthrough · 7 stages
01
STAGE 01 · COMPUTE & HBM MATHfoundation

Why Prefix Caching Is Mandatory at 2,000 Req/Sec.

Assume a 70B-parameter model (GQA, ~320 KB KV/token in FP16, or ~160 KB in FP8). An 8K-token prefix occupies 8,192 * 160 KB ≈ 1.3 GB per copy.

With 80% prefix sharing, RadixAttention stores that 1.3 GB once per active tenant/prompt instead of duplicating it per concurrent request, freeing 40+ GB HBM per GPU for higher batch sizes and cutting prefill compute by ~85%.

STAGE 01 · COMPUTE & HBM MATH

HBM & Compute Savings Math

HBMIngress / BaselineCompute Savings MathScale ConstraintValidation GateBottleneck SignalProduction OutcomeTarget SLAexposes

1.Stage Objective: Quantify prefill FLOPs and HBM savings for 8K-token shared prefixes across 64 H100s.

2.Level (Foundation): Assume a 70B-parameter model (GQA, ~320 KB KV/token in FP16, or ~160 KB in FP8). An 8K-token prefix occupies `8,192 * 160 KB ≈ 1.3 GB` per c

3.Operational Invariant: Zero-copy prefix sharing cuts per-request KV footprint and prefill FLOPs by ~85%.

Caption: Zero-copy prefix sharing cuts per-request KV footprint and prefill FLOPs by ~85%.

02
STAGE 02 · PREFIX-AWARE ROUTERfoundation

Consistent Hashing with Bounded Loads on `(tenant_id, prefix_hash)`.

At the API gateway/router, tokenize (or hash normalized UTF-8 chunks) of the leading K=512 tokens combined with tenant_id. Look up the target GPU pod on a consistent-hash ring.

Before forwarding, check the pod's real-time active_prefill_tokens telemetry: if overloaded (> 1.25x cluster average), walk clockwise on the ring to the secondary replica.

STAGE 02 · PREFIX-AWARE ROUTER

Bounded-Load Prefix Ring

Bounded-Load PrefixRingInput StateCore EngineCore MechanismGuard & VerifyInvariant CheckProduction OutcomeOutput Statetransforms

1.Stage Objective: Route matching prefixes to the same GPU pod without creating hot-spot queuing delays.

2.Level (Foundation): At the API gateway/router, tokenize (or hash normalized UTF-8 chunks) of the leading `K=512` tokens combined with `tenant_id`. Look up the t

3.Operational Invariant: Consistent hashing preserves cache affinity while bounded-load spillover prevents hot-GPU queuing.

Caption: Consistent hashing preserves cache affinity while bounded-load spillover prevents hot-GPU queuing.

03
STAGE 03 · ROUTER LOGICintermediate

Implementing Tenant-Isolated Prefix-Aware Routing.

import hashlib

class PrefixAwareRouter:
    def __init__(self, ring, max_load_factor: float = 1.25) -> None:
        self.ring = ring
        self.max_load_factor = max_load_factor

    def select_pod(self, tenant_id: str, prefix_bytes: bytes, pod_loads: dict[str, float], avg_load: float) -> str:
        if not tenant_id:
            raise ValueError('tenant_id is required for cache isolation')
        salted_key = hashlib.blake2b(
            tenant_id.encode('utf-8') + b':' + prefix_bytes[:2048],
            digest_size=16,
        ).hexdigest()
        candidates = self.ring.get_candidates(salted_key, count=3)
        if not candidates:
            raise RuntimeError('No healthy GPU worker pods available')
        cap = max(1.0, avg_load * self.max_load_factor)
        for pod_id in candidates:
            if pod_loads.get(pod_id, 0.0) <= cap:
                return pod_id
        # All candidates busy: pick least-loaded among candidates to bound TTFT
        return min(candidates, key=lambda p: pod_loads.get(p, 0.0))
STAGE 03 · ROUTER LOGIC

Tenant-Salted Candidate Selection

Tenant-SaltedCandidateValidated InputCore EngineAtomic StepGuard & VerifyError / Retry GuardProduction OutcomeCommitted Resultexecutes

1.Stage Objective: Build the deterministic prefix-hash router with bounded-load fallback and safe error handling.

2.Level (Intermediate): import hashlib

3.Operational Invariant: Router hashes tenant_id + prefix bytes and spills to least-loaded replica when primary exceeds cap.

Caption: Router hashes tenant_id + prefix bytes and spills to least-loaded replica when primary exceeds cap.

04
STAGE 04 · WORKER KV TIERINGintermediate

GPU HBM Radix Tree + Async CPU DRAM L2 Cache.

Each GPU worker runs a local Radix Tree over PagedAttention blocks. When HBM utilization crosses 85%, leaf nodes with ref_count == 0 offload asynchronously over PCIe Gen5 (~64 GB/s) to a 512 GB CPU-DRAM L2 pool.

Reloading an 8K prefix (1.3 GB) from CPU DRAM takes ~20 ms—over 10x faster than cold GPU prefill.

STAGE 04 · WORKER KV TIERING

2-Tier HBM + CPU-DRAM KV Pool

2-Tier HBM +CPU-DRAM KVHot PathCore EnginePartition / IndexGuard & VerifyLatency / RAM BoundProduction OutcomeScaled Statebounds

1.Stage Objective: Retain warm tenant prefixes across bursty traffic without exhausting 80 GB HBM.

2.Level (Intermediate): Each GPU worker runs a local Radix Tree over PagedAttention blocks. When HBM utilization crosses 85%, leaf nodes with `ref_count == 0` offlo

3.Operational Invariant: Async PCIe offload to 512 GB CPU DRAM retains warm tenant prefixes during HBM pressure.

Caption: Async PCIe offload to 512 GB CPU DRAM retains warm tenant prefixes during HBM pressure.

05
STAGE 05 · PREFILL-DECODE SPLITadvanced

Integrating RadixAttention with Disaggregated Prefill/Decode Pods.

At 2,000 req/sec, mixing 16K-token prefills with active decode batches causes inter-token latency (ITL) spikes. Route cache misses and partial suffix prefills to dedicated Prefill Pods (which own the Radix Tree), then RDMA/NVLink-transfer only the final KV blocks to Decode Pods for smooth token streaming.

STAGE 05 · PREFILL-DECODE SPLIT

Prefill-Owned Radix + Decode RDMA

Prefill-Owned Radix+ DeFault TriggerCore EngineIsolation BoundaryGuard & VerifyFallback / QuarantineProduction OutcomeProtected Pathisolates

1.Stage Objective: Prevent long prefills from stalling streaming decode tokens on the same GPU.

2.Level (Advanced): At 2,000 req/sec, mixing 16K-token prefills with active decode batches causes inter-token latency (ITL) spikes. Route cache misses and parti

3.Operational Invariant: Separating Prefill (Radix Tree) from Decode (RDMA-fed) eliminates decode stalls during prefill bursts.

Caption: Separating Prefill (Radix Tree) from Decode (RDMA-fed) eliminates decode stalls during prefill bursts.

06
STAGE 06 · TENANT ISOLATIONadvanced

Enforcing Tenant Quotas & Zero Cross-Tenant Leakage.

Salt every Radix Tree root with (tenant_id, model_version, lora_id) so cross-tenant hits are impossible. Pair global LRU with per-tenant HBM soft caps (e.g., max 25% of HBM per tenant) so a single tenant's 16K-token sweep cannot thrash other tenants' warm caches.

STAGE 06 · TENANT ISOLATION

Per-Tenant HBM Quota & Salting

Per-Tenant HBMQuotaCheckpoint / LogSaltingReplay / SyncValidation GateConsistency GateProduction OutcomeRecovered Stateverifies

1.Stage Objective: Prevent one noisy tenant from evicting all other tenants' prefixes or probing TTFT.

2.Level (Advanced): Salt every Radix Tree root with `(tenant_id, model_version, lora_id)` so cross-tenant hits are impossible. Pair global LRU with **per-tenant

3.Operational Invariant: Tenant-salted roots and per-tenant HBM caps block both timing leaks and noisy-neighbor eviction.

Caption: Tenant-salted roots and per-tenant HBM caps block both timing leaks and noisy-neighbor eviction.

07
STAGE 07 · TELEMETRY & SCALINGadvanced

Scaling on Prefix Hit Rate, TTFT & KV Eviction Churn.

Monitor prefix_cache_hit_ratio, ttft_p99_ms, hbm_to_dram_offload_rate, and bounded_load_spill_rate. Use consistent-hash virtual-node draining during scale-in so only 1/N keys remap, and pre-warm newly added pods from a shared NIXL/RDMA KV store when traffic ramps.

STAGE 07 · TELEMETRY & SCALING

Locality-Preserving Autoscaling

Locality-PreservingAutoTelemetry SignalCore EngineSLO ThresholdGuard & VerifyCircuit BreakerProduction OutcomeSafe Degradationtrips

1.Stage Objective: Autoscale GPU pods without shattering prefix cache locality.

2.Level (Advanced): Monitor `prefix_cache_hit_ratio`, `ttft_p99_ms`, `hbm_to_dram_offload_rate`, and `bounded_load_spill_rate`. Use **consistent-hash virtual-no

3.Operational Invariant: Consistent-hash virtual nodes and KV pre-warming keep hit rates > 85% across autoscaling events.

Caption: Consistent-hash virtual nodes and KV pre-warming keep hit rates > 85% across autoscaling events.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Question 1 of 3
Score: 0/3

Why does standard round-robin load balancing ruin RadixAttention prefix cache hit rates across 64 GPUs?

Recommended Learning Flow