Skip to main content
← PROBLEMS / AI-SYSTEMS
An end-to-end domain learning journey

Design a Distributed Multimodal Generative
Media & Diffusion Inference Platform.

Use a Two-Tier Priority Queue (Redis ZSET interactive + Kafka spot batch) feeding 8-GPU NVLink Ring Attention DiT pods with TinyVAE SSE previews, Dual-Gate CLIP/ViT safety, and HSM-signed C2PA provenance.
Start the journey7 stages
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Design a multimodal image and 1080p video diffusion platform serving 50M+ generations/day (45M images, 5M 1080p video clips; 1,736 peak QPS) across 2,000+ H100/B200 GPUs with <= 4.5s image p95, <= 45s 1080p video latency, progressive live previews, dual-gate safety filtering, and cryptographic C2PA provenance.

From patterns to decisions.

Use a Two-Tier Priority Queue (Redis ZSET interactive + Kafka spot batch) feeding 8-GPU NVLink Ring Attention DiT pods with TinyVAE SSE previews, Dual-Gate CLIP/ViT safety, and HSM-signed C2PA provenance.

Full walkthrough · 7 stages
01
STAGE 01 · CAPACITY & FLEET MATHfoundation

Sizing 50M Generations/Day & 202.5 TB/Day Media Egress.

Assume 45M images/day (~1.5s on 1 H100) + 5M 1080p videos/day (~35s on 8x H100 Ring) with 3x peak-to-average ratio (1,736 peak QPS).

  • Image Compute: ~780 GPU-days/day.
  • Video Compute: ~16,200 GPU-days/day (across 8-GPU rings).
  • Storage Ingress: ~202.5 TB/day written to S3 with CDN edge caching.
STAGE 01 · CAPACITY & FLEET MATH

Fleet & Bandwidth Sizing

FleetIngress / BaselineBandwidth SizingScale ConstraintValidation GateBottleneck SignalProduction OutcomeTarget SLAexposes

1.Stage Objective: Compute GPU fleet size, queue throughput, and S3/CDN bandwidth for 45M images + 5M videos/day.

2.Level (Foundation): Assume 45M images/day (~1.5s on 1 H100) + 5M 1080p videos/day (~35s on 8x H100 Ring) with 3x peak-to-average ratio (1,736 peak QPS).

3.Operational Invariant: Video generation dominates GPU-hours (> 95%), justifying spot-pool isolation from interactive image pods.

Caption: Video generation dominates GPU-hours (> 95%), justifying spot-pool isolation from interactive image pods.

02
STAGE 02 · PRIORITY SCHEDULINGfoundation

Redis Priority ZSET (Interactive) vs. Kafka Spot Queue (Batch).

Interactive user requests enter a Redis Priority ZSET scored by (tier_weight, enqueue_ts) and dispatch strictly to reserved on-demand GPUs. Bulk API and long video jobs flow into Kafka topics consumed by spot GPU workers.

If interactive queue depth spikes, reserved pods never accept batch work.

STAGE 02 · PRIORITY SCHEDULING

Two-Tier Queue Isolation

Two-Tier QueueIsolationInput StateCore EngineCore MechanismGuard & VerifyInvariant CheckProduction OutcomeOutput Statetransforms

1.Stage Objective: Eliminate head-of-line blocking between 4.5s image requests and 45s video jobs.

2.Level (Foundation): Interactive user requests enter a **Redis Priority ZSET** scored by `(tier_weight, enqueue_ts)` and dispatch strictly to reserved on-demand

3.Operational Invariant: Separating interactive Redis ZSETs from Kafka spot queues protects the <= 4.5s image p95 SLA.

Caption: Separating interactive Redis ZSETs from Kafka spot queues protects the <= 4.5s image p95 SLA.

03
STAGE 03 · PREEMPTION-SAFE WORKERintermediate

Implementing Spot-Resilient 2-Step Latent Checkpointing.

class DiffusionJobRunner:
    def __init__(self, checkpoint_store, dit_engine, total_steps: int = 8) -> None:
        self.store = checkpoint_store
        self.dit = dit_engine
        self.total_steps = total_steps

    def run_job(self, job_id: str, prompt_emb) -> bytes:
        state = self.store.load_latest(job_id)
        start_step = state.step if state else 0
        latent = state.latent if state else self.dit.sample_noise(job_id)
        for step in range(start_step, self.total_steps):
            latent = self.dit.denoise_step(latent, prompt_emb, step)
            if (step + 1) % 2 == 0 and (step + 1) < self.total_steps:
                self.store.save_async(job_id, step + 1, latent)
        return self.dit.vae_decode(latent)
STAGE 03 · PREEMPTION-SAFE WORKER

2-Step Latent Checkpoint Loop

2-Step LatentCheckpointValidated InputCore EngineAtomic StepGuard & VerifyError / Retry GuardProduction OutcomeCommitted Resultexecutes

1.Stage Objective: Resume preempted video jobs from the latest 2-step latent checkpoint instead of restarting.

2.Level (Intermediate): class DiffusionJobRunner:

3.Operational Invariant: Saving intermediate latents every 2 steps bounds spot preemption loss to at most 2 denoising steps.

Caption: Saving intermediate latents every 2 steps bounds spot preemption loss to at most 2 denoising steps.

04
STAGE 04 · WORKER EXECUTIONintermediate

8-GPU NVLink Ring Attention + TinyVAE SSE Streaming.

Each video worker pod binds 8 H100s over NVLink running Ring Attention + FP8 FlashAttention-3. At steps 1, 2, 4, and 8, a side-stream TinyVAE decodes a low-res WebP draft and publishes it to Redis Pub/Sub -> API Gateway SSE.

STAGE 04 · WORKER EXECUTION

Ring Worker + SSE Preview

Ring Worker + SSEPrevieHot PathCore EnginePartition / IndexGuard & VerifyLatency / RAM BoundProduction OutcomeScaled Statebounds

1.Stage Objective: Execute 243K-token video DiTs and stream < 20 ms progressive previews to clients.

2.Level (Intermediate): Each video worker pod binds 8 H100s over NVLink running Ring Attention + FP8 FlashAttention-3. At steps 1, 2, 4, and 8, a side-stream TinyVA

3.Operational Invariant: Side-stream TinyVAE pushes progressive WebP previews over Redis Pub/Sub to SSE clients.

Caption: Side-stream TinyVAE pushes progressive WebP previews over Redis Pub/Sub to SSE clients.

05
STAGE 05 · SAFETY FILTERINGadvanced

Pre-Gen INT8 CLIP Text Gate & Post-Gen ViT Classifier.

Gate 1 runs an INT8-quantized CLIP text safety classifier (< 3 ms) at ingress before enqueuing. Gate 2 runs a post-generation ViT NSFW/CSAM classifier on the decoded image/video keyframes before uploading to public S3/CDN.

STAGE 05 · SAFETY FILTERING

Dual-Gate Safety Pipeline

Dual-Gate SafetyPipelinFault TriggerCore EngineIsolation BoundaryGuard & VerifyFallback / QuarantineProduction OutcomeProtected Pathisolates

1.Stage Objective: Block unsafe prompts before wasting GPU compute and catch adversarial visual outputs.

2.Level (Advanced): Gate 1 runs an INT8-quantized CLIP text safety classifier (< 3 ms) at ingress before enqueuing. Gate 2 runs a post-generation ViT NSFW/CSAM

3.Operational Invariant: Pre-gen CLIP saves wasted GPU cycles; post-gen ViT catches adversarial prompt bypasses.

Caption: Pre-gen CLIP saves wasted GPU cycles; post-gen ViT catches adversarial prompt bypasses.

06
STAGE 06 · C2PA PROVENANCEadvanced

Invisible DCT Watermarking & Merkle-Batched HSM Signing.

Embed an invisible frequency-domain (DCT/SynthID) watermark into pixel output. Because signing 1,736 individual RSA/ECDSA payloads/sec would saturate HSMs, build a Merkle tree over batches of 256 asset SHA-256 hashes every 100 ms and sign only the Merkle root with the HSM, embedding the compact Merkle inclusion proof into each asset's C2PA manifest.

STAGE 06 · C2PA PROVENANCE

Merkle-Batched HSM C2PA Signing

Merkle-Batched HSMC2PACheckpoint / LogCore EngineReplay / SyncGuard & VerifyConsistency GateProduction OutcomeRecovered Stateverifies

1.Stage Objective: Sign 50M assets/day cryptographically without bottlenecking on Hardware Security Modules (HSMs).

2.Level (Advanced): Embed an invisible frequency-domain (DCT/SynthID) watermark into pixel output. Because signing 1,736 individual RSA/ECDSA payloads/sec would

3.Operational Invariant: Signing one Merkle root per 256 assets scales C2PA cryptographic provenance to 1,736 QPS.

Caption: Signing one Merkle root per 256 assets scales C2PA cryptographic provenance to 1,736 QPS.

07
STAGE 07 · AUTOSCALING & SLOsadvanced

Predictive Queue-Depth Scaling & Cold-Weight Caching.

Cache 30 GB FP8 DiT weights on local NVMe RAID-0 + host page cache across worker nodes and pre-warm standby pods using queue_wait_p95 predictive scaling. Alert if interactive_ttft_p95 > 4.5s or spot_preemption_rate > 15%.

STAGE 07 · AUTOSCALING & SLOs

NVMe Weight Cache & SLO Alerts

NVMe Weight CacheTelemetry SignalSLO AlertsSLO ThresholdValidation GateCircuit BreakerProduction OutcomeSafe Degradationtrips

1.Stage Objective: Prevent 30 GB DiT model-weight cold-start spikes during traffic surges.

2.Level (Advanced): Cache 30 GB FP8 DiT weights on local NVMe RAID-0 + host page cache across worker nodes and pre-warm standby pods using `queue_wait_p95` pred

3.Operational Invariant: Local NVMe model-weight caching eliminates 30s+ S3 download stalls during autoscaling.

Caption: Local NVMe model-weight caching eliminates 30s+ S3 download stalls during autoscaling.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Question 1 of 3
Score: 0/3

How does the platform sign 1,736 generated images/videos per second with C2PA provenance without overwhelming HSM hardware?

Recommended Learning Flow