Design a real-time chat and presence platform supporting 50M concurrent WebSocket connections, 1M messages/sec, < 100 ms p99 message delivery, multi-device read-cursor sync, and both 1:1 DMs and 100K-member channels.
From patterns to decisions.
Terminate stateful WebSockets at edge Gateway Pods backed by Redis session maps, sequence messages per channel with Snowflake/monotonic IDs into ScyllaDB/Cassandra, and route offline users to APNs/FCM.
Scale Envelope & Latency Budget.
Design a real-time chat and presence platform supporting 50M concurrent WebSocket connections, 1M messages/sec, < 100 ms p99 message delivery, multi-device read-cursor sync, and both 1:1 DMs and 100K-member channels.
Scale Envelope & Latency Budget
1.Stage Objective: Size throughput, storage, and p99 latency targets.
2.Level (Foundation): Design a real-time chat and presence platform supporting 50M concurrent WebSocket connections, 1M messages/sec, < 100 ms p99 message deliver
3.Operational Invariant: Size throughput, storage, and p99 latency targets.
Caption: Size throughput, storage, and p99 latency targets.
Core Architecture & Data Flow.
Each incoming message receives a time-ordered Snowflake ID (or per-channel sequence ID) so clients sort messages deterministically across devices without relying on client clocks. Messages persist asynchronously to ScyllaDB / Cassandra partitioned by (channel_id, time_bucket) to bound partition sizes.
Core Architecture & Data Flow
1.Stage Objective: Assemble the primary data path and storage layout.
2.Level (Foundation): Each incoming message receives a time-ordered **Snowflake ID** (or per-channel sequence ID) so clients sort messages deterministically acros
3.Operational Invariant: Assemble the primary data path and storage layout.
Caption: Assemble the primary data path and storage layout.
Gateway Routing & Offline Push Fallback.
def route_chat_message(presence_store, pubsub, push_queue, recipient_ids: list[str], msg: dict) -> None:
gateways = presence_store.mget_gateways(recipient_ids)
for uid, gw_id in zip(recipient_ids, gateways):
if gw_id:
pubsub.publish(f'gw:{gw_id}', {'user_id': uid, 'msg': msg})
else:
push_queue.enqueue({'user_id': uid, 'channel_id': msg['channel_id'], 'preview': msg['snippet']})Gateway Routing & Offline Push Fallback
1.Stage Objective: Route messages to online recipient gateways and fall back to push notifications for offline devices.
2.Level (Intermediate): def route_chat_message(presence_store, pubsub, push_queue, recipient_ids: list[str], msg: dict) -> None:
3.Operational Invariant: Route messages to online recipient gateways and fall back to push notifications for offline devices.
Caption: Route messages to online recipient gateways and fall back to push notifications for offline devices.
Sharding & Hot-Spot Mitigation.
Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.
Sharding & Hot-Spot Mitigation
1.Stage Objective: Prevent single-partition bottlenecks at peak load.
2.Level (Intermediate): Partition state deterministically and apply bounded-load or sub-bucket sharding for hot keys.
3.Operational Invariant: Prevent single-partition bottlenecks at peak load.
Caption: Prevent single-partition bottlenecks at peak load.
Failure Modes & Graceful Degradation.
Apply circuit breakers, fallback caches, and dead-letter quarantine queues.
Failure Modes & Graceful Degradation
1.Stage Objective: Keep critical paths healthy during partial outages.
2.Level (Advanced): Apply circuit breakers, fallback caches, and dead-letter quarantine queues.
3.Operational Invariant: Keep critical paths healthy during partial outages.
Caption: Keep critical paths healthy during partial outages.
Consistency, Idempotency & Recovery.
Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.
Consistency, Idempotency & Recovery
1.Stage Objective: Guarantee correctness across retries and node crashes.
2.Level (Advanced): Enforce atomic state transitions, fencing/idempotency keys, and fast snapshot recovery.
3.Operational Invariant: Guarantee correctness across retries and node crashes.
Caption: Guarantee correctness across retries and node crashes.
Telemetry, Circuit Breakers & Failure Modes.
Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants degrade.
Production Guardrails
1.Stage Objective: Monitor tail latency, error budgets, and invariant violations in production.
2.Level (Advanced): Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants de
3.Operational Invariant: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.
Caption: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.
Practice: Make the trade-off yourself.
Why partition ScyllaDB/Cassandra messages by `(channel_id, month_bucket)` instead of just `channel_id`?