Skip to main content
← CONCEPTS / SYSTEM DESIGN
An end-to-end domain learning journey

Real-Time Presence, Heartbeat Hysteresis
& Hybrid Fan-Out.

Real-time presence uses periodic WebSocket heartbeats into TTL-backed Redis keys (SETEX 45s) with online/offline hysteresis, paired with hybrid push/pull fan-out for small vs. massive channels.
Start the journey7 sections
FIRST PRINCIPLES. PRODUCTION SCALE.
The idea to keep with you

Real-time presence uses periodic WebSocket heartbeats into TTL-backed Redis keys (`SETEX 45s`) with online/offline hysteresis, paired with hybrid push/pull fan-out for small vs. massive channels.

TL;DR30-second simplified summary
▾

Broadcasting online/offline status changes for 50M concurrent users to every contact on every flaky mobile reconnect triggers a quadratic fan-out storm (O(N * F) events/sec).

Production presence systems decouple Heartbeat Ingestion (clients ping every 30s, refreshing SETEX presence:user_id 45s in Redis) from Lazy Presence Subscriptions (clients only subscribe to visible friends/active DM headers) and apply Offline Hysteresis (wait 45s+ before flipping offline so brief tunnel drops don't flicker).

For message delivery, small groups/DMs (<= 500 members) push directly via Redis Pub/Sub to sticky WebSocket gateways, while 100K-member Discord guilds switch to Hybrid/Pull or sharded guild relays.

From patterns to decisions.

Real-time presence uses periodic WebSocket heartbeats into TTL-backed Redis keys (`SETEX 45s`) with online/offline hysteresis, paired with hybrid push/pull fan-out for small vs. massive channels.

Full walkthrough · 7 stages
01
STAGE 01 · CORE INTUITIONfoundation

Core Intuition & Failure Mode.

Real-time presence uses periodic WebSocket heartbeats into TTL-backed Redis keys (SETEX 45s) with online/offline hysteresis, paired with hybrid push/pull fan-out for small vs. massive channels.

STAGE 01 · CORE INTUITION

Core Intuition & Failure Mode

Core IntuitionIngress / BaselineFailure ModeScale ConstraintValidation GateBottleneck SignalProduction OutcomeTarget SLAexposes

1.Stage Objective: Understand the baseline failure mode.

2.Level (Foundation): Real-time presence uses periodic WebSocket heartbeats into TTL-backed Redis keys (`SETEX 45s`) with online/offline hysteresis, paired with h

3.Operational Invariant: Understand the baseline failure mode.

Caption: Understand the baseline failure mode.

02
STAGE 02 · CORE MECHANICSfoundation

Architectural Mechanics.

Production presence systems decouple Heartbeat Ingestion (clients ping every 30s, refreshing SETEX presence:user_id 45s in Redis) from Lazy Presence Subscriptions (clients only subscribe to visible friends/active DM headers) and apply Offline Hysteresis (wait 45s+ before flipping offline so brief tunnel drops don't flicker).

STAGE 02 · CORE MECHANICS

Architectural Mechanics

ArchitecturalMechanicsInput StateCore EngineCore MechanismGuard & VerifyInvariant CheckProduction OutcomeOutput Statetransforms

1.Stage Objective: Trace the core mechanism end-to-end.

2.Level (Foundation): Production presence systems decouple **Heartbeat Ingestion** (clients ping every 30s, refreshing `SETEX presence:user_id 45s` in Redis) from

3.Operational Invariant: Trace the core mechanism end-to-end.

Caption: Trace the core mechanism end-to-end.

03
STAGE 03 · SAFE IMPLEMENTATIONintermediate

Heartbeat TTL & Presence Batch Reader.

def refresh_presence(redis_client, user_id: str, gateway_id: str, ttl_sec: int = 45) -> None:
    if not user_id or not gateway_id:
        raise ValueError('user_id and gateway_id are required')
    redis_client.set(f'presence:{user_id}', gateway_id, ex=ttl_sec)

def get_visible_presence(redis_client, friend_ids: list[str]) -> dict[str, bool]:
    if not friend_ids:
        return {}
    keys = [f'presence:{uid}' for uid in friend_ids[:200]]  # Cap batch size
    vals = redis_client.mget(keys)
    return {uid: (val is not None) for uid, val in zip(friend_ids[:200], vals)}
STAGE 03 · SAFE IMPLEMENTATION

Heartbeat TTL & Presence Batch Reader

Heartbeat TTLValidated InputPresence BatchReaderAtomic StepValidation GateError / Retry GuardProduction OutcomeCommitted Resultexecutes

1.Stage Objective: Refresh presence TTLs cleanly and batch-query visible contacts without fan-out storms.

2.Level (Intermediate): def refresh_presence(redis_client, user_id: str, gateway_id: str, ttl_sec: int = 45) -> None:

3.Operational Invariant: Refresh presence TTLs cleanly and batch-query visible contacts without fan-out storms.

Caption: Refresh presence TTLs cleanly and batch-query visible contacts without fan-out storms.

04
STAGE 04 · SCALING & TRADE-OFFSintermediate

Throughput & Latency Trade-Offs.

Evaluate asymptotic bounds, cache locality, and tail-latency impact under peak load.

STAGE 04 · SCALING & TRADE-OFFS

Throughput & Latency Trade-Offs

ThroughputHot PathLatency Trade-OffsPartition / IndexValidation GateLatency / RAM BoundProduction OutcomeScaled Statebounds

1.Stage Objective: Quantify operational complexity and memory bounds.

2.Level (Intermediate): Evaluate asymptotic bounds, cache locality, and tail-latency impact under peak load.

3.Operational Invariant: Quantify operational complexity and memory bounds.

Caption: Quantify operational complexity and memory bounds.

05
STAGE 05 · FAULT ISOLATIONadvanced

Blast-Radius & Failure Isolation.

Isolate shared resources, apply bounded queues, and prevent retry amplification.

STAGE 05 · FAULT ISOLATION

Blast-Radius & Failure Isolation

Blast-RadiusFault TriggerFailure IsolationIsolation BoundaryValidation GateFallback / QuarantineProduction OutcomeProtected Pathisolates

1.Stage Objective: Contain failures without cascading across tenants or nodes.

2.Level (Advanced): Isolate shared resources, apply bounded queues, and prevent retry amplification.

3.Operational Invariant: Contain failures without cascading across tenants or nodes.

Caption: Contain failures without cascading across tenants or nodes.

06
STAGE 06 · RECOVERY & CONSISTENCYadvanced

Durability, Consistency & Recovery.

Use idempotent state transitions, atomic commits, and deterministic replay.

STAGE 06 · RECOVERY & CONSISTENCY

Durability, Consistency & Recovery

DurabilityConsistencyCheckpoint / LogRecoveryReplay / SyncValidation GateConsistency GateProduction OutcomeRecovered Stateverifies

1.Stage Objective: Preserve invariants across crashes and restarts.

2.Level (Advanced): Use idempotent state transitions, atomic commits, and deterministic replay.

3.Operational Invariant: Preserve invariants across crashes and restarts.

Caption: Preserve invariants across crashes and restarts.

07
STAGE 07 · OBSERVABILITY & GUARDRAILSadvanced

Telemetry, Circuit Breakers & Failure Modes.

Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants degrade.

STAGE 07 · OBSERVABILITY & GUARDRAILS

Production Guardrails

ProductionGuardrailsTelemetry SignalCore EngineSLO ThresholdGuard & VerifyCircuit BreakerProduction OutcomeSafe Degradationtrips

1.Stage Objective: Monitor tail latency, error budgets, and invariant violations in production.

2.Level (Advanced): Track p99 latency, retry/conflict ratios, and saturation metrics; fail fast with bounded retries and circuit breakers whenever invariants de

3.Operational Invariant: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.

Caption: Bounded retries, telemetry alerts, and circuit breakers prevent cascading failures.

Practice · Explain the consequences

Practice: Make the trade-off yourself.

Question 1 of 3
Score: 0/3

Why do presence systems use a 45-second TTL with 30-second heartbeats?

Recommended Learning Flow