Write-Time vs Read-Time Fan-Out: How Social Feeds Scale

Scenario context

Fan-out is the act of distributing one post to many followers' feeds. Write-time fan-out (push) pre-computes feeds at post time — fast reads but catastrophic write amplification for celebrities. Read-time fan-out (pull) computes feeds on demand — no amplification but slow reads at scale. Production systems like Twitter/X, Instagram, and Facebook use a hybrid: write-time fan-out for users with ≤ 10K followers, re…

This walkthrough focuses on The Social Feed Problem: One Post, Ten Million Inboxes.

Before making a design choice, clarify:

• When Justin Bieber tweets, roughly 300 million accounts are eligible to see it.
• Your system has milliseconds to decide: do you immediately push that tweet into every follower's pre-built feed cache, or do you wait and assemble each follower's feed the moment they open the app?
• This is the fan-out problem — the act of distributing a single write event to many downstream consumers — and it is the central architectural decision in every social feed, notification system, and activity stream.

Concepts in play: caching, system-design-hld-architectures, database-indexing, event-driven-architecture

Stage 1 — Deep Dive: Inside the Redis Feed Cache and the Async Fan-Out Pipeline

• The Internals: Redis Sorted Sets, Kafka Topics, and Fan-Out Workers The canonical feed cache data structure is a Redis sorted set ( ZSET ).
• Each follower's feed is stored as a key like feed:{userId} where each member is a postId and the score is the post's Unix timestamp in milliseconds.
• This enables: - ZADD feed:42 1711720800000 postId:9999 — insert a post reference - ZREVRANGE feed:42 0 49 WITHSCORES — fetch the 50 most recent posts - ZREMRANGEBYRANK feed:42 0 -501 — evict entries beyond a 500-post ca…
• Post content is fetched separately in a second read from a post cache or DB, allowing the feed cache to remain compact and fast.

Kafka Topic: post.created
  Partition key: authorId (ensures ordered delivery per author)
  Consumer Group 1: fan-out-workers (writes to Redis feed caches)
  Consumer Group 2: notification-workers (sends push notifications)
  Consumer Group 3: analytics-ingest (feeds the engagement pipeline)

Observable outcome: Post content is fetched separately in a second read from a post cache or DB, allowing the feed cache to remain compact and fast.

Stage 2 — The Hybrid Architecture: Routing Posts by Follower Count

• The hybrid model routes the fan-out decision at post time, not at read time.
• The Fan-out Router — consuming from the post.created Kafka topic — checks the author's follower count against a configurable threshold (typically 10,000).
• Normal users fan out immediately; celebrity posts skip the write fan-out entirely and are stored in a dedicated Celebrity Post Store instead.
• The partition key for post.created should be authorId .

@Configuration
public class FanOutKafkaConfig {

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, PostCreatedEvent> fanOutContainerFactory(
            ConsumerFactory<String, PostCreatedEvent> consumerFactory) {

        ConcurrentKafkaListenerContainerFactory<String, PostCreatedEvent> factory =
            new ConcurrentKafkaListenerContainerFactory<>();

        factory.setConsumerFactory(consumerFactory);
        factory.setConcurrency(8);             // 8 threads per pod, one per Kafka partition
        factory.getContainerProperties()
               .setAckMode(ContainerProperties.AckMode.MANUAL_IMMEDIATE); // Commit only after Redis write succeeds

        return factory;
    }
}

@Component
public class FanOutConsumer {

    @KafkaListener(
        topics = "post.created",
        groupId = "fan-out-workers",
        containerFactory = "fanOutContainerFactory"
    )
    public void consume(PostCreatedEvent event, Acknowledgment ack) {
        fanOutService.handlePostCreated(event);
        ack.acknowledge(); // Only commit offset once fan-out is complete
    }
}

Observable outcome: The partition key for post.created should be authorId .

End-to-end outcome

After working through the stages, you should be able to explain:

• Forgetting to handle the celebrity path during cold-start backfill.
• When a user's feed cache expires and is rebuilt from the post DB, it is easy to only restore write-time fan-out posts.
• • Write-time fan-out (push) pre-computes follower feeds at post time using Redis sorted sets.
• Reads are O(1) but writes are O(N followers).

Alternatives considered

Fan-Out Fundamentals: The Push vs. Pull Decision

Fan-out refers to the branching factor of a write event: when one user posts, the event must "fan out" to all followers who should see it.

Write-Time Fan-Out: Paying the Cost at Post Time

When a user with 500 followers posts, a fan-out worker writes 500 entries into Redis — one per follower's sorted set feed cache — with the score set to the post's Unix timestamp.

Read-Time Fan-Out: Paying the Cost at Read Time

Read-time fan-out inverts the model entirely.

Failure and degraded modes

The source does not isolate a separate failure mode. Limits that matter for the scenario are called out in the stage decisions above.