Twitter / Social Media Feed

1. Hook

#

Twitter at peak serves 600K tweet reads per second while simultaneously processing tens of thousands of new tweets. The naive approach — querying who you follow, then fetching all their tweets, then sorting — collapses instantly at scale. The real architecture is a masterclass in the write-amplification vs read-latency trade-off, and the edge cases (Lady Gaga following Justin Bieber, or vice versa) reveal why no single strategy wins.

2. Problem Statement

#

Functional Requirements

#

1. Users can post tweets (text, images, videos).
2. Users can follow/unfollow other users.
3. A user’s home timeline shows tweets from people they follow, reverse-chronologically.
4. Users can like, retweet, and reply to tweets.
5. Search for tweets by keyword or hashtag.

Non-Functional Requirements

#

Out of Scope

#

• Direct messages
• Trending topics algorithm
• Ad insertion
• Abuse/spam detection

3. Scale Estimation

#

Assumptions:

• 300M DAU; each user reads timeline ~10×/day, posts ~0.5 tweet/day.
• Average user follows ~200 accounts; average follower count ~200.
• Celebrity accounts: up to 100M followers (e.g., Katy Perry).
• Tweet object: ~1KB (text + metadata). Media stored separately on object storage.

Fan-out is the dominant load: 350K cache writes/second at peak. This is the central problem the entire architecture is designed around.

4. High-Level Design

#

The system has two completely separate paths that are optimised independently: a write path that fans out new tweets to follower timelines, and a read path that assembles and serves a pre-built timeline in under 300ms.

flowchart TD
    subgraph CL["Client Layer"]
        MOB["Mobile / Web Client"]
    end
    subgraph AL["API Layer"]
        GW["API Gateway\nAuth · Rate Limit · Routing"]
    end
    subgraph WP["Write Path (async fan-out)"]
        TW["Tweet Service\nValidate · Store · Publish"]
        KF["Kafka\ntopic: tweet.created"]
        FO["Fan-out Service\nWorker Pool"]
    end
    subgraph RP["Read Path (pre-built timelines)"]
        TL["Timeline Service\nFetch · Merge · Respond"]
        HY["Hydration Service\nBatch-fetch tweet objects"]
    end
    subgraph SL["Storage Layer"]
        CAS["Cassandra\nTweet Store (canonical)"]
        RD["Redis Cluster\nTimeline Cache\nSorted Sets per user"]
        SGS["Social Graph Service\nFollower Sets"]
        S3["S3 + CDN\nMedia Storage"]
    end
    MOB -->|"POST /tweet"| GW
    MOB -->|"GET /timeline"| GW
    GW -->|"write request"| TW
    GW -->|"read request"| TL
    TW -->|"1. persist tweet"| CAS
    TW -->|"2. publish event"| KF
    TW -->|"media upload URL"| S3
    KF -->|"consume events"| FO
    FO -->|"lookup followers"| SGS
    FO -->|"ZADD tweet_id\nnormal users only"| RD
    TL -->|"ZRANGE latest 800"| RD
    TL -->|"pull celebrity tweets\nauthor follower_count > 10K"| CAS
    TL -->|"merged tweet ID list"| HY
    HY -->|"batch MGET by tweet_id"| CAS

Component Reference

#

5. Deep Dive

#

Fan-out Service — The Heart of the System

#

The fan-out service is where the hardest engineering lives. Its job sounds simple: when a tweet is created, write its ID into every follower’s timeline. The problem is scale — a user with 1 million followers generates 1 million Redis writes from a single tweet. With 1,750 tweets per second average, and many of those from accounts with large followings, the fan-out service needs to sustain 350,000+ Redis writes per second with minimal lag.

The solution is parallelism at every level. Fan-out workers run as a pool of Java 21 virtual threads — each Kafka consumer thread can handle hundreds of concurrent Redis pipeline calls without blocking an OS thread. Each fan-out event’s Redis writes are batched into a single pipeline per Redis shard, not sent as individual commands. This collapses 5,000 round-trips into roughly 30 pipeline calls (one per Redis shard that owns those follower IDs).

The celebrity threshold (default: 10,000 followers) is the key architectural escape valve. When an author exceeds it, the fan-out service stops writing their tweets into follower timelines entirely. Instead, their tweets are pulled at read time by the Timeline Service. This trade-off is asymmetric: for a user with 50 million followers, pushing to all timelines would cost 50 million Redis writes per tweet. Pulling means at most one Cassandra query per follower’s timeline load — and only when they’re actively reading. For a celebrity who tweets 5 times a day to 50 million followers, the push cost is 250 million Redis writes/day. The pull cost (assuming 10% of followers actively read daily): 5 million Cassandra reads/day. Pull wins by 50×.

The fan-out is eventually consistent by design. A user with 500 followers will see a tweet appear in all their followers’ timelines within a few seconds under normal load. During Kafka consumer lag events (fan-out backlog > 100K events), freshness SLO degrades gracefully — timelines are stale but correct. The system never corrupts or loses a tweet; it just delivers it later.

// Fan-out worker — Java 21, virtual threads, pipeline batching
public class FanoutWorker implements Runnable {

    private final FollowerRepository followerRepo;
    private final TimelineCache timelineCache;
    private static final int CELEBRITY_THRESHOLD = 10_000;
    private static final int TIMELINE_CAP = 800;

    @Override
    public void run() {
        try (var scope = new StructuredTaskScope.ShutdownOnFailure()) {
            kafkaConsumer.poll(Duration.ofMillis(100)).forEach(record -> {
                TweetEvent event = record.value();
                List<Long> followers = followerRepo.getFollowers(event.authorId());

                if (followers.size() > CELEBRITY_THRESHOLD) {
                    return; // pull model — Timeline Service handles this at read time
                }

                // Batch writes per Redis shard via pipeline
                scope.fork(() -> {
                    timelineCache.pipelinedZAdd(followers, event.tweetId(),
                        event.createdAtMs(), TIMELINE_CAP);
                    return null;
                });
            });
            scope.join().throwIfFailed();
        } catch (Exception e) {
            // Don't commit Kafka offset — event will be replayed
            log.error("Fan-out failed, offset not committed", e);
        }
    }
}

Timeline Cache — Why Sorted Sets, Not Lists

#

Redis offers several data structures that could store a user’s timeline. The choice matters because it affects both memory cost and merge performance.

A Redis List (LPUSH/LRANGE) offers O(1) prepend and O(N) range reads. It’s the obvious choice for an ordered feed. But lists have a critical flaw: they can’t be efficiently merged with an externally ordered set. When the Timeline Service needs to combine the pre-built list (from Redis) with celebrity tweets (from Cassandra), it needs to merge two time-sorted sequences. With a list, you’d have to load all entries, merge in memory, and re-sort — O(N log N) per request.

A Redis Sorted Set (ZADD/ZRANGE) uses the tweet’s unix timestamp as the score. Adding a new tweet is O(log N). Fetching the latest 20 is ZRANGE timeline:{userId} -1 -21 BYSCORE REV — O(log N + M). The merge operation for celebrity tweets becomes a simple merge of two pre-sorted sequences (like merge-sort’s merge step), which is O(M) where M is the number of celebrity tweets to interleave. This is the decisive advantage.

The cap at 800 entries is enforced on every write:

ZADD timeline:{userId} {timestamp} {tweet_id}
ZREMRANGEBYRANK timeline:{userId} 0 -801   # trim to newest 800

The 800-entry cap is chosen to cover 2-3 scroll sessions (most users read 20-50 tweets per session) while keeping per-user memory bounded at 800 × 8 bytes = 6.4KB. For 300M active users, that’s ~1.9TB of sorted set data across the Redis cluster — manageable with memory-optimised instances.

Social Graph Service — The Fan-out Enabler

#

The Social Graph Service is often underestimated in system design discussions. It’s not just a “followers table.” At Twitter’s scale, it’s a multi-layer system:

Redis layer (hot path): Each user’s follower set is stored as a Redis Set: followers:{userId} → {Set of follower userIds}. The Fan-out Service calls SMEMBERS followers:{authorId} to get all follower IDs in O(1) amortised time. For a user with 10,000 followers, this returns 10,000 IDs in a single Redis command.

MySQL layer (source of truth): All follow/unfollow operations write to MySQL first (follows table with an index on followee_id). MySQL enforces referential integrity (can’t follow a deleted account) and accurate follow counts. After a MySQL commit, the Social Graph Service asynchronously updates Redis. The Redis set can be slightly stale (by seconds) — acceptable.

Why not a graph database? Neo4j or similar graph DBs excel at multi-hop traversals (“friends of friends”). We don’t need multi-hop — we only need one-hop fan-out (get all direct followers). Redis Sets give us exactly that in O(1) at 10× lower latency than a graph DB query.

One subtle problem: the unfollow edge case. When a user unfollows someone, the Fan-out Service must stop writing that author’s tweets into the unfollower’s timeline. The unfollow event triggers an async cleanup — the last N tweet IDs from that author are removed from the user’s Redis timeline. This cleanup can be delayed by minutes, meaning the user may briefly still see tweets from someone they just unfollowed. This is an intentional product trade-off, not a bug.

Read Path — Timeline Assembly in Under 300ms

#

The read path has a strict 300ms p99 SLA, which means every call in the chain needs to be fast and parallelised. Here’s the detailed flow with latency budget:

Steps 2, 3, and 6 are all executed in parallel via CompletableFuture. The Hydration Service uses a singleflight pattern to prevent N concurrent requests for the same viral tweet ID each triggering a separate Cassandra read — the first request fetches and all others wait for its result.

6. Data Model

#

Tweet Table (Cassandra)

#

Partition strategy: Composite key (author_id, bucket) where bucket = YYYYMM. A prolific author tweeting 100 times/day generates 36,500 tweets/year — a single author_id partition would grow unboundedly and create a hotspot during celebrity read-time pulls. With monthly buckets, the Timeline Service queries at most 1-2 buckets per celebrity pull.

Why TIMEUUID as clustering key? TIMEUUID embeds the creation timestamp, so ORDER BY tweet_id DESC is equivalent to reverse-chronological without a separate sort. Cassandra stores clustering columns in sorted order on disk, making this a zero-cost O(N) scan.

Counter columns are separate tables: like_count and retweet_count are Cassandra COUNTER type, which must live in their own table (Cassandra limitation). Counter updates are commutative and idempotent — exactly what you need for concurrent like operations from millions of users.

Timeline Cache (Redis)

#

Key:   timeline:{userId}
Type:  Sorted Set
Score: unix timestamp (milliseconds)
Value: tweet_id (string representation of TIMEUUID)
Cap:   800 entries — enforced via ZREMRANGEBYRANK after every ZADD
TTL:   7 days for inactive users (EXPIRE set on cache miss + reconstruction)
       No TTL for active users (kept hot by continuous fan-out writes)

Follower Index (Redis + MySQL)

#

Redis (hot path):
  followers:{userId}  → SET  { followerId1, followerId2, ... }
  following:{userId}  → SET  { followeeId1, followeeId2, ... }
  celeb_following:{userId} → SET { celebId1, ... }  (subset where follower_count > 10K)

MySQL (source of truth):
  follows(
    follower_id  BIGINT NOT NULL,
    followee_id  BIGINT NOT NULL,
    created_at   TIMESTAMP DEFAULT NOW(),
    PRIMARY KEY  (follower_id, followee_id),
    INDEX        idx_followee (followee_id, follower_id)  -- for follower list queries
  )

7. Trade-offs

#

Fan-out on Write vs Fan-out on Read

#

Conclusion: Hybrid with a tunable celebrity threshold. The threshold is a knob: lower it (e.g., 1,000) to push more load to read time; raise it (e.g., 100,000) to fan-out more aggressively. Twitter reportedly used different thresholds for different scenarios.

Redis Sorted Set vs List for Timeline

#

Conclusion: Sorted Set. The merge operation for celebrity tweets is the deciding factor.

Cassandra vs MySQL for Tweets

#

Conclusion: Cassandra for tweets (write-heavy, scale-critical, simple access patterns). MySQL for social graph (relational integrity on follow counts, privacy settings).

Kafka vs Direct Redis Write for Fan-out

#

Conclusion: Kafka. User-facing write latency (POST /tweet) must be fast and predictable. Async fan-out keeps tweet creation at ~20ms regardless of follower count.

8. Failure Modes

#

9. Security & Compliance

#

AuthN/AuthZ: OAuth 2.0 with short-lived JWTs (15-min access token, 30-day refresh). API Gateway validates the JWT signature and injects userId into the request header downstream — internal services never re-validate the token. Tweets from private accounts are fan-outed only to approved followers: the fan-out worker checks the author’s privacy setting before writing to each follower’s timeline. Non-approved followers are skipped entirely.

Encryption: TLS 1.3 for all in-transit data (client to gateway, service to service via mutual TLS). At-rest encryption for Cassandra (AES-256) and Redis (encrypted EBS volumes on AWS). Media on S3 with SSE-S3 (server-side encryption). Encryption keys managed via AWS KMS with automatic 90-day rotation.

Input Validation: Tweet content sanitized server-side (XSS stripping, null byte removal). URLs expanded and checked against a malicious domain blocklist before storage. Media type validated against MIME type (not just file extension). Media scanned for CSAM using PhotoDNA before CDN upload — this is async, but the tweet is held from public display until scan completes (typically < 500ms).

Rate Limiting: Write: 300 tweets per 3 hours per user (token bucket in Redis, keyed by userId). Read: 1,500 timeline requests per 15 minutes per user (Twitter’s actual v2 API limit). These limits are enforced at the API Gateway, not inside microservices, using a shared Redis rate limit store. DDoS protection at edge via Cloudflare (Layer 3/4 scrubbing + Layer 7 WAF rules).

PII/GDPR: User deletion triggers async tombstone propagation — tweet content is cleared in Cassandra (is_deleted=true, content=null), then tweet IDs are removed from all follower timelines via a reverse fan-out using the Social Graph. Full deletion SLA: 30 days (Kafka retention must be ≥ 30 days to guarantee the deletion event reaches all consumers). Account deletion events are published to a dedicated user.deleted Kafka topic consumed by every service that holds user data.

Audit Log: All privileged actions (shadow banning, content removal, account suspension) written to an immutable append-only audit store: a dedicated Kafka topic with cleanup.policy=delete and infinite retention, archived nightly to S3 with object lock (WORM). This provides a tamper-evident record required for legal holds and SOC 2 compliance.

10. Observability

#

RED Metrics

#

Saturation Metrics

#

• Redis memory utilisation per shard: warn at 70% (before eviction), page at 85%
• Kafka partition throughput: warn at 80% of broker capacity
• Cassandra read latency p99 per node: warn at 50ms, page at 100ms
• Fan-out worker thread pool saturation: warn at 80%

Business Metrics

#

• Tweets per second (real-time, 1-min rolling window) — deviation > 30% from baseline triggers anomaly alert
• Timeline impressions per second
• Fan-out amplification ratio: (fan-out Redis writes / tweets created) — should track average follower count; sudden drop indicates fan-out workers falling behind

Tracing

#

OpenTelemetry with tail-based sampling (sample 10% of normal requests, 100% of requests where any span exceeds 200ms, 100% of errors). This ensures slow requests are always captured while keeping trace storage costs manageable.

Key trace spans: tweet.write → kafka.publish → fanout.dispatch → redis.pipeline.write, and separately: timeline.read → redis.zrange → celebrity.pull → hydration.batch_get → user.enrich. Jaeger for trace storage; Grafana dashboards for RED metrics and fan-out lag.

11. Scaling Path

#

Phase 1 — MVP (< 1K RPS reads)

#

Single Postgres DB, no Redis, no fan-out. Timeline query: SELECT * FROM tweets WHERE author_id = ANY($1) ORDER BY created_at DESC LIMIT 20. Simple, correct, fast enough at low scale.

What breaks first: DB CPU at ~5K timeline reads/s. The author_id = ANY(following_list) query becomes a full index scan when a user follows 200+ accounts.

Phase 2 — 10K RPS reads

#

Migrate tweets to Cassandra. Add Redis for timeline caching. Implement basic fan-out (synchronous, in-process, called during the POST /tweet handler). Add read replicas for Social Graph MySQL.

What breaks first: Synchronous fan-out adds latency to tweet creation. A user with 10,000 followers causes a 10-second tweet API response. Celebrity accounts are unusable.

Phase 3 — 100K RPS reads

#

Async fan-out via Kafka. Introduce celebrity threshold (10K followers → pull model). Redis Cluster (sharded by userId mod N). Cassandra multi-DC replication. Introduce Hydration Service as a separate tier.

What breaks first: Redis memory. 300M users × 800 IDs × 8 bytes = ~192GB. Need aggressive LRU eviction of inactive user timelines (TTL = 7 days). Also: Social Graph Redis starts showing memory pressure from large follower sets.

Phase 4 — 1M+ RPS reads

#

Tiered timeline storage: hot (Redis, last 800 tweets), warm (Memcached, last 7 days), cold (DynamoDB, full history). Geo-distributed fan-out workers per region (fan-out happens in the region where the tweet was created, replicates to follower regions via cross-region Kafka mirroring). Edge caching of celebrity tweet objects via CDN. Predictive pre-warm: pre-build timelines for users in push-notification cohorts before they open the app.

12. Enterprise Considerations

#

Brownfield Integration: If migrating from a monolith, use the Strangler Fig pattern — route /timeline to the new service while legacy handles everything else. Timeline data can be bootstrapped from existing MySQL with a one-time backfill job that reads the follows table and reconstructs Redis sorted sets per user. This backfill runs at low priority during off-peak hours to avoid impacting production MySQL.

Build vs Buy:

• Social Graph: Build (custom follow semantics, privacy rules, celebrity detection logic are too domain-specific for off-the-shelf)
• Message Queue: Kafka (Confluent Cloud or MSK for managed) — do not build
• Cache: Redis Cluster (ElastiCache on AWS, or self-managed for cost) — not Memcached (no sorted sets, no cluster-mode atomic operations)
• Object Storage: S3 / GCS — never build this
• CDN: Cloudflare or CloudFront for media; Fastly for API edge caching of public timelines
• Search: Elasticsearch / OpenSearch — feed tweets async via Kafka consumer

Multi-Tenancy: Not applicable for a Twitter-style consumer product. For a B2B social platform (Slack feeds, enterprise activity streams), isolate by workspace at the Redis key (timeline:{workspaceId}:{userId}) and Cassandra partition level. Fan-out workers use separate Kafka consumer groups per workspace tier to ensure one noisy tenant doesn’t delay another’s fan-out.

TCO Ballpark (100K RPS reads):

CDN and data transfer costs add 20-40% on top. At 1M+ RPS, egress becomes the dominant cost line.

Conway’s Law: Fan-out Service, Social Graph Service, Timeline Service, and Hydration Service should each be owned by separate teams. The fan-out amplification ratio (fan-out writes / tweets created) is a shared KPI owned by all four teams — it’s the single number that best describes system health and requires coordination to improve.

13. Interview Tips

#

• Always ask the celebrity problem upfront. “What’s the max follower count we need to handle?” This unlocks the hybrid fan-out discussion and signals you know the edge case. Without this, you’ll design a pure push model and get interrupted.
• Fan-out on write is the obvious first answer; immediately challenge it yourself. Mention write amplification for high-follower accounts, then pivot to the hybrid model. Interviewers want to see you reason through trade-offs, not recite a pattern without friction.
• Timeline freshness is a hidden non-functional requirement. Ask: “Is eventual consistency acceptable for timeline delivery? How stale can a timeline be?” This opens the discussion of Kafka consumer lag as a freshness SLA.
• Don’t forget cold start. What happens when an inactive user opens the app after 6 months? Their Redis TTL has expired. Timeline reconstruction from DB is expensive. Discuss async pre-warm on the user.login Kafka event.
• Vocabulary that signals fluency: fan-out amplification ratio, write-behind cache, sorted set capping, celebrity threshold, timeline hydration, social graph denormalisation, singleflight pattern, tail-based sampling.

14. Further Reading

#

• Twitter Engineering Blog — “Storing data at massive scale”: Twitter’s original Gizzard sharding framework and why they moved to Manhattan (internal distributed DB).
• “Feeding Frenzy: Selectively Materializing Users’ Event Feeds” (SIGMOD 2010) — The canonical academic treatment of social feed materialisation strategies. Directly describes the push/pull hybrid model.
• Martin Kleppmann, “Designing Data-Intensive Applications” — Chapter 11 covers stream processing for feed systems with Kafka; Chapter 5 covers replication lag in cached timelines.
• RFC 7519 — JWT spec; relevant for the OAuth 2.0 token lifecycle in the security section.