WhatsApp / Chat Messaging System

1. Hook

#

WhatsApp delivers 100 billion messages every day to 2 billion users across 180+ countries — all end-to-end encrypted (E2EE), with sub-second latency, and with a global engineering team historically smaller than 50 engineers. The system does this while providing strong delivery guarantees (a message is either delivered exactly once or the sender knows it was not), preserving per-conversation message ordering even when users switch networks mid-send, and maintaining ephemeral server storage so that once a message is delivered it lives only on client devices.

Three hard problems must be solved simultaneously: keeping persistent TCP (Transmission Control Protocol) connections alive for every active user at planetary scale; coordinating delivery receipts across a sender, multiple recipient devices, and the server; and doing all of this end-to-end encrypted so the server never sees plaintext — meaning even WhatsApp’s own engineers cannot read your messages. Understanding how these three constraints interact is the central challenge of this design.

2. Problem Statement

#

Functional Requirements

#

1. Users can send and receive 1-on-1 text messages.
2. Users can send and receive group messages (up to 1,024 members).
3. Messages carry delivery status: Sent (one grey tick), Delivered (two grey ticks), Read (two blue ticks).
4. Users can send media: photos, videos, voice notes, documents.
5. Users can see a contact’s online/typing status and last-seen timestamp.
6. All messages are end-to-end encrypted; the server never holds decryptable content.
7. Messages sent while the recipient is offline are queued and delivered on reconnection.
8. Multi-device: the same account can be active on up to 4 linked devices simultaneously.

Non-Functional Requirements

#

Out of Scope

#

• Payments (WhatsApp Pay).
• Business API / WhatsApp Business Platform.
• Full-text message search across history.

3. Scale Estimation

#

Assumptions:

• 1 B Daily Active Users (DAU) out of 2 B registered users.
• Average of ~100 messages sent per DAU per day.
• Peak traffic is 3× the daily average (evening hours in major timezones overlap).
• 20% of messages include media (photos, video, voice).
• Average media size: 500 KB after client-side compression.
• 5% of messages queue for offline recipients at any given moment.

Daily active users (DAU):          ~1 B
Messages/day:                      100 B  →  ~1.16 M msgs/sec average
Peak multiplier:                   3×     →  ~3.5 M msgs/sec
Concurrent WebSocket connections:  ~500 M (50% of DAU active at peak)

Message size (text, median):       ~200 bytes
Text message throughput:           1.16 M × 200 B = ~230 MB/s raw, ~2 TB/day

Media messages (~20% of total):    20 B/day
Average media size:                500 KB
Media upload bandwidth:            ~20 B × 500 KB / 86,400 s = ~115 GB/s (median)
Media storage (30-day retention):  20 B × 500 KB × 30 ≈ 300 PB (before dedup)

Undelivered message queue:
  ~5% of messages queue for offline recipients
  5 B messages × 200 B = 1 TB in queue at any time (manageable in Cassandra)

The most constraining resource is concurrent TCP connections (500 M sockets), not raw message throughput. Each socket consumes kernel memory (~4 KB minimum), so the gateway fleet needs ~2 TB of RAM just for connection state. This is why WhatsApp’s original Erlang/OTP stack (which handles millions of lightweight processes per node) was such a good fit — each process maps to one connection with almost no overhead.

4. High-Level Architecture

#

The system decomposes into five independent planes: a connection plane (WebSocket gateways holding persistent sockets), a routing plane (Message Service distributing messages to the right gateways), a storage plane (offline queue + key store), a media plane (client-direct uploads to object storage), and a presence plane (online/typing status).

flowchart TD
    subgraph CL["Client Layer"]
        A1["Alice — Phone"]
        A2["Alice — Linked Tablet"]
        B1["Bob — Phone"]
    end

    subgraph EDGE["Edge — Connection Plane"]
        LB["Load Balancer / Anycast"]
        GW1["Chat Gateway 1\nWebSocket Server"]
        GW2["Chat Gateway 2\nWebSocket Server"]
    end

    subgraph CORE["Core Services — Routing Plane"]
        MS["Message Service\nrouting + fan-out"]
        GS["Group Service\nmembership + fan-out"]
        PS["Presence Service\nonline / typing"]
        NS["Notification Service\nAPNs / FCM"]
        KS["Key Distribution Service\npublic keys + prekeys"]
    end

    subgraph STORE["Storage Layer"]
        MQ["Message Queue\nKafka"]
        MDB["Message Store\nCassandra — offline queue"]
        KDB["Key Store\nPostgreSQL"]
        PDB["Presence Store\nRedis"]
        OB["Media Object Store\nS3-compatible"]
    end

    A1 -->|"WebSocket / TLS"| LB
    A2 -->|"WebSocket / TLS"| LB
    B1 -->|"WebSocket / TLS"| LB
    LB --> GW1
    LB --> GW2

    GW1 -->|"publish"| MQ
    GW2 -->|"publish"| MQ

    MQ --> MS
    MS -->|"online delivery"| GW1
    MS -->|"online delivery"| GW2
    MS -->|"offline queue write"| MDB
    MS -->|"push notification"| NS
    MS --> GS

    GS -->|"member list"| MDB
    GS -->|"fan-out per member"| MS

    PS -->|"heartbeat reads/writes"| PDB
    GW1 -->|"presence update"| PS
    GW2 -->|"presence update"| PS

    A1 -->|"key fetch"| KS
    KS -->|"store/fetch"| KDB

    A1 -->|"media upload URL"| OB
    A1 -->|"download URL"| OB

Component Reference

#

5. Deep Dive — Connection Layer (WebSocket)

#

Why WebSocket over HTTP polling?

#

HTTP long-polling works like this: the client sends a request, the server holds it open until a message arrives, sends a response, and then the client immediately sends another request. This creates a constant cycle of connection teardown and re-establishment, which wastes server file descriptors and adds one full RTT (Round-Trip Time, typically 20–100 ms on mobile) of latency on every message. At 500 M concurrent users, that overhead is catastrophic.

WebSocket upgrades a standard HTTP connection into a persistent, full-duplex TCP channel. After the upgrade handshake, both client and server can send frames at any time without the request-response ceremony. A message from Alice reaches Bob in exactly one RTT once both sides are connected — no polling, no re-connection overhead.

The WebSocket handshake over TLS (Transport Layer Security — the encryption protocol that replaces the older SSL) looks like this:

Client → Server:  GET /ws HTTP/1.1
                  Upgrade: websocket
                  Connection: Upgrade
                  Sec-WebSocket-Key: <base64 nonce>

Server → Client:  HTTP/1.1 101 Switching Protocols
                  Upgrade: websocket
                  Connection: Upgrade
                  Sec-WebSocket-Accept: <HMAC of nonce>

After this exchange, the TCP connection is handed off to the WebSocket framing layer. The original HTTP server is no longer involved.

Gateway design and the routing registry

#

Each Chat Gateway is a stateful process holding tens of thousands of open WebSocket connections. The key challenge: when the Message Service wants to deliver a message to Bob, which of the potentially hundreds of gateway nodes is Bob connected to?

The answer is a routing registry — a Redis hash that maps user_id → gateway_id. Every gateway registers its connected users here on connect and deregisters on disconnect. The lookup is O(1).

Bob connects to GW-7:
  Redis: SET routing:{bob_id} "gw-7" EX 90

Message Service receives message for Bob:
  1. Redis GET routing:{bob_id}  →  "gw-7"
  2. gRPC call to GW-7: deliver(msg)
  3. GW-7 writes msg to Bob's WebSocket

Bob's heartbeat every 60 s:
  GW-7 refreshes:  EXPIRE routing:{bob_id} 90

If the routing registry returns no entry (Bob is offline), the Message Service writes to the Cassandra queue instead.

For multi-device support (Bob on phone + tablet), Bob has two routing entries — one per device — and the Message Service delivers to all of them in parallel.

Keep-alive and reconnection

#

Mobile networks aggressively kill idle TCP connections (NAT (Network Address Translation) timeout is often 30 s on cellular). Clients send a WebSocket ping frame every 60 seconds; the gateway responds with a pong frame. Missing two consecutive pings (120 s of silence) causes the gateway to close the connection and delete the routing entry.

Clients implement exponential backoff on reconnect: 1 s, 2 s, 4 s, 8 s, … capped at 60 s. This prevents a thundering herd (a situation where thousands of clients all reconnect simultaneously after a gateway restart) from overwhelming the recovering gateway.

Java: connection registry interaction

#

// Simplified gateway connection registration (Java 21, virtual threads)
public class GatewayConnectionHandler {

    private final RedisClient redis;
    private final String gatewayId;
    private static final int PRESENCE_TTL_SECONDS = 90;

    public void onConnect(String userId, WebSocketSession session) {
        // Register in routing registry — O(1) lookup for Message Service
        redis.set("routing:" + userId, gatewayId, PRESENCE_TTL_SECONDS);
        localSessions.put(userId, session);
    }

    public void onHeartbeat(String userId) {
        // Refresh TTL so the entry doesn't expire while client is active
        redis.expire("routing:" + userId, PRESENCE_TTL_SECONDS);
    }

    public void onDisconnect(String userId) {
        redis.delete("routing:" + userId);
        localSessions.remove(userId);
    }

    public void deliver(String userId, byte[] messageFrame) {
        WebSocketSession session = localSessions.get(userId);
        if (session != null && session.isOpen()) {
            session.sendBinary(messageFrame); // non-blocking with virtual threads
        }
    }
}

6. Deep Dive — Message Flow & Ordering

#

Send path (Alice → Bob, both online)

#

The numbered steps below trace a single message from Alice’s keypress to Bob’s screen. Pay close attention to where the ACKs (acknowledgements) travel — this is what drives the tick system.

sequenceDiagram
    participant Alice
    participant GW_A as Gateway A
    participant Kafka
    participant MsgSvc as Message Service
    participant GW_B as Gateway B
    participant Bob

    Alice->>GW_A: SEND {msg_id, to: Bob, ciphertext, seq}
    GW_A->>Kafka: publish(msg)
    GW_A-->>Alice: ACK — one grey tick (Sent)

    Kafka->>MsgSvc: consume(msg)
    MsgSvc->>GW_B: deliver(msg) — Bob online on GW_B
    GW_B->>Bob: PUSH message

    Bob-->>GW_B: DELIVERED ACK {msg_id}
    GW_B->>MsgSvc: forward delivered receipt
    MsgSvc->>GW_A: delivery_receipt(msg_id, bob)
    GW_A->>Alice: two grey ticks (Delivered)

    Bob-->>GW_B: READ ACK {conversation_id, up_to_msg_id}
    GW_B->>MsgSvc: forward read receipt
    MsgSvc->>GW_A: read_receipt(conversation_id)
    GW_A->>Alice: two blue ticks (Read)

Why does the single tick appear before the message reaches Bob? Because the single tick means the server received the message — not that Bob received it. This is an intentional design: Alice gets instant feedback that her message is safely in the server queue, even if Bob’s device is slow to respond. The two-tick system requires a round-trip to Bob’s device, which may take longer on a slow network.

Why does the delivered receipt go server→Gateway A→Alice instead of server→Alice directly? Because Alice may not be on the same gateway as Bob. The Message Service acts as an intermediary that knows which gateway each user is on. It routes the receipt through Alice’s gateway just like a normal message, but in the reverse direction.

Ordering guarantees

#

WhatsApp guarantees per-conversation ordering (not global ordering across all your conversations):

• Each client assigns a monotonically increasing client sequence number per conversation (seq = 1, 2, 3, …). This number is included in every SEND frame.
• Kafka preserves insertion order within a partition. All messages in a conversation are sent to the same Kafka partition, keyed by conversation_id. This means the Message Service always consumes messages in the order the sender sent them.
• If the recipient’s client detects a gap in sequence numbers (seq jumps from 5 to 7, meaning 6 is missing), it sends a SYNC request to fetch the missing message from the Cassandra queue.

For group messages, the Group Service assigns a server-side sequence number at fan-out time, giving all members a consistent global order within the group conversation — even if two members sent messages at the same millisecond.

Offline delivery

#

When Bob is offline at delivery time:

1. The Message Service writes to Cassandra: (recipient_id=bob, conversation_id, server_ts, msg_id) → ciphertext_blob.
2. Triggers a silent push (an invisible background notification that wakes the app without showing a banner) via FCM or APNs.
3. On reconnect, Bob’s client sends SYNC {last_server_ts: T}. The Message Service does a Cassandra range scan for all rows where server_ts > T and recipient_id = bob, streams them to Bob’s gateway, which pushes them to Bob’s client.
4. Bob’s client sends a DELIVERED ACK for each message. The Message Service deletes those rows from Cassandra.

TTL: 30 days. After TTL, messages are dropped from the queue and the sender receives a permanent delivery failure notification. WhatsApp does not store messages beyond this point — the server is a transient relay, not a long-term archive.

7. Deep Dive — End-to-End Encryption (Signal Protocol)

#

WhatsApp adopted the Signal Protocol (developed by Open Whisper Systems) in 2016. The server routes encrypted blobs and never holds the keys to decrypt them. Even a subpoena to WhatsApp yields only metadata (who talked to whom, when), not message content.

Key types and their roles

#

X3DH handshake — establishing a session with a new contact

#

X3DH stands for Extended Triple Diffie-Hellman. When Alice sends her first message to Bob (someone whose session she has never opened), she cannot just encrypt because she doesn’t share a secret with Bob yet. X3DH solves this without Alice and Bob needing to be online at the same time.

Alice fetches from Key Distribution Service:
  Bob's IK_B (Identity Key),
  SPK_B (Signed PreKey, with IK_B's signature),
  OPK_B (One-Time PreKey — consumed after this)

Alice generates her own ephemeral key pair: EK_A

Alice computes four DH operations:
  DH1 = DH(IK_A,  SPK_B)   — "I know Bob's medium-term key"
  DH2 = DH(EK_A,  IK_B)   — "Bob knows my ephemeral key via his identity key"
  DH3 = DH(EK_A,  SPK_B)   — "ephemeral meets medium-term"
  DH4 = DH(EK_A,  OPK_B)   — "one-time use ensures no replay"

Master secret = KDF(DH1 || DH2 || DH3 || DH4)
  KDF = Key Derivation Function (HKDF-SHA256 in practice)
  || = concatenation

Why four DH operations? Each one contributes a different security property. Combining all four means an attacker would have to break all four independently to compromise the session — defence in depth at the cryptographic layer.

This gives forward secrecy: if Bob’s long-term IK is later compromised, past sessions (which used OPKs that are now deleted) cannot be decrypted.

Double Ratchet — per-message key evolution

#

After the X3DH handshake, every subsequent message advances two ratchets (think of a ratchet as a one-way click — each click derives a new key from the previous, but you cannot go backwards):

1. Symmetric-Key Ratchet (KDF chain): each message derives a unique message key from the current chain key. The message key is used once then discarded — even if an attacker records all traffic and later steals the current chain key, past messages (which used already-discarded message keys) remain undecryptable.

2. Diffie-Hellman Ratchet: every time Alice and Bob exchange new messages, they also exchange new ephemeral DH public keys. This rotates the root key, providing break-in recovery: if an attacker somehow steals the current session state, future messages (encrypted with the next DH ratchet step) become inaccessible to the attacker.

The practical result: every single message in a WhatsApp conversation uses a unique key, derived from a key that existed briefly and was then deleted.

Multi-device E2EE

#

Each linked device (up to 4 per account) has its own independent Identity Key and prekey bundle registered in the KDS. When Alice sends a message:

• Alice’s primary device encrypts the plaintext once per recipient device — one ciphertext for Bob’s phone, one for Bob’s tablet.
• Additionally, one ciphertext is encrypted for each of Alice’s own linked devices — so her tablet also shows the sent message.
• For a message from Alice (2 devices) to Bob (3 devices): the KDS bundles fetched are for 5 devices, and 5 independent ciphertexts are generated from the same plaintext.

This is the correct E2EE approach — the alternative (encrypting once and having devices share a key) would mean any one compromised device exposes all devices.

8. Deep Dive — Group Messaging

#

Sender Keys — why group encryption is different

#

In a 1-on-1 conversation, Alice encrypts once for Bob. In a group with N members, naively Alice would encrypt once per member: O(N) encryption operations per message. At 1,024 members sending several messages per hour each, this becomes prohibitive.

Sender Keys solve this. The sender (Alice) generates a symmetric Group Session Key — a shared AES key for this group session. She then:

1. Encrypts the Group Session Key once per member using their pairwise Signal session (X3DH + Double Ratchet). This happens once, or whenever the key rotates.
2. For every subsequent group message, encrypts the plaintext with the Group Session Key — a single fast AES operation, O(1) regardless of group size.

First message to group (N members):
  Encrypt Group Session Key N times (one per member) → O(N)

Every subsequent message:
  Encrypt plaintext with Group Session Key → O(1)

The Group Session Key rotates (a new key is generated and re-distributed) whenever:

• A member leaves the group — the departing member must not be able to decrypt future messages (forward secrecy for departed members).
• A member is removed by an admin.

Key rotation requires another O(N) distribution round — the cost of maintaining group membership changes.

Fan-out for large groups (up to 1,024 members)

#

The Sender Key protocol handles the encryption side. The delivery side is handled by the Group Service fanning out one delivery task per member to Kafka:

1 group message → Group Service reads membership list (N members)
                → N delivery tasks published to Kafka
                  (partitioned by recipient_id, not by group_id)
                → Message Service handles each task independently
                → Each member's gateway delivers in parallel

Thundering herd mitigation: publishing all 1,024 tasks to the same Kafka partition would create a serialisation bottleneck (only one consumer processes a partition at a time). Instead, tasks are partitioned by recipient_id, distributing them across many partitions and consumer instances — fully parallel delivery.

Group metadata schema

#

The group_sender_keys table is the most write-amplified: every time Alice sends a group message for the first time (or after a key rotation), one row per (sender device, recipient device) pair is written.

9. Deep Dive — Media Handling

#

Upload flow — why the gateway never touches binary data

#

The Chat Gateway’s job is to hold hundreds of millions of persistent connections. If media bytes flowed through the gateway, a single user uploading a 50 MB video would consume 50 MB of gateway RAM for the duration of the upload — and with millions of concurrent uploads, this would exhaust gateway memory in seconds. Instead:

1. Alice selects a photo on her phone.

2. Alice's client encrypts the media blob locally:
   - AES-256-CBC for confidentiality
   - HMAC-SHA256 for integrity (ensures the blob was not tampered with in transit)
   - Unique IV (Initialisation Vector) and media key generated per upload.

3. Client calls Media Service → receives a pre-signed S3 PUT URL (valid 5 min).
   A pre-signed URL is a time-limited, single-object-scoped URL that grants
   write permission to one specific S3 key without exposing AWS credentials.

4. Client uploads the encrypted blob directly to object storage.
   The Chat Gateway is not involved in this step.

5. Client sends a text message containing:
   {
     media_url:          "https://cdn.whatsapp.net/.../blob.enc",
     encrypted_media_key: <AES key, itself encrypted for recipient>,
     media_sha256:        <integrity hash of the encrypted blob>,
     mime_type:          "image/jpeg",
     thumbnail_ciphertext: <tiny blurred preview, also encrypted>
   }

6. Bob's client downloads the blob from media_url, verifies the SHA256,
   decrypts with the media key, and renders the image.

The server stores an encrypted blob it cannot read. The media key inside the message is itself encrypted end-to-end — only Bob’s device can decrypt it.

Content deduplication

#

Clients compute a SHA256 of the plaintext before encryption. The Media Service maintains a content-addressed index (plaintext hash → S3 key). If the same photo was uploaded recently (e.g., many people sharing a viral meme), the server issues a download URL for the existing blob without requiring a new upload — dramatically reducing storage and egress costs. This deduplication is invisible to the encryption model: the encrypted blob on disk may differ per sender (because different IVs produce different ciphertexts for the same plaintext), but the server-side plaintext hash comparison still works because the Media Service computes it before encryption (the client sends the plaintext hash alongside the encrypted upload).

10. Deep Dive — Presence & Typing Indicators

#

Online/Offline presence

#

Presence is implemented as a short-TTL key in Redis, refreshed by heartbeats:

Client connects → Gateway sets:
  Redis SETEX presence:{user_id} 90 "{gateway_id, connected_at}"

Client heartbeat (every 60 s) → Gateway refreshes:
  Redis EXPIRE presence:{user_id} 90

Client disconnects → Gateway deletes:
  Redis DEL presence:{user_id}
  (or the TTL expires naturally if the disconnect is unclean — e.g., phone battery dies)

When Alice’s presence key is created or expires, the Presence Service fans out a presence update to subscribers (Alice’s mutual contacts who have opted into presence visibility). This fanout is itself bounded by WhatsApp’s privacy model: “last seen” and online status are only shared with contacts, and can be restricted further in Settings.

Typing indicators

#

Typing indicators are the most ephemeral signals in the system — they have no durability requirement, no storage, and no retry:

• When the user types the first character, the client sends a COMPOSING event.
• After 5 seconds of no keystrokes, the client sends a PAUSED event.
• If the network drops during this, the indicator simply disappears — no retry, no queue, no Cassandra write.
• The COMPOSING event is forwarded by the gateway in-band over the WebSocket connection directly to the recipient’s gateway via the Message Service. The Presence Service is not involved (keeping the path as low-latency as possible — a separate service hop would add ~5 ms of unnecessary overhead).

11. Data Models

#

Offline message queue — Apache Cassandra

#

Cassandra is chosen over a relational database (like PostgreSQL) because it is append-optimised: writes are sequential (appended to a commit log, then a sorted string table — SSTable). At billions of messages per day, PostgreSQL’s B-tree index maintenance costs would create write bottlenecks. Cassandra’s wide-row data model also perfectly matches the access pattern: append messages for a user, then bulk-read-and-delete all of them on reconnect.

CREATE TABLE offline_messages (
    recipient_id     UUID,
    conversation_id  UUID,
    server_ts        TIMESTAMP,
    msg_id           UUID,
    sender_id        UUID,
    ciphertext       BLOB,
    msg_type         TINYINT,   -- 0=text, 1=media, 2=control (receipts, key updates)
    ttl              TIMESTAMP,
    PRIMARY KEY ((recipient_id), server_ts, msg_id)
) WITH CLUSTERING ORDER BY (server_ts ASC)
  AND default_time_to_live = 2592000;  -- 30 days in seconds

Why recipient_id as the partition key? All pending messages for one user live on the same Cassandra node (or a small set of replicas). When Bob reconnects and requests a sync, the Message Service issues a single range scan: SELECT * FROM offline_messages WHERE recipient_id = bob AND server_ts > last_sync_ts. This is one sequential disk read, not a scatter-gather across 100 nodes.

Why compound clustering on (server_ts, msg_id)? server_ts provides chronological ordering. msg_id (a UUID) breaks ties when two messages arrive in the same millisecond and ensures uniqueness.

Conversation index — Cassandra

#

CREATE TABLE conversations (
    user_id          UUID,
    conversation_id  UUID,
    counterpart_id   UUID,    -- other user_id (1-on-1) or group_id (group chat)
    is_group         BOOLEAN,
    last_msg_ts      TIMESTAMP,
    last_msg_preview BLOB,    -- encrypted snippet for local display (never decryptable server-side)
    unread_count     INT,
    PRIMARY KEY ((user_id), last_msg_ts, conversation_id)
) WITH CLUSTERING ORDER BY (last_msg_ts DESC);

Clustered by last_msg_ts DESC so the most recently active conversations are at the top of the Cassandra partition — matching the inbox view that users see on app open.

Public key storage — PostgreSQL

#

A relational database fits here because key material is looked up by exact (user_id, device_id) primary key — no range scans, no time-series patterns. PostgreSQL’s ACID (Atomicity, Consistency, Isolation, Durability) guarantees are important for key operations: you must not lose a prekey upload or double-consume a one-time prekey.

CREATE TABLE identity_keys (
    user_id      UUID,
    device_id    UUID,
    identity_key BYTEA  NOT NULL,
    created_at   TIMESTAMPTZ DEFAULT now(),
    PRIMARY KEY (user_id, device_id)
);

CREATE TABLE prekeys (
    user_id       UUID,
    device_id     UUID,
    prekey_id     INT,
    prekey_type   VARCHAR(10),  -- 'signed' or 'onetime'
    public_key    BYTEA,
    signature     BYTEA,        -- present only for signed prekeys (proves IK signed it)
    used          BOOLEAN DEFAULT FALSE,
    PRIMARY KEY (user_id, device_id, prekey_id)
);

-- Index to efficiently count remaining one-time prekeys per device
CREATE INDEX ON prekeys (user_id, device_id) WHERE prekey_type = 'onetime' AND used = FALSE;

The KDS alerts a client to upload new prekeys when this count falls below 10. If it reaches 0, the server falls back to the Signed PreKey for new session initiations — still secure, but with reduced forward secrecy for that specific handshake.

12. API Design

#

WhatsApp uses a custom binary framing protocol over TLS (originally XMPP (Extensible Messaging and Presence Protocol), now replaced by a compact binary format similar to Protocol Buffers). Below are the logical semantics expressed as WebSocket event types and REST-style endpoints.

WebSocket events (client → server)

#

Media REST endpoints

#

Key Distribution Service REST endpoints

#

13. Trade-offs & Alternatives

#

Server-side message storage vs. client-only vs. permanent storage

#

WhatsApp’s 30-day queue is a deliberate middle path: enough durability for typical offline periods (vacation, phone replacement) without becoming a permanent archive.

Fan-out on write vs. fan-out on read for groups

#

For very large groups or broadcast channels (WhatsApp’s Channels product), fan-out on read becomes more attractive: store one copy, let members pull. WhatsApp’s 1,024-member limit exists partly to keep fan-out on write tractable.

XMPP vs. custom binary protocol

#

Original WhatsApp used XMPP, an XML-based messaging protocol. Each XML tag adds bytes — a simple message could be ~1 KB on the wire. Moving to a custom binary framing (similar to protobuf length-prefixed frames) reduced per-message overhead from ~1 KB to ~100 bytes. At 100 B messages/day, that 10× reduction saves ~100 TB/day of network bandwidth — roughly $10 M/year at cloud egress pricing.

WebSocket vs. QUIC

#

QUIC (the transport protocol underlying HTTP/3) provides two advantages over WebSocket-over-TLS:

1. 0-RTT (Zero Round-Trip Time) resumption: reconnecting after a network change (WiFi → LTE) takes near-zero time instead of a full TCP + TLS handshake (~300 ms).
2. No head-of-line blocking: TCP blocks all data behind a lost packet; QUIC streams are independent, so one lost packet only delays that stream.

WhatsApp has explored QUIC but WebSocket over TLS remains the primary protocol. The switch requires client + server updates and careful fallback handling for older OS versions that don’t support QUIC.

14. Failure Modes & Mitigations

#

15. Monitoring & Observability

#

Key metrics (RED framework: Rate, Errors, Duration)

#

Distributed tracing

#

Every message carries a trace_id that propagates through: Client SEND → Kafka publish → Message Service consume → Gateway deliver → Client PUSH → DELIVERED ACK → Sender receipt

This full trace (implemented with OpenTelemetry, a vendor-neutral observability standard) allows engineers to pinpoint exactly where a specific message was delayed — whether in the Kafka consumer, the routing registry lookup, the gRPC call to the target gateway, or the WebSocket write itself.

Dashboards

#

• Connection health: active WebSocket counts per gateway, reconnect rate, heartbeat miss rate, gateway memory utilisation.
• Message pipeline: Kafka throughput (messages/sec), consumer group lag per partition, delivery success rate, grey-tick to double-tick latency distribution.
• E2EE health: prekey fetch error rate, prekey exhaustion incidents, KDS cache hit ratio.
• Media: upload success rate, CDN (Content Delivery Network) cache-hit ratio, median upload latency, pre-signed URL expiry failures.

16. Interview Signals

#

What separates a strong answer

#

Common mistakes to avoid

#

1. Using a relational database as the primary message store — B-tree index maintenance creates write bottlenecks at 100 B messages/day. Cassandra’s LSM (Log-Structured Merge) tree and wide-row model are purpose-built for append-heavy workloads.

2. Storing decryptable messages on the server — breaks E2EE. The server should hold only ciphertext blobs it cannot decrypt. State this explicitly; interviewers at privacy-focused companies will probe this.

3. Encrypting group messages once per member per message — O(N²) for active group threads (N members × N messages). Sender Keys bring this to O(N) for key distribution and O(1) per subsequent message.

4. Conflating server ACK with delivery ACK — the single grey tick (server received) and double grey tick (recipient device received) are different signals with different latency profiles. The server ACK is synchronous; the device ACK requires a round-trip to the recipient.

5. Single Kafka topic without partitioning — all messages on one partition is a serialisation bottleneck. Partition by conversation_id to preserve ordering within each conversation without global serialisation.

6. Forgetting about multi-device — a single user can have 4 devices. Every delivery decision (online lookup, offline queue, push notification) must iterate over all devices. Every message encrypted for a contact must also be encrypted for the sender’s own linked devices.

17. Scaling Path

#

18. Summary

#

WhatsApp’s architecture is built around four core principles:

1. Persistent connections over polling — WebSocket gateways hold hundreds of millions of concurrent sockets, routed via a Redis registry, enabling sub-200 ms delivery for online recipients.

2. Ephemeral server storage — Cassandra holds undelivered messages transiently; on DELIVERED ACK the row is deleted, limiting legal exposure and storage cost.

3. End-to-end encryption by default — X3DH bootstraps sessions (no prior shared secret needed); Double Ratchet ensures forward secrecy and break-in recovery per message; Sender Keys amortise group encryption from O(N) per-message to O(1).

4. Decoupled media path — clients encrypt and upload media directly to object storage via pre-signed URLs, keeping the chat gateway message-only and avoiding bandwidth bottlenecks.

The delivery receipt system (Sent → Delivered → Read) is as architecturally interesting as the forward message path: it requires a reverse ACK flow from the recipient’s device, through the Message Service, back to the sender’s gateway — a fully bidirectional signalling system layered on top of the one-way message channel.

19. References

#

• WhatsApp Engineering Blog — End-to-End Encryption
• The Signal Protocol — Technical Overview
• WhatsApp — Two Billion Users and Counting
• Erlang at WhatsApp
• Cassandra Data Modelling for Chat
• QUIC Protocol — RFC 9000
• X3DH Key Agreement Protocol — Signal
• The Double Ratchet Algorithm — Signal