← All Problems

Messenger App

Messenger App

Real-timeMessagingDistributed Systems

0) Problem Restatement

Design a real-time messaging system (like WhatsApp / Messenger) that supports 1:1 and group chat, message delivery guarantees, presence, typing indicators, push notifications, and scales to hundreds of millions of users. Focus on architecture, delivery semantics, storage, and scaling.

1) Requirements

1.1 Functional

  • 1:1 and group text messaging (send/receive).
  • Delivery receipts: sent, delivered, read.
  • Presence (online/offline) and typing indicators.
  • Push notifications for offline devices.
  • Message history sync across devices.
  • Support attachments (images, video) — store in object store.

1.2 Non-Functional

  • Low latency: real-time <200–500ms for active sessions.
  • High throughput: hundreds of millions of users, billions of messages/day.
  • Durable storage for history.
  • Strong privacy (optionally E2EE).
  • High availability, region-distributed.

2) High-Level Architecture

2.1 Overview (what components connect)

  • Clients (mobile/web) ↔ Gateway (Auth, TLS) ↔ Connection Layer (WebSocket / MQTT)
  • Connection Layer ↔ Routing / Session Service ↔ Delivery Workers
  • Delivery Workers ↔ Persistent Message Store (Primary DB + Cold store)
  • Delivery Workers ↔ Fanout / Push Service (APNs / FCM)
  • Presence Service (Redis) for online status
  • Attachment Service ↔ Object Store (S3) + CDN
  • Control Plane: Auth Service, User/Profile Service, Group Service, ACLs

2.2 Flow Diagram

Architectural Diagram Locked

3) Core Components (what & why)

Client

  • Maintains persistent connection (WebSocket or MQTT) to receive messages in real-time.
  • Falls back to push notifications when offline.
  • Syncs message history and state (read receipts, deletions).

API Gateway

  • TLS termination, authentication (OAuth/JWT), rate limiting, routing to connection brokers or REST endpoints.

Connection Layer (WebSocket / MQTT Brokers)

  • Handles millions of concurrent TCP connections.
  • Routes frames to Routing/Session Service; maintains per-connection metadata (node, heartbeat).
  • Use clustered brokers (e.g., MQTT brokers, custom WS gateway) with sticky sessions.

Routing / Session Service

  • Knows which gateway node holds which user session(s).
  • Accepts incoming messages and routes to Delivery Workers or directly to target session node.
  • Maintains session registry (user_id → gateway_node(s)) in a fast store (Redis / Consul).

Delivery Worker / Router

  • Decouples ingest from persistence/delivery.
  • Publishes incoming messages to primary store, adds to topic keyed by conversation_id, and fans out to subscribers.
  • Handles retries, ordering, deduplication, and acknowledgment tracking.

Message Queue / Stream (Kafka)

  • Durably buffer messages; keyed by conversation_id for per-conversation ordering.
  • Enables replays, backpressure, multiple consumers (archiving, analytics).

Primary DB / Message Store

  • Store message metadata and pointers. Options:
    • Cassandra / DynamoDB / Scylla for multi-region, high-write throughput.
    • Relational DB for small-scale / metadata.
  • Store message: {message_id, conversation_id, sender_id, ts, body_ref or body, status_flags}.

Presence Service

  • Hot store (Redis) to track online/offline status and last-seen.
  • Publishes presence changes for subscribers.

Typing Indicator Service

  • Lightweight ephemeral events broadcasted to conversation participants via same routing infra.

Push Service

  • Converts server-side events to APNs / FCM for offline devices.
  • Manages tokens, platform differences, payload size, and rate-limits.

Attachment Service & Object Store

  • Upload endpoint (signed URL) → object store (S3) → thumbnail generation → CDN for delivery.
  • Messages reference attachment URLs.

Control Plane (Auth, Groups)

  • Manages user profiles, group membership, ACLs, and access tokens.

4) Data Model (minimal)

Message

sql
Message {
  message_id STRING PRIMARY KEY,
  conversation_id STRING,
  sender_id STRING,
  ts_ms BIGINT,
  body_ref STRING,   -- small text inline, larger stored in blob store
  attachments [url...],
  status_flags INT,  -- bits: sent/delivered/read
  seq_no BIGINT
}

Conversation / Thread

sql
Conversation {
  conversation_id STRING,
  type ENUM('one_to_one','group'),
  participants [user_id...],
  created_at TIMESTAMP,
  last_activity_ts
}

Presence (Redis)

  • Key: presence:{user_id} → {node_id, last_seen, status}

Delivery State (optional)

  • Key: conv:{conversation_id}:cursor → last_seq_no persisted / per-device cursor

5) Key Flows

5.1 Send Message (Active path)

  1. Client sends message over WS with local client_msg_id and seq_no.
  2. Gateway authenticates and forwards to Routing/Session Service.
  3. Routing service forwards to Delivery Worker; Delivery Worker:
    • Assigns server-side message_id, timestamp.
    • Persists message to Primary DB (or logs to Kafka first and then persists).
    • Publishes to conversation topic on Kafka (keyed by conversation_id).
  4. Delivery Worker fans out message to active participant sessions via Connection Layer (node lookup).
  5. Recipients' clients ACK reception; server marks delivered.
  6. If recipient offline, Push Service triggers APNs/FCM.

5.2 Delivery Receipts & Read Receipts

  • Client sends receipt events (delivered/read) back to server.
  • Server updates message flags and optionally notifies sender.

5.3 Message Sync (new device / reconnect)

  • Client requests conversation history since last_seq_no.
  • Server returns persisted messages from Primary DB or cold store.
  • For long histories, provide pagination.

5.4 Attachments

  • Client requests signed upload URL from Attachment Service.
  • Uploads to S3; Attachment Service creates thumbnails, stores metadata.
  • Message references attachment URL.

5.5 Presence & Typing

  • Presence: client heartbeats to Presence Service; presence changes broadcast to friends.
  • Typing: ephemeral events routed to active participants (no DB persist).

6) Deep Dive A: Message Delivery Guarantees & Ordering (~8–10 mins)

Delivery Semantics Options

  • At-most-once: simplest, may lose messages.
  • At-least-once: ensure message delivered, may cause duplicates — requires dedup logic.
  • Exactly-once (per recipient): desirable but complex; achieve via idempotency + dedup.
  • Producer-side idempotency: client attaches client_msg_id. Server persist operation uses idempotency to avoid duplicate writes.
  • Kafka stream + consumer offsets: Use per-conversation partitioning (key = conversation_id) to preserve ordering within a conversation.
  • Deduplication and idempotency at consumer: track seen client_msg_id or message_id in a small cache (Redis) to drop duplicates.
  • Per-device sequencing: maintain per-device cursors to resume from last_seq_no reliably.

Ordering

  • Ordering guaranteed per-conversation by using the same Kafka partition (keyed by conversation_id). For group chat, a single partition may become a hotspot — consider sharding by conversation_id + time windows or virtual partitions with ordering guarantees at application level.

Acks & End-to-End

  • Use at-least-once delivery with idempotent writes and ACK semantics:
    • Server sends ACK to sender when persisted.
    • Server marks delivered when recipient ACKs.
    • Sender sees sent/delivered/read states updated.

7) Deep Dive B: Scaling, Sharding & Fanout (~8–10 mins)

Scaling Writes & Reads

  • Partitioning: partition message streams by conversation_id. Use consistent hashing to distribute partitions across Kafka brokers and delivery workers.
  • Hot conversation mitigation: large groups can overload a single partition. Strategies:
    • Use group-specific optimizations: tiered delivery (batch updates), ephemeral sampling, or multiple partitions per large group + merge ordering at consumers.
    • Offload to push-based summary updates for very large groups.

Connection & Socket Scaling

  • Use many WebSocket gateway nodes behind LB. Keep sticky sessions. Store mapping user_id → node_id in Redis (session registry).
  • Cross-node fanout: if recipient on different node, the delivery worker forwards message to that node (via internal RPC/queue) which then pushes to client.

Fanout Strategies

  • Direct push: push to each active session (works for small groups / 1:1).
  • Interest topics: consumers subscribe to conversation topics (WS gateways subscribe to Kafka topics for conversations they have sessions for). This reduces central fanout work.
  • Batching & compression: group messages into frames for same recipient; compress payload for large bursts.

Storage Scaling

  • Use wide-column DB (Cassandra / DynamoDB) for high-write throughput, partitioned by conversation_id and time.
  • Use TTL or cold archiving for older messages (move to object store).

8) Offline & Push Notification Handling (~4 mins)

  • When recipient offline, queue message for push.
  • Push payload should be minimal (summary) to avoid leaking content; for privacy, fetch server-side or use encrypted payloads.
  • Respect platform rate-limits and backoff when push failures occur.

9) Security & Privacy (~3 mins)

  • TLS for transport.
  • Auth tokens (short-lived).
  • Optionally End-to-End Encryption (E2EE):
    • Server routes only encrypted blobs; cannot read contents.
    • Key management (Signal protocol, OT is complex for groups).
    • Trade-offs: server-side features (search, spam detection, indexing) limited with E2EE.
  • Content moderation via metadata, client-side reporting, and optional server-side scanning for non-E2EE deployments.

10) Failure Modes & Recovery (~3 mins)

  • Gateway node crash: sessions reconnect; session registry updated; undelivered messages replayed from Kafka or DB.
  • Delivery worker crash: Kafka ensures messages retained; consumers resume from offsets.
  • DB outage: degrade to readonly for history, continue in-memory buffering and accept messages to Kafka with later persistence.
  • Network partitions: favor availability for messaging delivery; reconcile using sequence numbers and reconciliation jobs.

11) Observability & SLOs (~2 mins)

  • Metrics: message ingests/s, delivery latency p50/p95/p99, queue backlog, active connections, ack rates, push success rates.
  • Tracing: propagate trace ids through send → persist → fanout → ack.
  • SLO examples: p95 in-app delivery <500ms, message durability 99.999% (depends on storage SLAs).

12) Trade-offs & Alternatives (~2 mins)

  • E2EE vs server-side features: E2EE maximizes privacy but limits server functionality.
  • Relational vs NoSQL store: NoSQL (Cassandra/DynamoDB) favored for writes and horizontal scaling.
  • Push vs pull for large groups: pull/chunked sync for huge groups reduces server cost.

13) Interview Time Allocation (45 min)

  • 5 min: Clarify requirements & scope
  • 10 min: High-level architecture drawing & walkthrough
  • 10 min: Key flows (send/receive/sync/push)
  • 15 min: Deep dives (Delivery guarantees & Scaling/Fanout)
  • 5 min: Failure modes, security, summary

14) Summary

  • Core pieces: Connection layer (WS/MQTT), Routing/Session service, Delivery workers, Message stream (Kafka), Primary message store (Cassandra/DynamoDB), Presence (Redis), Push for offline.
  • Use per-conversation partitioning for ordering; idempotency + deduplication for correctness; and efficient fanout patterns for scale.
  • Consider E2EE for privacy with trade-offs. Monitor key metrics and design for graceful degradation during outages.
Take Mock Interview for this Problem with a FAANG Engineer
ResumeSkool Logo

No-Nonsense AI Resume
Building Platform for Free

©2026 - ResumeSkool

Product

Engage

Earn Money