10. Message Queues & Event Streaming

The Problem with Synchronous Processing

Some operations are inherently slow: PDF generation, video transcoding, email fanout to millions of users, payment processing with third-party gateways. If you perform this work inside the HTTP request path, the user stares at a spinner for 30 seconds and your server capacity collapses under load.

Real-World Example: YouTube Video Uploads

When you upload a video to YouTube, you get a success message almost immediately. But in the background, YouTube''s pipeline is transcoding your video into dozens of formats and resolutions (144p through 4K, VP9, AV1, H.264). This takes minutes, even hours for long videos. The upload API just enqueues the transcoding job and returns—the heavy lifting happens asynchronously via message queues.

The Async Pattern

1. API validates the request and enqueues a job/event to the message queue.
2. API returns HTTP 202 Accepted immediately to the user.
3. Background workers consume and process jobs at their own pace.

[!NOTE] Queues act like shock absorbers. They protect your live API from traffic spikes by buffering work and letting workers process at a sustainable rate. During Black Friday, Amazon enqueues order processing events rather than trying to handle everything synchronously.

Traditional Queues vs Event Streams

Message Queues (RabbitMQ, AWS SQS, ActiveMQ)

• Model: Each message is processed exactly once by one consumer, then deleted.
• Retention: Messages are removed after acknowledgment. Failed jobs retry or move to a Dead Letter Queue (DLQ).
• Best for: Background jobs, email/SMS notifications, retry logic, rate smoothing.

Stripe''s Example: When you make a payment, Stripe''s webhook system uses message queues to reliably deliver payment events to merchant servers. If a merchant''s server is down, the queue retries with exponential backoff for up to 72 hours.

Event Streaming (Apache Kafka, AWS Kinesis, Pulsar)

• Model: Durable, append-only event log. Messages persist and multiple consumers can read independently at their own offsets.
• Retention: Events remain for a configurable duration (days, months, or forever). Consumers track their own position.
• Best for: Analytics pipelines, Change Data Capture (CDC), event-driven architectures, real-time stream processing.

LinkedIn (Kafka''s Birthplace): LinkedIn created Apache Kafka to handle their massive data pipeline. Today, LinkedIn processes over 7 trillion messages per day through Kafka—tracking user activity, search queries, and recommendations in real-time across hundreds of consumers.

Pub/Sub Pattern

In the Publish/Subscribe pattern, publishers emit events without knowing who will consume them. Multiple subscribers can independently listen to the same topic and process events for their own purposes.

Example: When a new user signs up, a single "UserCreated" event might trigger:

• The Email Service sends a welcome email.
• The Analytics Service records the signup.
• The Recommendation Service initializes suggestions.
• The CRM Service creates a customer record.

No service knows about the others. This extreme decoupling is why event-driven architectures scale so well organizationally.

Delivery Semantics and Idempotency

• At-most-once: Messages may be lost, but never duplicated. Acceptable for non-critical analytics.
• At-least-once: No message loss, but duplicates are possible. Most common in production.
• Exactly-once: Extremely hard to achieve. Usually approximated with idempotent consumers + transactional outbox pattern.

Critical design principle: Always make consumers idempotent—processing the same message twice should produce the same final state. Use unique event IDs and database constraints to detect and skip duplicates.

Operational Must-Haves

• Timeouts + retries with exponential backoff—avoid retry storms that amplify failures.
• Dead Letter Queues (DLQ)—capture poison messages that consistently fail, for manual investigation.
• Backpressure and rate limiting—when consumers lag, slow down producers rather than crashing consumers.
• Observability—correlation IDs across services, consumer lag metrics, processing time dashboards. Uber uses custom tooling to monitor millions of Kafka consumer offsets in real-time.

[!TIP] Interview tip: Whenever you see "send notifications to users" or "process this in the background" in a system design question, immediately mention a message queue. It shows you understand async processing, which is fundamental to every production system at scale.