🚀 Apache Kafka: A Beginner-Friendly Guide to Event Streaming

1. What Is Apache Kafka?

Apache Kafka is a distributed event-streaming platform used to build real-time data pipelines and streaming applications. It allows systems to publish, store, and process streams of events reliably and at scale.

Kafka is commonly used for:

• Real-time analytics

• Log aggregation

• Event-driven microservices

• Data streaming between systems

2. Why Use Kafka?

Traditional systems process data in batches, which introduces delays. Kafka enables real-time data flow , allowing applications to react instantly to events.

Example Use Case

When a user places an order:

Order Service → Kafka → Inventory / Payment / Notification Services

Each service reacts independently without tight coupling.

3. Core Kafka Concepts

🔹 Producer

Sends (publishes) messages to Kafka topics.

🔹 Consumer

Reads (subscribes to) messages from topics.

🔹 Topic

A category or stream of messages.

🔹 Partition

A topic is split into partitions for scalability and parallelism.

🔹 Broker

A Kafka server that stores data and serves clients.

4. Kafka Architecture Overview

Kafka runs as a cluster of brokers. Data is:

• Written to topics

• Split into partitions

• Replicated across brokers for fault tolerance

Kafka guarantees:

• High throughput

• Message durability

• Horizontal scalability

5. Installing Kafka (Local Setup)

Prerequisites

• Java 8 or higher

Start Zookeeper (Kafka ≤2.x)

bin/zookeeper-server-start.sh config/zookeeper.properties

Start Kafka Broker

bin/kafka-server-start.sh config/server.properties

6. Creating a Topic

bin/kafka-topics.sh --create \
--topic orders \
--bootstrap-server localhost:9092 \
--partitions 3 \
--replication-factor 1

7. Producing & Consuming Messages

Produce Messages

bin/kafka-console-producer.sh \
--topic orders \
--bootstrap-server localhost:9092

Consume Messages

bin/kafka-console-consumer.sh \
--topic orders \
-…