- What Is Apache Kafka?
Apache Kafka is a distributed event-streaming platform used to build real-time data pipelines and streaming applications. It allows systems to publish, store, and process streams of events reliably and at scale.
Kafka is commonly used for:
• Real-time analytics
• Log aggregation
• Event-driven microservices
• Data streaming between systems
- Why Use Kafka?
Traditional systems process data in batches, which introduces delays. Kafka enables real-time data flow , allowing applications to react instantly to events.
Example Use Case
When a user places an order:
Order Service → Kafka → Inventory / Payment / Notification Services
Each service reacts independently without tight coupling.
- Core Kafka Concepts
🔹 Producer
Sends (publishes) messages to Kafka topics.
🔹 Consumer
Reads (subscribes to) messages from topics.
🔹 Topic
A category or stream of messages.
🔹 Partition
A topic is split into partitions for scalability and parallelism.
🔹 Broker
A Kafka server that stores data and serves clients.
- Kafka Architecture Overview
Kafka runs as a cluster of brokers. Data is:
• Written to topics
• Split into partitions
• Replicated across brokers for fault tolerance
Kafka guarantees:
• High throughput
• Message durability
• Horizontal scalability
- Installing Kafka (Local Setup)
Prerequisites
• Java 8 or higher
Start Zookeeper (Kafka ≤2.x)
bin/zookeeper-server-start.sh config/zookeeper.properties
Start Kafka Broker
bin/kafka-server-start.sh config/server.properties
- Creating a Topic
bin/kafka-topics.sh --create \
--topic orders \
--bootstrap-server localhost:9092 \
--partitions 3 \
--replication-factor 1
- Producing & Consuming Messages
Produce Messages
bin/kafka-console-producer.sh \
--topic orders \
--bootstrap-server localhost:9092
Consume Messages
bin/kafka-console-consumer.sh \
--topic orders \
-…
Preview this lesson for free
Sign in to continue reading the full post.