[!NOTE] Kafka consumer questions test whether you understand asynchronous processing and the failure cases around retries, duplicates, ordering, and lag.
🧭 At a Glance
| Area | What To Remember |
|---|---|
| Scaling | Use consumer groups. Kafka assigns partitions across consumers in the same group. |
| Offset commit | Commit only after processing succeeds if you want to avoid losing messages. |
| Duplicates | Assume at-least-once delivery and make processing idempotent. |
| Poison messages | Use bounded retries and a dead-letter queue. |
📌 Real Interview Prompt
Question: How would you design a Kafka consumer so that message processing is reliable?
✅ Short Answer
I would put consumers in a consumer group, process messages from assigned partitions, and commit offsets only after successful processing. Because Kafka consumers are usually at-least-once, I would make handlers idempotent using event IDs or business keys. For failures, I would use limited retries with backoff, then move poison messages to a dead-letter topic. I would monitor consumer lag, processing latency, retry rate, DLQ count, and rebalance behavior.
🏗️ Consumer Flow
Kafka Topic -> Consumer Group -> Process Message
| success -> commit offset
| retryable failure -> retry with backoff
| permanent failure -> DLQ + commit/skip safely
💬 Expandable Q/A
When should offsets be committed?
Commit offsets after processing succeeds. If you commit before processing and the consumer crashes, Kafka thinks the message is done even though work was lost.
How do you handle duplicate messages?
Make processing idempotent. Store processed event IDs, use unique constraints, or make updates naturally idempotent through business keys.
How do you handle poison messages?
Retry a limited number of times with backoff. After that, publish the message and error context to a dead-letter topic for investigation.
How do you preserve ordering?
Put events that require ordering on the same partition key. Kafka preserves order within a partition, not across all partitions.
⚠️ Common Mistakes
- Committing offsets before processing succeeds.
- Assuming exactly-once behavior without idempotency.
- Retrying forever and blocking a partition.
- Ignoring consumer lag.
- Using the wrong partition key for ordered workflows.
📝 Final Summary
A strong Kafka consumer answer includes consumer groups, partition-aware scaling, safe offset commits, idempotent processing, retry/backoff, DLQ, ordering trade-offs, and monitoring.