Message Queue and Kafka Interview Questions
Last updated 2026-11-19
Message queues show up in design rounds whenever work is slow, bursty or needs to be decoupled, and Kafka is the system interviewers name most. The questions go past definitions: how Kafka preserves order, what acks=all actually guarantees, why duplicates occur and how many partitions a topic needs. This page answers these with examples, including a partition-sizing calculation and short client code.
It builds on distributed systems questions and pairs with microservices questions and system design questions. For a simpler stream use, see Redis questions. The round plan is in how to crack a backend interview.
| Systems covered | Kafka, RabbitMQ, general queue concepts |
|---|---|
| Core ideas | Partitions, consumer groups, delivery semantics |
| Level | Medium to hard |
Queue versus log
A traditional queue such as RabbitMQ or SQS delivers each message to one consumer and removes it once acknowledged. Kafka is a distributed commit log: messages stay for a retention period, any number of consumer groups read them independently at their own offsets, and a group can replay by moving its offset back. That difference drives most of the trade-offs.
Client code that matches the answers
With the confluent-kafka Python client, a producer that waits for all in-sync replicas and avoids duplicates on retry, and a consumer that commits an offset only after processing, look like this.
from confluent_kafka import Producer, Consumer
p = Producer({"bootstrap.servers": "localhost:9092",
"acks": "all", "enable.idempotence": True})
p.produce("orders", key="user42", value=b'{"order": 1}') # same key, same partition
p.flush()
c = Consumer({"bootstrap.servers": "localhost:9092", "group.id": "billing",
"auto.offset.reset": "earliest", "enable.auto.commit": False})
c.subscribe(["orders"])
while True:
msg = c.poll(1.0)
if msg is None or msg.error():
continue
handle(msg.value()) # your idempotent processing
c.commit(message=msg) # commit after success: at-least-onceMessage Queue and Kafka Interview Questions: questions with worked answers
1 Why use a message queue?
easyA queue decouples producers from consumers in time and rate. A web request can enqueue work, such as sending an email or resizing an image, and respond immediately; workers process at their own speed. It absorbs traffic spikes, allows retries, lets you scale consumers independently and isolates failures, since a down consumer just lets messages accumulate. The costs are eventual results, duplicate handling and one more system to operate.
2 Explain the core Kafka concepts: topic, partition, offset, broker, producer, consumer group.
easyA topic is a named stream of records. It is split into partitions, each an ordered, append-only log; each record in a partition has an increasing offset. Brokers are the servers that store partitions. Producers append records, choosing the partition by key hash or round robin. Consumers read by offset, and consumers sharing a group id divide the topic's partitions among themselves, so each partition is read by one consumer in the group at a time. Different groups each get every record.
3 How does Kafka guarantee ordering?
mediumOnly within a partition. Records with the same key hash to the same partition and so are read in the order written; there is no ordering guarantee across partitions. If you need all events for one order or user in sequence, use that id as the key. Beware that retries can reorder messages within a partition if a producer allows several in-flight requests without idempotence; enabling the idempotent producer keeps order with retries.
4 Explain replication, the ISR and the acks setting.
hardEach partition has a leader and follower replicas, set by the replication factor. The in-sync replicas (ISR) are the followers that are caught up with the leader. With acks=0 the producer does not wait, acks=1 waits for the leader to write, and acks=all waits for all current ISR members. Combined with min.insync.replicas=2 and replication factor 3, a write is acknowledged only when at least two replicas have it, so one broker can fail with no loss of acknowledged data; if the ISR shrinks below two, producers get an error rather than risking loss. If the leader dies, a new leader is elected from the ISR.
5 What delivery semantics does Kafka support, and how do duplicates arise?
hardAt-most-once: commit the offset before processing, so a crash loses messages. At-least-once: process then commit, so a crash between them causes reprocessing, giving duplicates. Duplicates can also come from producer retries after a lost acknowledgement. The idempotent producer (enable.idempotence=true) removes duplicates from retries by sequence numbers per partition. Exactly-once within Kafka uses transactions that write output records and consumer offsets atomically; for external side effects you still need idempotent consumers. See distributed systems questions.
6 What happens during a consumer group rebalance?
mediumWhen a consumer joins, leaves or crashes, or partitions change, the group coordinator reassigns partitions among members. Under the classic eager protocol all consumers stop and give up their partitions, causing a pause; cooperative rebalancing moves only the affected partitions. Frequent rebalances, caused by slow processing exceeding max.poll.interval.ms or unstable consumers, hurt throughput, so keep processing per poll bounded and tune session timeouts.
7 How do offsets and commits work? What is consumer lag?
mediumA consumer group stores the next offset to read for each partition in an internal topic, __consumer_offsets. Auto-commit does this periodically, which can lose or duplicate messages around crashes; manual commit after processing gives control. Consumer lag is the difference between the latest offset in a partition and the group's committed offset. Growing lag means consumers are slower than producers and is the main metric to alert on.
8 Size the partition count for 100 MB/s of throughput.
hardMeasure per-partition throughput for a producer, p, and for a consumer, c. Suppose p is 10 MB/s and c is 20 MB/s per partition. The target T is 100 MB/s, so you need at least max(T/p, T/c) = max(10, 5) = 10 partitions. Add headroom for growth, say 12 to 15. Partitions also bound consumer parallelism: a group can use at most as many active consumers as partitions. More partitions add open files, replication overhead and longer recovery, and changing the count later breaks key-to-partition mapping, so plan ahead.
9 Why is Kafka fast?
mediumIt writes and reads sequentially on disk, which is fast and well served by the operating system page cache. Records are batched and compressed, so network and disk operations handle many records at once. Zero-copy transfer with sendfile moves data from the page cache to the network socket without copying it through application memory. Partitioning spreads load across brokers, and consumers pull at their own pace, so brokers do not track delivery per message.
10 What are retention and log compaction?
mediumRetention deletes old segments after a time (for example seven days) or a size limit, regardless of whether they were consumed. Log compaction instead keeps at least the latest record for each key and discards older ones, so a topic works as a changelog of current state per key, useful for rebuilding caches and tables. A record with a null value, a tombstone, eventually deletes the key. Compaction preserves offsets, and the retained records keep their original order.
11 Kafka versus RabbitMQ: how do you choose?
mediumRabbitMQ is a broker for flexible routing: exchanges (direct, fanout, topic, headers) route messages to queues, consumers acknowledge messages, and per-message features such as TTL, priority and dead-letter exchanges are built in. It suits task queues and complex routing at moderate throughput. Kafka is a distributed log for very high throughput, retention, replay and stream processing, with ordering per partition. Choose RabbitMQ for work distribution and routing; choose Kafka for event streaming, audit trails and multiple independent consumers of the same events.
12 What is a dead letter queue and how do you handle poison messages?
mediumA dead letter queue (DLQ) receives messages that cannot be processed after a set number of retries, so one bad message does not block the rest. The consumer catches the failure, retries with backoff for transient errors, and after the limit publishes the message with error details to the DLQ for inspection and reprocessing. In Kafka there is no built-in DLQ, so consumers publish failed records to a separate topic. Alert on DLQ growth.
13 How do you handle backpressure when consumers are slower than producers?
mediumQueues absorb temporary imbalance, but a sustained gap grows the backlog. Options: scale out consumers up to the partition count, make processing faster or batch it, add partitions in advance, rate limit or shed load at the producer, and use bounded queues so producers block or fail instead of exhausting memory. Monitor lag and queue depth with alerts, and prioritise or drop low-value work under overload.
14 What replaced ZooKeeper in Kafka?
mediumKRaft mode. Older Kafka versions kept cluster metadata and controller election in ZooKeeper, a separate system to run. KRaft moves metadata management into Kafka itself using a Raft-based quorum of controllers, simplifying deployment, scaling metadata and speeding controller failover. Recent Kafka releases run in KRaft mode only, so new clusters should use it.
Frequently asked questions
Is Kafka a message queue?
It can serve as one, but it is a distributed log. Messages are retained and re-readable, unlike a classic queue where acknowledged messages disappear.
How many consumers can read one partition?
Within a consumer group, one at a time. Different groups can each read the same partition independently.
Can Kafka lose messages?
Yes, if misconfigured, for example acks=1 with a failing leader or unclean leader election. With acks=all, replication factor 3 and min.insync.replicas=2, acknowledged messages survive a single broker failure.
What is the difference between a queue and a topic?
A queue delivers each message to one consumer. A topic with publish-subscribe delivers each message to every subscribing group.
Related guides
More in CS Fundamentals Interview Questions
- Microservices Interview Questions and Answers
- MongoDB Interview Questions and Answers
- Network Security Interview Questions: Cryptography, Firewalls and Attacks
- Normalization Interview Questions with Worked Examples
- NoSQL Interview Questions and Answers
- OOPs Interview Questions and Answers
- Operating Systems Interview Questions with Worked Answers
- OSI Model Interview Questions: Layers, PDUs, Devices and Encapsulation
- Paging and Virtual Memory Interview Questions with Numericals
- Process and Thread Interview Questions with fork() Output Problems
- Process Synchronization Interview Questions: Semaphores, Mutex and Classic Problems
- Redis Interview Questions and Answers