Kafka Interview Questions

A set of practical Kafka interview questions — the kind that separate "read the docs once" from "has actually run this in production."

Q: Does Kafka guarantee message ordering? Under what conditions? Only within a single partition, never across an entire topic. Two messages in different partitions can be consumed in either order relative to each other. To keep a set of related messages in order (e.g., every event for one orderId), you must use the same key for all of them, so they hash to the same partition — ordering is a property of the key/partition choice, not something Kafka gives you automatically across a whole topic.

Q: What triggers a consumer group rebalance, and what actually happens during one? A rebalance is triggered when a consumer joins or leaves the group (including being evicted for not calling poll() within max.poll.interval.ms, or a slow/crashed instance's session timing out), or when the topic's partition count changes. During a rebalance, partitions are reassigned among the group's remaining consumers, and — depending on the assignment strategy — processing on the affected partitions pauses until the new assignment is settled. Static group membership (group.instance.id) and cooperative sticky assignors reduce how disruptive this is by avoiding a full stop-the-world reassignment for routine restarts.

Q: Why does Kafka default to at-least-once delivery, and what does that mean for consumer design? Because a crash can happen between processing a record and committing its offset, in either order — commit-before-process risks losing it, process-before-commit risks reprocessing it on restart. Kafka's client APIs are built around this trade-off rather than eliminating it, so consumers need to be idempotent: processing the same message twice (e.g., an upsert keyed by a unique event ID) should produce the same end state as processing it once.

Q: What's the practical difference between acks=0, acks=1, and acks=all? acks=0 doesn't wait for any broker acknowledgment — fastest, but a message can be silently lost with no error at all. acks=1 waits only for the partition leader, which can still lose the message if the leader fails before followers replicate it. acks=all waits for every in-sync replica to acknowledge, surviving any single broker failure at the cost of higher latency — it's the setting to use whenever losing a message is actually unacceptable.

Q: What's the difference between log retention and log compaction? Retention deletes records after a configured time or size limit, regardless of their content — appropriate for event streams where only recent history matters. Compaction instead keeps only the latest record for each key indefinitely, discarding older values for the same key — appropriate for topics acting as a changelog of current state (e.g., Kafka Streams' internal state-store topics, or a topic representing "current customer address" rather than "every address change ever").

Q: How does Kafka achieve much higher throughput than a traditional message broker? Several design choices compound: writes are sequential appends to disk (fast, even on spinning disks, unlike random-access I/O), consumers use zero-copy transfer (sendfile) to stream bytes straight from the page cache to the network socket without copying through user space, producers batch and optionally compress records before sending, and partitions let both producers and consumers parallelize across a cluster instead of funneling everything through one process.

Q: Is Kafka's "exactly-once" processing really exactly once? Not in the strictest sense — it's more accurately described as "effectively once" within Kafka's own boundary. An idempotent producer prevents duplicate broker-side writes caused by the producer's own retries, and transactions make a read-process-write cycle (consume, produce, and commit the consumer offset) atomic as seen by consumers reading with read_committed — but any side effect a consumer performs outside Kafka itself, like calling an external API, is never covered by that guarantee and still needs its own idempotency.

Q: Why would a production system introduce a schema registry instead of just serializing messages as JSON? Once a topic is read by multiple independently-deployed consumer teams over a long lifetime, a producer team can silently rename or drop a field a consumer depends on, with no warning until something breaks in production. A schema registry stores every version of a topic's schema, encodes each message with a small ID referencing the exact schema it was written with, and enforces a compatibility mode (like backward-compatible) on every new schema version, catching a breaking change at registration time instead of at runtime in some other team's consumer.

Q: What's the single most useful metric for spotting a Kafka consumer in trouble, and what does it actually measure? Consumer lag — the difference between the latest offset in a partition and a consumer group's last committed offset for it, i.e., how far behind the live end of the log that group currently is. Rising, non-recovering lag is usually the earliest visible sign that a consumer can't keep up with production rate, well before anything actually throws an error, which is why it's typically alerted on directly rather than discovered after users notice stale data.

Q: How do you decide on a topic's replication factor and partition count for production, rather than just picking Kafka's defaults? Replication factor is chosen for the failure tolerance a topic actually needs — 3 is the standard production default, tolerating one broker failure while acks=all writes still succeed against min.insync.replicas=2. Partition count is chosen from expected peak throughput and the number of consumers you want to parallelize across in the busiest consumer group, with some headroom to grow — since partition count can only be increased later, never decreased, and over-provisioning it "to be safe" still carries real per-partition overhead on the brokers even while mostly idle.