Topics, Partitions & Offsets

Topics and partitions#

A topic is a named stream of messages (orders, payments, user-signups). Under the hood, a topic is split into one or more partitions — each partition is its own ordered, append-only log, and each is replicated across brokers independently:

Topic "orders"  (3 partitions, replication factor 3)

Partition 0: [msg0][msg1][msg2][msg3] ...   <- leader on broker 1, replicas on 2 & 3
Partition 1: [msg0][msg1][msg2]       ...   <- leader on broker 2, replicas on 1 & 3
Partition 2: [msg0][msg1][msg2][msg3][msg4] <- leader on broker 3, replicas on 1 & 2

Every message written to a partition gets a monotonically increasing offset — its position in that partition's log — which is how Kafka and consumers both refer to "where" a message is, instead of by content or timestamp.

Partitioning is what enables parallelism#

A single partition can only be read by one consumer at a time within a given consumer group — that's the mechanism that guarantees ordering (more on that below). So the number of partitions on a topic is the hard ceiling on how many consumers in one group can do useful work on it simultaneously:

• 3 partitions, 3 consumers in the group → each consumer handles exactly one partition, full parallelism.
• 3 partitions, 5 consumers in the group → 3 consumers get one partition each, 2 sit completely idle — Kafka never splits a single partition across multiple consumers in the same group.
• 3 partitions, 1 consumer → that one consumer reads all three partitions, in some interleaved order, sequentially.

This is why partition count is a capacity-planning decision, not an afterthought: it caps your consumer-side parallelism for that topic, permanently, until you add more partitions (see the caveat on that below).

Consumer groups and offset tracking#

A consumer group is a named set of consumers cooperating to read a topic, identified by a group.id. Kafka assigns each partition to exactly one consumer within a group, and tracks the last-committed offset per group, per partition — internally, in a topic called __consumer_offsets. That's what lets multiple independent groups replay the exact same topic completely independently of one another:

Consumer Group "inventory-service"  (offset=2 on partition 0) --> reading live, caught up
Consumer Group "fraud-detection"    (offset=0 on partition 0) --> replaying from the very start

Both groups see the same messages; neither affects the other's progress. This is the concrete mechanism behind "multiple independent teams can consume the same event stream" from the System Design track's Message Queues & Event-Driven Architecture page.

Partition keys and the ordering guarantee#

Producers send a (key, value) pair. If a key is given, Kafka hashes it to deterministically pick a partition — every message with the same key always lands on the same partition:

key="order-482" --> hash --> always partition 1
key="order-901" --> hash --> always partition 0

This is the detail that trips up more engineers than any other Kafka concept: Kafka only guarantees ordering within a single partition — never across an entire topic. If order-482's events (Created, PaymentReceived, Shipped) all use order-482 as the key, they're guaranteed to arrive at a consumer in that order, because they're all in the same partition. But order-482's events and order-901's events, sitting in different partitions, have no ordering relationship to each other whatsoever — a consumer reading both partitions could see them interleaved in any order.

If you don't specify a key, Kafka distributes messages round-robin (or by a sticky-batching strategy, depending on client version) across partitions for better load balancing — which means you get no ordering guarantee at all between any two messages. Only pick "no key" when you've confirmed you genuinely don't need order between any two related messages.

Choosing a partition count and key#

• Key selection: pick a key that groups everything that needs to stay in order, and nothing more. Keying by orderId keeps one order's events in sequence; keying by something coarser like customerId also works but reduces parallelism if one customer dominates traffic (a "hot partition").
• Partition count: provision for your expected peak number of parallel consumers in your busiest consumer group, with room to grow — but know that you can only increase a topic's partition count later, never decrease it.
• The re-keying trap: increasing partition count changes the hash-to-partition mapping for every key, since it's computed as hash(key) % partition_count. A key that used to always land on partition 1 can land on a different partition after the change. Existing ordering guarantees for that key are only preserved going forward from the change, not retroactively — plan partition count up front for anything where historical ordering matters.

Common mistakes#

• Assuming Kafka guarantees ordering across an entire topic — it only does so within one partition. Two messages in different partitions can be consumed in either order.
• Not using a key for messages that need to stay in order relative to each other (e.g., all events for one entity), then being surprised when a consumer sees them out of sequence.
• Increasing a topic's partition count on a live system without accounting for the fact that it changes which partition every key maps to going forward.
• Over-provisioning partition count "to be safe" — every partition adds overhead (open file handles on brokers, more replication traffic, longer leader-election time during a failover) even when nothing is consuming from it in parallel.