Kafka in Production: Reference Architecture

A reference architecture#

A real production Kafka deployment looks considerably more involved than the single-broker Docker Compose setup from this track's introduction — brokers spread across multiple availability zones for fault tolerance, plus the supporting services that make schemas and data movement manageable at scale:

                     ┌───────────────────────────────┐
                     │      Kafka cluster (KRaft)     │
  Producers  ------> │   Broker 1 (AZ-a)               │  <---- Consumers
  (services)         │   Broker 2 (AZ-b)               │        (services)
                     │   Broker 3 (AZ-c)               │
                     └───────────────────────────────┘
                              |                |
                     Schema Registry      Kafka Connect
                     (Avro schemas)     (source/sink connectors)

Replication factor: choosing it deliberately#

The introduction page used --replication-factor 1 for local development, explicitly flagged as unsafe for anything beyond that. In production, replication factor 3 is the standard default for anything that matters, paired with min.insync.replicas=2 and a producer using acks=all: with those settings, up to one broker can be down and writes still succeed (2 of the 3 replicas acknowledge), while losing a second broker causes writes to be rejected outright rather than silently continuing with less redundancy than configured — an intentional trade-off favoring consistency over availability once fault tolerance is already this degraded.

Partition count planning#

The topics-and-partitions page in this track already established that partition count caps consumer-side parallelism per group. In production, plan partition count around expected peak throughput and the largest consumer group you expect to run against the topic, with some room to grow — while remembering two constraints from that same page: partition count can only be increased later, never decreased, and increasing it changes which partition every key maps to going forward.

The mistake to avoid runs the other direction too: every partition carries a real, ongoing cost on the brokers — an open file handle, its own share of replication traffic, and a longer total leader-election time during a failover as the number of partitions the cluster has to reassign grows. Provisioning thousands of partitions "to be safe" on a topic that will only ever have a handful of consumers pays that overhead indefinitely for no real benefit.

Multi-datacenter and multi-region considerations#

Spreading a single cluster's brokers across multiple availability zones within one region — what the replication factor table above assumes — tolerates a whole AZ going down. A separate region entirely is a different problem: serving geographically distributed consumers with lower latency, or maintaining a disaster-recovery target, needs cross-region replication, most commonly via MirrorMaker 2 (built on top of Kafka Connect), which continuously replicates topics from a source cluster to a target cluster in another datacenter or region.

Region A (primary)                       Region B (DR)
┌─────────────────┐    MirrorMaker 2     ┌─────────────────┐
│  Kafka cluster    │  ----------------->  │  Kafka cluster    │
│  topic: orders    │   (continuous         │  topic: orders    │ (mirrored,
└─────────────────┘    replication)       └─────────────────┘  independent offsets)

The detail that catches teams off guard during an actual regional failover: a mirrored topic in the target cluster is not offset-identical to the source. A consumer that fails over from the primary region to the DR region cannot simply "resume at the same offset" — the two clusters' offsets for the same logical topic don't line up 1:1. In practice, a real disaster-recovery plan either implements explicit offset translation, or — more commonly — just accepts reprocessing a batch of already-seen messages after a regional failover, leaning on the same at-least-once-plus-idempotent-consumer assumption that a well-built Kafka consumer should already be designed around anyway.

Sizing checklist for a new production topic#

Common mistakes#

• Running production topics at replication factor 1 or 2 "temporarily" — exactly the kind of decision that quietly becomes permanent and is only discovered during an actual broker failure.
• Setting min.insync.replicas equal to the full replication factor (e.g., 3 of 3) — that removes all the fault tolerance acks=all was supposed to provide, since losing even a single broker then makes every write fail outright instead of merely reducing redundancy.
• Assuming a MirrorMaker-replicated cluster in another region is a drop-in failover target with identical offsets — without an explicit offset-handling plan, a regional failover means reprocessing a batch of messages, not a seamless resume.
• Deciding partition count once at topic creation and never revisiting it as real traffic and consumer group size change, ending up either badly under-parallelized or carrying needless per-partition overhead for years.