NOA Labs

Kafka Introduction

Run Kafka locally with Docker Compose in KRaft mode and create your first topic from the CLI.

A quick recap of the log

Kafka is architecturally different from a traditional message broker: instead of a queue that removes messages once they're consumed, Kafka is an append-only, partitioned, replicated log. Consumers track their own position (an offset) in that log, so multiple independent consumer groups can each read — and even replay — the same data at their own pace. The System Design track's Message Queues & Event-Driven Architecture page covers why that architecture matters and when to reach for Kafka over something like RabbitMQ. This page — and the rest of this track — assumes that context and gets straight into running Kafka and using it from real code.

Running Kafka locally with Docker Compose

Modern Kafka (3.3+) no longer needs ZooKeeper — a single broker can run its own metadata quorum using KRaft mode. Here's a real, working single-node setup using the official Apache Kafka image:

YAML
services:
  kafka:
    image: apache/kafka:3.7.0
    container_name: kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://kafka:19092,CONTROLLER://kafka:19093,PLAINTEXT_HOST://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:19092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:19093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1

The two listener names are worth understanding rather than copy-pasting blindly: PLAINTEXT (on kafka:19092) is what other containers on the same Docker network use to reach the broker — your application container would connect to kafka:19092. PLAINTEXT_HOST (on localhost:9092) is what a client running directly on your machine, outside Docker, uses instead. Mixing these up is the single most common "why can't my producer connect" problem when running Kafka in Docker — a client inside the container network that tries to connect to localhost:9092 will fail, because localhost resolves to the client's own container, not the broker's.

Start it:

Bash
docker compose up -d

Creating a topic via the CLI

Bash
docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
  --create --topic orders \
  --bootstrap-server localhost:9092 \
  --partitions 3 --replication-factor 1

Verify it was created, and see how it laid out its partitions:

Bash
docker exec -it kafka /opt/kafka/bin/kafka-topics.sh \
  --describe --topic orders --bootstrap-server localhost:9092
Text
Topic: orders   TopicId: 8f3c...   PartitionCount: 3   ReplicationFactor: 1   Configs: ...
    Topic: orders   Partition: 0   Leader: 1   Replicas: 1   Isr: 1
    Topic: orders   Partition: 1   Leader: 1   Replicas: 1   Isr: 1
    Topic: orders   Partition: 2   Leader: 1   Replicas: 1   Isr: 1

With --replication-factor 1 on a single broker, every partition's only replica is itself — fine for local development, never for production (a broker restart would briefly make that partition unavailable, and a broker failure would lose it). The next page covers exactly what "partition" and "replica" mean and why they matter.

Proving it's alive: console producer and consumer

Kafka ships CLI tools for quickly publishing and reading messages without writing any client code — useful for smoke-testing a topic:

Bash
docker exec -it kafka /opt/kafka/bin/kafka-console-producer.sh \
  --topic orders --bootstrap-server localhost:9092

Type a few lines and press Enter after each — each line becomes one message. In a second terminal:

Bash
docker exec -it kafka /opt/kafka/bin/kafka-console-consumer.sh \
  --topic orders --from-beginning --bootstrap-server localhost:9092

--from-beginning reads the whole topic from offset 0 instead of only new messages published after the consumer started — handy for confirming your test messages actually landed.

Common mistakes

  • Connecting a client running on your host machine to the container's internal hostname (kafka:19092) instead of localhost:9092 — or the reverse for a client running inside another container. Match the listener to where the client actually runs.
  • Running with --replication-factor 1 in anything beyond local development — there's no redundancy at all if that one broker goes down.
  • Forgetting --from-beginning when testing and concluding "nothing is being produced," when actually the consumer simply started after the test messages were already published.