Exactly-Once Semantics

Why "exactly once" is a marketing simplification#

The previous page's interview questions already established Kafka's practical default: at-least-once delivery, with the consumer responsible for tolerating reprocessing. Kafka does offer features marketed as "exactly-once semantics" (often abbreviated EOS), and they're genuinely useful — but the honest description is closer to effectively once: no distributed system where a producer sends over an unreliable network and a consumer can crash between processing and committing achieves single-attempt delivery in the literal sense the phrase implies. What Kafka's EOS features actually guarantee is narrower, and worth understanding precisely rather than taking the marketing term at face value.

Idempotent producers: deduplicating retries, not application logic#

The producers-and-consumers page in this track already introduced ENABLE_IDEMPOTENCE_CONFIG = true. Here's the mechanism: an idempotent producer gets a unique producer ID (PID) from the broker, and tags every batch it sends with a per-partition sequence number. The broker remembers the last several sequence numbers it has written for each (PID, partition) pair, and if a retry arrives carrying a sequence number it already wrote — because the original write actually succeeded but the acknowledgment was lost, causing the client to retry blindly — the broker recognizes the duplicate and simply re-acknowledges it without writing the record a second time.

This solves exactly one problem: duplicate writes caused by the producer's own network-level retries. It does not deduplicate two messages your application code deliberately sends separately (that's not a bug — those are genuinely different events, and shouldn't be merged), and on its own it says nothing about atomicity across multiple partitions or topics.

Transactions: atomic writes across a read-process-write cycle#

Stream processing commonly follows a read-process-write pattern: consume a record from topic A, transform it, produce a result to topic B, and commit the offset back on topic A — three separate operations. Kafka transactions make all three atomic as a unit: either all of them are considered to have happened, or (as observed by a read_committed consumer) none of them are — never a partial state where the output landed on topic B but the offset commit on topic A never happened, which would cause a restart to reprocess the same input and produce a duplicate output downstream.

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "order-enricher-1");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);

KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();

try {
    ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
    producer.beginTransaction();

    Map<TopicPartition, OffsetAndMetadata> offsetsToCommit = new HashMap<>();

    for (ConsumerRecord<String, String> record : records) {
        String enriched = enrich(record.value());
        producer.send(new ProducerRecord<>("enriched-orders", record.key(), enriched));

        offsetsToCommit.put(
            new TopicPartition(record.topic(), record.partition()),
            new OffsetAndMetadata(record.offset() + 1)
        );
    }

    // The consumer's offset commit becomes part of THIS SAME transaction
    producer.sendOffsetsToTransaction(offsetsToCommit, consumer.groupMetadata());
    producer.commitTransaction();
} catch (Exception e) {
    producer.abortTransaction();
}

TRANSACTIONAL_ID_CONFIG gives the producer a stable identity that survives restarts, which matters for a specific failure mode: if a producer instance hangs and a new instance takes over the same work (during a rebalance or a failover), the broker uses this ID to fence off the old, possibly-still-running "zombie" instance — any further writes it attempts under the same transactional ID are rejected, preventing it from corrupting a transaction the new instance already committed or aborted.

read_committed vs. read_uncommitted#

None of this matters to a downstream consumer unless it explicitly opts in with isolation.level=read_committed. The default, read_uncommitted, means a consumer sees every write immediately, including ones belonging to a transaction that later aborts — which means the end-to-end "effectively once" guarantee only actually holds if every consumer along the pipeline reads with read_committed; a single consumer left on the default setting reintroduces exactly the duplicate/partial-state visibility the transaction was meant to prevent.

What none of this covers#

• Side effects outside Kafka. A consumer that calls an external payment API or writes to a non-Kafka database as part of "processing" a record has stepped outside Kafka's transactional boundary entirely — a Kafka transaction has no way to roll back an HTTP call that already fired. That external action needs its own idempotency mechanism (an idempotency key, an upsert keyed by a unique event ID) if being triggered twice would actually cause harm.
• Deliberate application-level duplicate sends. If your own code calls producer.send() twice for what it considers two separate logical events, idempotence correctly does not merge them — that's not what it's for.

Comparison table#

Common mistakes#

• Enabling transactions and assuming an external side effect performed in the same processing step (sending an email, calling a third-party API) is now "exactly once" too — it isn't; only Kafka-internal reads, writes, and offset commits are covered.
• Forgetting to set isolation.level=read_committed on every downstream consumer, silently seeing (and acting on) writes from transactions that later abort.
• Reusing the same transactional.id across genuinely concurrent producer instances instead of one instance replacing another after a restart — the broker's fencing mechanism assumes only one live writer per transactional ID at a time, and will reject one of them.
• Treating "idempotent producer" and "transactional producer" as interchangeable — idempotence alone removes duplicate writes from retries within a single partition; it says nothing about atomicity across multiple partitions or topics, which is what transactions are specifically for.