Schema Registry and Avro

Why a long-lived topic needs a schema contract#

Kafka itself is payload-agnostic — a topic's messages are just bytes, and every producer/consumer example so far in this track has serialized values as a plain JSON string with no enforced structure at all. That's fine for a short-lived example, but it breaks down on a real production topic: a topic is often written to and read from by different teams, deployed independently, over months or years. With no shared schema, a producer team can rename or drop a field a consumer team's code silently depends on, and the first anyone hears about it is a consumer crashing (or worse, silently misbehaving) in production.

Avro, briefly#

Avro is a compact binary serialization format built around a schema-first philosophy: every Avro-encoded message is written according to an explicit schema describing its fields and types, and reading it back requires that same schema. A simple Avro schema, as JSON:

{
  "type": "record",
  "name": "OrderCreated",
  "namespace": "com.example.orders",
  "fields": [
    { "name": "orderId", "type": "string" },
    { "name": "amount", "type": "double" },
    { "name": "currency", "type": "string", "default": "USD" }
  ]
}

Compared to JSON text, Avro's binary encoding is significantly smaller on the wire and faster to parse — real advantages at Kafka's typical scale — but the bigger reason it's the default choice for serious production topics is the tooling built around schema management, not the encoding efficiency alone.

The Schema Registry#

Embedding a full schema in every single message would be wasteful. Instead, a Schema Registry (Confluent's is the most widely used) is a separate service that stores every version of every schema ever registered for a topic. Each Avro-encoded message on the wire carries just a small schema ID — a few bytes — referencing which registered schema version it was written with:

[magic byte][4-byte schema ID][Avro-encoded payload bytes]

A consumer looks up that ID against the registry (caching the result locally, since schema IDs don't change) to decode the message correctly — even if the registry's current schema for that topic has since evolved further than the version this particular message was written with.

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, KafkaAvroSerializer.class.getName());
props.put("schema.registry.url", "http://localhost:8081");

KafkaProducer<String, GenericRecord> producer = new KafkaProducer<>(props);

Schema evolution and compatibility modes#

The registry's real value is enforcing that a new schema version doesn't silently break producers or consumers still running against the previous version. Every new schema registration is automatically checked against a configured compatibility mode:

Concretely: adding a new field with a default value is backward-compatible — an old consumer reading new data simply ignores the field it doesn't know about, and a new consumer reading old data that lacks the field falls back to the default. Removing a field, or renaming one, are the classic ways to break compatibility unintentionally:

// GOOD: adding an optional field with a default — backward compatible
{ "name": "priority", "type": "string", "default": "NORMAL" }

// RISKY: removing "currency" entirely, or renaming it to "curr" — an old
// consumer expecting "currency" now silently gets nothing back for it

Avro (like most schema systems) has no concept of "rename" — renaming a field is indistinguishable, from a compatibility-checking point of view, from removing one field and adding an unrelated one. The correct evolution path for a rename is: add the new field alongside the old one, migrate consumers to the new field, and only remove the old field once nothing depends on it anymore.

Common mistakes#

• Treating a schema registry as optional tooling for a topic that's actually read by multiple independently-deployed consumer teams — without it, a silent, breaking field change ships straight to production with no warning to anyone downstream.
• Renaming a field directly instead of adding a new one and deprecating the old — schema systems generally have no "rename" operation, so this reads as an unrelated field removal plus an unrelated field addition, and breaks compatibility.
• Choosing NONE compatibility mode "to move fast" on a shared production topic, which removes the exact safety net the registry exists to provide.
• Assuming plain JSON messages with no registry and no shared schema (the simplified pattern used earlier in this track for clarity) offer any protection at all against a field being silently renamed or dropped — fine for a demo, genuinely risky for a long-lived topic with independent producers and consumers.