NOA Labs

Monitoring and Operations

Consumer lag, key broker and consumer metrics to alert on, and common operational issues and their causes.

Consumer lag: the single most important number

Consumer lag is the difference between the latest offset written to a partition and a consumer group's last committed offset on that same partition — in plain terms, how far behind the live end of the log that group currently is. Rising, non-recovering lag is usually the earliest and clearest sign that a consumer group can't keep up with production rate, typically visible well before anything actually throws an error or a user notices stale data.

Bash
docker exec -it kafka /opt/kafka/bin/kafka-consumer-groups.sh \
  --describe --group inventory-service --bootstrap-server localhost:9092
Text
GROUP             TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
inventory-service orders  0          10482            10600           118
inventory-service orders  1          9921             9921            0
inventory-service orders  2          10310            10850           540

Reading this: partition 2 is 540 messages behind the head of its log — worth investigating specifically (is the consumer instance handling that partition slower than the others? is a downstream dependency it calls being slow? is the partition assignment uneven?), rather than treating "inventory-service has some lag" as one undifferentiated number.

Broker-side metrics worth alerting on

Metric What it tells you
UnderReplicatedPartitions Partitions whose replicas have fallen out of sync — an early sign of a struggling broker or a network issue, before anything is actually unavailable
ActiveControllerCount Should be exactly 1 across the whole cluster at all times; 0 or more than 1 signals a controller election problem
RequestHandlerAvgIdlePercent How saturated a broker's request-handling threads are; near 0 means the broker is maxed out
BytesInPerSec / BytesOutPerSec Raw throughput — useful for capacity planning and spotting traffic anomalies
OfflinePartitionsCount Partitions with no available leader at all — actively unavailable, needs immediate attention, not just a leading indicator

Consumer-side metrics

Beyond raw lag, a few client-side metrics round out the picture: records-lag-max (the worst lag across any single partition the consumer owns, which can hide behind a healthy-looking average), records-consumed-rate (throughput, to correlate against lag trends), fetch-latency-avg (how long the broker is taking to answer fetch requests), and rebalance count/rate — frequent rebalances indicate an unstable consumer group, not routine operation.

Common operational issues and their usual causes

Symptom Likely cause
Growing, unbounded consumer lag The consumer is too slow per-record (a slow downstream call), there are too few partitions/consumers for the actual throughput, or an instance in the group is stuck or has silently crashed
Frequent consumer group rebalances The poll loop is exceeding max.poll.interval.ms (slow processing), consumers are crash-looping, or session-timeout settings are too aggressive for the environment
UnderReplicatedPartitions rising A broker under load, disk I/O contention, or a network issue between brokers preventing timely replication
Producer seeing frequent timeouts/retries An overloaded broker, acks=all combined with a slow in-sync replica, or a network issue between the producer and the cluster
A spike in duplicate processing downstream A rebalance occurred mid-batch, exposing the normal at-least-once reprocessing window — worth checking whether the affected consumer's side effects are actually idempotent

A minimal production monitoring stack

Kafka brokers and clients expose their metrics over JMX by default. The common, well-trodden path is scraping those with a Prometheus JMX exporter, visualizing them in Grafana, and alerting on the thresholds above — consumer lag and UnderReplicatedPartitions first, since they're the two that most reliably catch a real problem before users do.

Common mistakes

  • Only checking consumer lag reactively, after users report stale data, instead of alerting on it proactively — by the time it's visibly broken, it's often already been silently growing for a while.
  • Treating any rebalance as inherently a problem — an occasional rebalance from a deploy or autoscaling event is normal; the metric that actually signals instability is rebalance frequency, when it's high and continuous rather than an expected, occasional event.
  • Not distinguishing "lag is high but shrinking" (a consumer catching up after a blip) from "lag is high and growing" (a consumer that will never catch up without intervention) — the trend matters far more than any single snapshot.
  • Ignoring UnderReplicatedPartitions because "nothing's actually down yet" — it's a leading indicator of reduced fault tolerance, not a cosmetic metric to check only after an outage.