NOA Labs

Production Monitoring

Key metrics like queue depth and consumer utilization, and querying the management API.

Why monitoring a broker is different from monitoring a service

An HTTP service's health is mostly about that one process — is it up, is it responding fast enough. A message broker sits between producers and consumers, so its health also depends on whether the two sides are keeping pace with each other. A queue can be perfectly healthy from RabbitMQ's own point of view (accepting publishes, serving deliveries) while quietly building an ever-growing backlog because consumers have stopped keeping up — and nothing about the broker process itself looks wrong until that backlog eventually causes a real problem (memory pressure, disk alarms, or SLA-breaking delivery delays).

The metrics that matter most

Metric What it tells you Why it matters
Queue depth (ready messages) How many messages are waiting, unconsumed, right now A depth that's steadily climbing means consumers aren't keeping up with producers — the single most important early-warning signal
Consumer utilization The fraction of time a queue's consumers actually spend processing vs. idle waiting for messages Utilization near 100% with a growing queue depth means you need more/faster consumers, not just a config tweak
Publish/deliver/ack rates Messages per second flowing through each stage A gap between publish rate and ack rate over time is exactly what produces a growing queue depth
Unacked message count Messages delivered to a consumer but not yet ack'd/nack'd A count that only grows, never shrinks, usually means a consumer is stuck, crashed without reconnecting cleanly, or has a bug that never calls ack/nack at all
Connection & channel counts How many are currently open A steadily climbing count with no matching growth in actual traffic usually means something is leaking connections/channels instead of reusing them (see Performance Tuning)
Memory & disk alarms Whether RabbitMQ has hit a configured memory or disk high-watermark Once tripped, RabbitMQ blocks all publishers cluster-wide until the alarm clears — this is a hard stop, not a warning

Queue depth alone, without consumer utilization alongside it, is easy to misread: a queue with 10,000 messages sitting in it could mean "consumers are overwhelmed" or simply "a burst just arrived a second ago and consumers are draining it quickly" — utilization is what tells the two apart.

The management API

Every metric in the management UI is also available as JSON over HTTP, which is what makes it possible to feed RabbitMQ's state into a real monitoring stack instead of only checking a dashboard by eye:

Bash
# Overall cluster health and message rate summary
curl -u guest:guest http://localhost:15672/api/overview
Bash
# Per-queue detail — this is where queue depth and consumer counts actually live
curl -u guest:guest http://localhost:15672/api/queues/%2f/orders.queue
JSON
{
  "name": "orders.queue",
  "messages": 842,
  "messages_ready": 830,
  "messages_unacknowledged": 12,
  "consumers": 3,
  "consumer_utilisation": 0.94,
  "message_stats": {
    "publish_details": { "rate": 120.5 },
    "deliver_get_details": { "rate": 118.2 },
    "ack_details": { "rate": 117.9 }
  }
}

messages_ready is queue depth in the strict sense — messages waiting to be delivered. messages_unacknowledged is the count flagged above as a leak indicator when it only grows. consumer_utilisation of 0.94 means this queue's consumers are busy 94% of the time — close to saturated, worth watching for a growing messages_ready alongside it as the sign it's time to scale consumers up.

Wiring metrics into Prometheus

RabbitMQ ships a built-in rabbitmq_prometheus plugin that exposes the same data the management API returns, in Prometheus's scrape format, without needing a separate exporter process:

Bash
rabbitmq-plugins enable rabbitmq_prometheus
YAML
# prometheus.yml
scrape_configs:
  - job_name: 'rabbitmq'
    static_configs:
      - targets: ['rabbitmq-host:15692']
YAML
# An example alerting rule — fires if a queue's backlog keeps growing for 10 minutes straight
groups:
  - name: rabbitmq
    rules:
      - alert: QueueBacklogGrowing
        expr: rate(rabbitmq_queue_messages_ready{queue="orders.queue"}[10m]) > 0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: 'orders.queue backlog has grown steadily for 10 minutes'

Alerting on the rate of change of queue depth (is it consistently growing) rather than a single static threshold catches the actual failure mode — "consumers can't keep up" — earlier and with fewer false alarms than a fixed "alert if depth > 10,000" rule, which fires just as easily on a brief legitimate burst.

Common mistakes

  • Alerting on queue depth alone, with no consumer utilization or trend data alongside it — a queue depth that spiked and is already draining looks identical, in a single snapshot, to one that's genuinely and steadily backing up.
  • Treating a growing messages_unacknowledged count as "just a busy queue" instead of investigating — it usually means a specific consumer stopped calling ack/nack at all, which also silently caps how many new messages that consumer can receive under its prefetch limit.
  • Ignoring memory/disk alarms until they trip in production — once tripped, RabbitMQ blocks all publishers cluster-wide as a hard safety stop, which is a far more disruptive outage than catching the underlying disk/memory pressure earlier would have been.
  • Polling the management API from application code as a substitute for real metrics infrastructure — it works for a quick manual check, but a proper Prometheus/Grafana (or equivalent) pipeline is what actually enables alerting, historical trend graphs, and catching a slow backlog building over hours or days.