NOA Labs

Pub/Sub and Streams

PUBLISH/SUBSCRIBE, Redis Streams as a durable alternative, and when to use Streams vs Kafka/RabbitMQ.

PUBLISH / SUBSCRIBE

Redis's simplest messaging tool is a straightforward publish/subscribe channel: a publisher sends a message to a named channel, and every currently-subscribed client receives it immediately.

In one redis-cli session, subscribe to a channel:

Bash
SUBSCRIBE notifications

From another session, publish a message to it:

Bash
PUBLISH notifications "New order #4821 placed"

The subscribed client receives it right away:

Bash
1) "message"
2) "notifications"
3) "New order #4821 placed"

PSUBSCRIBE subscribes to a pattern rather than one exact channel name, matching multiple channels at once:

Bash
PSUBSCRIBE orders.*
# matches orders.created, orders.shipped, orders.cancelled, etc.

The key limitation: no persistence

Redis pub/sub is strictly fire-and-forget — if no client is subscribed to a channel at the moment a message is published, that message is simply gone. There's no history, no replay, no delivery guarantee. It's a genuinely good fit for ephemeral, "only matters right now" broadcasts (a live dashboard update, a chat typing indicator, invalidating a cache across multiple app servers) — and a poor fit for anything that must be reliably delivered even if the consumer was briefly offline.

Redis Streams — a durable alternative

A Stream is an append-only log of entries, each with a unique, ordered id — closer in spirit to Kafka than to Redis's own pub/sub. Unlike pub/sub, stream entries persist in memory (and optionally to disk, alongside Redis's usual persistence settings) until explicitly trimmed, so a consumer that was offline can catch up on everything it missed.

Bash
XADD orders * event "created" orderId "4821" amount "24.99"

* tells Redis to auto-generate the entry's id (a timestamp-sequence pair like 1706198400000-0), guaranteeing ids are strictly increasing. Reading entries back:

Bash
XRANGE orders - +          # all entries, oldest to newest
XREVRANGE orders + -       # all entries, newest to oldest

XREAD COUNT 10 STREAMS orders 0    # up to 10 entries, starting right after id "0"

Consumer groups

Streams support consumer groups — multiple consumers cooperatively processing one stream, each entry delivered to only one consumer within the group, with explicit acknowledgment:

Bash
XGROUP CREATE orders processing-group '$'

XREADGROUP GROUP processing-group worker-1 COUNT 5 STREAMS orders '>'
Bash
XACK orders processing-group 1706198400000-0

A consumer that crashes before acknowledging leaves its entries in a pending state, retrievable by another consumer via XPENDING/XCLAIM — this is the durability pub/sub fundamentally lacks: a message survives a consumer restart instead of vanishing the moment it's published to nobody.

When to reach for Streams vs. Kafka/RabbitMQ

Pub/Sub Streams Kafka / RabbitMQ
Delivery guarantee None (fire-and-forget) At-least-once, with consumer groups At-least-once (or exactly-once with care)
Message history/replay No Yes, until trimmed Yes, configurable retention
Built into Redis Yes Yes No — separate infrastructure
Throughput ceiling Very high, but no durability High, bound by a single Redis instance's memory/CPU Designed to scale horizontally across many brokers
Best fit Ephemeral broadcasts, live UI updates Lightweight durable queues/event logs where you're already running Redis High-volume, business-critical event pipelines, multiple independent consumer services, long retention

Redis Streams is the right call when you're already running Redis, the durability and consumer-group semantics of a real message queue are genuinely needed, but the throughput and retention requirements don't justify standing up and operating a dedicated system like Kafka. Reach for Kafka or RabbitMQ once message volume, retention requirements, or the number of independent consuming services grows large enough that a dedicated, horizontally-scalable broker earns its operational overhead — or when guarantees like exactly-once processing or long-term (weeks/months) log retention are hard requirements.

Common mistakes

  • Using plain pub/sub for anything that must be reliably delivered (an order confirmation, a payment event) — if the consumer service happens to be restarting at that exact moment, the message is lost permanently with no way to recover it.
  • Never trimming a Redis Stream (XTRIM), letting it grow unbounded in memory — unlike Kafka's disk-backed, horizontally-scaled log storage, a Redis Stream's entries live in the same memory budget as everything else in that Redis instance.
  • Forgetting XACK in a consumer group, leaving entries permanently "pending" and never retried or cleaned up.
  • Reaching immediately for Kafka/RabbitMQ for a genuinely small, single-service messaging need where Redis Streams (already running, no new infrastructure) would be sufficient and far simpler to operate.