Messaging in .NET, part 2: Redis Streams, or the message log you may already be running

Part 2from the Messaging in .NET series · 3 parts in all

In part 1 we put a broker between two programs and learned the two rules that govern every message system: the reader remembers how far it has read, and it must record that position only after the work is done. This article builds the same order pipeline again, but on Redis — a piece of software a great many teams already run, usually for caching, and very often without realising it can also carry messages.

The sample is RedisDotNet. It is deliberately the same shape as the Kafka one, so the two can be read side by side: an HTTP API takes orders, a background worker processes them, and the only thing that changes is what sits in the middle.

Redis in one paragraph

Redis is a server full of useful data structures that several programs can share: strings, lists, sets, hashes, and sorted sets. You connect, run a command, get an answer. It keeps everything in memory (and optionally writes it to disk), which is why it is fast, and it is why you should think of it as a cache and a coordination point rather than a permanent archive.

Two of those structures do the messaging for us, and one more does the retrying. That is the whole plan.

The stream: a list that keeps a receipt for every entry

A Redis stream is an append-only list of entries. Adding one is a single command:

XADD orders:placed * payload '{"orderId":"..."}' event-type OrderPlaced attempt 0

Every entry gets an id that Redis generates for you, and the id begins with a timestamp: 1790826755718-0. That is the first real difference from Kafka. Kafka's bookmark is an offset — a plain count of how many messages into the partition you are. Redis gives you a timestamp id instead, which means you can ask for "everything since 3pm yesterday" without any extra machinery.

Notice the entry is a set of field/value pairs rather than one blob. Kafka has headers for that job; here the metadata just lives next to the payload. The sample writes payload (the JSON), event-type, attempt and order-key.

Groups, and the receipt you must not forget

Two commands make the receiving side work, and they are quite different from Kafka's.

# "Give this group any entries nobody in it has seen yet."
XREADGROUP GROUP order-processor processor-1 COUNT 10 STREAMS orders:placed >

# "I have finished with entry 1790826755718-0."
XACK orders:placed order-processor 1790826755718-0

Reading does not remove the entry, and it does not acknowledge it either. Every entry you have been handed but have not yet XACKed goes onto the group's pending list. That is Redis's version of Kafka's uncommitted offset, and the analogy is a shelf of letters you have picked up but not yet replied to.

This is why the sample acknowledges only after the work is done:

var outcome = await processor.ProcessAsync(orderKey, order, attempt, cancellationToken);

// Only now do we admit we have dealt with it. Crash before this line and the
// entry stays pending, which means it will be handled again rather than lost.
await database.StreamAcknowledgeAsync(OrderStreams.Placed, consumerGroup, entry.Id);

Same rule as part 1, same consequence: at-least-once. Your handler has to tolerate being called twice.

The shelf nobody empties

Now the part Kafka hides from you. When a Kafka consumer dies, the broker notices and gives its work to somebody else — automatically. Redis does not do that, because it does not know whether your process is slow or dead. The pending list just sits there with an entry on it, possibly forever.

The fix is explicit, and it is one command:

# Take over anything in my group that has been pending (untouched) for 30 seconds.
XAUTOCLAIM orders:placed order-processor processor-1 30000 0-0 COUNT 10

That is PendingEntryReclaimer in the sample: it wakes up every couple of seconds and adopts anything that has been sitting unacknowledged for longer than PendingIdleMs. Less magic than Kafka, but you can see exactly what it does and tune the threshold — which is a real advantage when a worker is legitimately slow and you do not want its messages stolen mid-flight.

The clever bit: waiting without holding anything

Redis Streams have no more of a "deliver this in five seconds" feature than Kafka does. But Redis happens to ship the perfect tool for it: the sorted set, which is a collection where every member has a number attached, kept in sorted order. If you use the number as "the moment this becomes due", a sorted set is a delay queue.

# Put the message aside with its payload...
HSET orders:delayed:payload:abc123 payload '{"orderId":"..."}' target orders:placed

# ...and note when it should come back, as a Unix timestamp in milliseconds.
ZADD orders:delayed:schedule 1790826800000 abc123

# A second later, or an hour later, ask what is due:
ZRANGEBYSCORE orders:delayed:schedule -inf 1790827000000

A small poller runs ZRANGEBYSCORE every half second, moves anything that is due back onto the main stream with an incremented attempt count, and deletes it from the set. That is the whole retry mechanism — and it is arguably nicer than Kafka's, because the waiting message is one number in a data structure rather than a record parked inside a consumer. It even survives a restart, which a paused partition does not.

When the attempts run out, the order is appended to a dead-letter stream (orders:placed:dlq) with the reason attached, exactly as in part 1.

Redis does a second job here

So far Redis is a broker. But it is also the natural place to keep the answer to "what happened to my order?", which every customer-facing system ends up needing.

HSET order:1f2e3d... status Confirmed attempts 1 totalAmount 39.98
EXPIRE order:1f2e3d... 3600

The worker writes a hash per order — Redis's name for a small record of named fields — and the API reads it back to answer GET /orders/{id}. It is one round trip, no schema, no migration, and it uses the same connection the messages travel over.

The EXPIRE is the honest part, and it deserves a sentence: that line is Redis telling you this is a cache, not a system of record. After an hour the answer is gone, and the API says "unknown" rather than lying. If you need the history to outlive an hour, it belongs in a database — and you would treat Redis as the fast copy in front of it.

Pub/Sub, or what "fire and forget" costs

Redis has a third messaging mechanism, and the sample uses it for exactly the case where losing a message is fine: telling the customer their order is confirmed.

PUBLISH orders:notifications "Thank you customer-1, order ... is confirmed."

Nothing is stored. The command returns how many subscribers received it, and if nobody was listening — or the listener restarted a second later — that message simply never existed. No pending list, no replay, no history.

That is the right trade for a toast on a phone and completely the wrong one for the order itself. Having both in one repository, two inches apart in the source, is the point: "can I lose this?" is the question that decides which mechanism to use, and it is a question about your data, not about Redis.

Run the sample

git clone https://github.com/bobhuang1/RedisDotNet
cd RedisDotNet

docker compose up -d                                       # Redis, with disk persistence on
dotnet run --project src/RedisDotNet.OrderProcessor        # terminal 1
dotnet run --project src/RedisDotNet.OrderApi              # terminal 2

Then place an order and read it back. Take the orderId out of the first response:

curl -X POST http://localhost:5090/orders \
  -H 'Content-Type: application/json' \
  -d '{ "customerId": "customer-1",
        "items": [ { "sku": "SKU-001", "quantity": 2, "unitPrice": 19.99 } ] }'

curl http://localhost:5090/orders/<orderId>

Run that second call twice. The first may say Retrying, or 404 while the order is still queued; a moment later it will say Confirmed. Watching one field change is watching the whole pipeline work.

The demo endpoint reaches every branch in one call, as before:

curl -X POST 'http://localhost:5090/orders/demo?count=6'

Look inside Redis

docker exec -it redisdotnet-server redis-cli

XLEN orders:placed                    # how many orders were ever appended
XINFO GROUPS orders:placed            # pending count and lag per group
ZRANGE orders:delayed:schedule 0 -1   # what is waiting to be retried, and when

Beginner mistakes to avoid

• Sharing one consumer name across processes. The pending list is tracked per consumer name; two processes using the same name will keep stealing each other's unacknowledged entries. Give each instance a unique name.
• Confusing > with 0. Reading with > means "give me what is new"; reading with 0 means "replay my own pending list". Mixing them up is how you get duplicates — or silence.
• Forgetting that streams live in memory. Nothing is deleted automatically. Add MAXLEN or trim periodically, or a busy stream will quietly eat the instance.
• Assuming persistence is on by default. Redis is fast because it is in memory. The sample enables append-only persistence; without it, a restart starts from an empty server.
• Using Pub/Sub for anything you cannot afford to lose. It has no history by design.

How this compares to part 1

Redis is the smaller, more legible machine, and if you already run it the messaging is very nearly free. Kafka is the one built for volume, for replay, and for not being the thing that falls over.

One question is left, and it is the one people actually argue about in code review: why would anyone use a third broker when these two exist? In part 3 we implement this pipeline once more on RabbitMQ and find the answer is about routing and acknowledgements — and about a feature RabbitMQ has that both of the others make you build by hand.