RabbitMQ Queues vs Streams

While RabbitMQ queues have been the traditional mechanism for handling messages, RabbitMQ streams offer a newer, more advanced approach to managing message storage and consumption. This post aims to delve into the key differences between RabbitMQ queues and streams, explore how they handle message storage and consumption, identify scenarios where streams may be more advantageous, and examine their performance implications, scalability, and reliability.

Whereas many people are familiar with RabbitMQ queues, RabbitMQ streams may be unfamiliar to some. We highly recommend checking this Introducing Stream Support in RabbitMQ to have an overview of RabbitMQ streams if needed or contact our RabbitMQ troubleshooting team..

Jump to the full explanation.

RabbitMQ Queues and Streams: Infographic

To embed this infographic on your own site you can use the following code:

<a href="https://seventhstate.io/rabbitmq-queues-vs-streams"><img style="width:800; height:2800" src="https://seventhstate.io/wp-content/uploads/2024/06/Queues-vs-Stream-infographic-1.jpg"></a><br>Courtesy of <a href="https://seventhstate.io/">Seventh State</a> - Experts in RabbitMQ

Key Differences Between Queues and Streams

Queue has been a key concept of RabbitMQ since its very early days. It offers a “buffer” where messages are operated on a FIFO basis. New messages are added to the queue when they are published and removed when they are consumed. RabbitMQ queues have evolved with numerous features and enhancements. However, queues are designed to be drained and they are optimised as such.

Stream is a new data structure introduced in RabbitMQ 3.9. It enhances RabbitMQ by introducing an append-only model for messages. Messages published to a stream will be stored and delivered in a non-destructive manner, allowing them to be re-readable by any number of consumers.

The following table provides a comparison between different aspects of RabbitMQ queues and streams.

	Queues	Streams
Message Handling	Messages in a queue are processed in a FIFO (First-In-First-Out) order. Once a message is consumed, it is removed from the queue.	Streams provide append-only logs where messages are appended and consumed multiple times by different consumers.
Message Storage	Messages are transient unless explicitly set to be persistent. They are removed from the queue after consumption.	Messages are retained based on configurable retention policies (time-based or size-based), allowing for longer-term storage.
Consumption	Messages are consumed by a single consumer or in a competing fashion among multiple consumers.	Allows multiple consumers to read the same message independently, offering more flexible consumption patterns.

When Are Streams More Advantageous Than Queues?

RabbitMQ streams and queues serve different purposes and are optimised for different use cases. Streams are particularly useful in many scenarios:

– Event Sourcing and Event Log Use Cases

Streams are ideal for use cases where the entire history of events needs to be stored and possibly replayed later. This is useful for event sourcing patterns where the state of an application is derived from a sequence of events.

Streams allow consumers to replay messages from any point in the stream, which is not possible with traditional queues.

– High Throughput and Low Latency

Streams are designed for high throughput and low latency, handling millions of messages per second. They can better handle the load for applications requiring real-time processing of large volumes of messages.

Streams support multiple consumers reading from the same stream at different offsets, allowing for parallel processing of messages, which can improve performance and scalability.

– Retention

Streams allow for time-based or size-based retention policies. This means that messages can be stored for a specified duration or until the stream reaches a certain size, making them suitable for applications that need to retain historical data for analysis or compliance purposes.


On the other hand, RabbitMQ queues also have their advantages in other scenarios:

– Task Distribution

Queues are well-suited for distributing messages among multiple consumers, where each message should be processed by exactly one consumer. This is common in background job processing or task scheduling systems.

– Simplicity

For simpler messaging patterns where messages are consumed and processed in a fire-and-forget manner, queues are easier to set up and manage.

– Acknowledgment Mechanism

Queues provide a straightforward mechanism for message acknowledgement, ensuring messages are only removed from the queue after successful processing by a consumer.

RabbitMQ queues and streams each offer unique advantages for different messaging requirements. These distinctions can help us choose the right messaging model based on specific use cases and application requirements.

Can RabbitMQ Streams Replace Queues?

As discussed above, RabbitMQ streams and RabbitMQ queues serve different but complementary purposes. They’re not direct replacements for each other but can be used together to leverage their unique strengths.

RabbitMQ quorum queues provide a high availability solution for RabbitMQ queues in a cluster environment. Yet, RabbitMQ quorum queues cannot fully be replaced by RabbitMQ streams in many aspects.

Quorum queues:

• RabbitMQ quorum queues are designed for high availability and data safety. They are based on the Raft consensus algorithm, ensuring messages are replicated across multiple nodes in a cluster. This replication provides resilience against node failures.
  
• Quorum queues offer strong delivery guarantees, ensuring that messages are safely processed, thanks to the replication and confirmation mechanisms. Messages are guaranteed stored, offering high reliability even in the face of node crashes. 
  
• Quorum queues are ideal for mission-critical applications where data safety and high availability are paramount, such as financial transactions, order processing, and critical notifications.

Streams:

• RabbitMQ streams, on the other hand, are built for high-throughput and scalable messaging. They enable efficient stream processing and real-time data analytics. Optimised for processing millions of messages per second, RabbitMQ streams are suitable for big data and real-time analytics. They allow for large-scale data processing across multiple nodes.
  
• Messages in RabbitMQ streams are stored in an immutable log, which allows for efficient replay and retention policies. Consumers can re-read messages from specific offsets, making it ideal for event sourcing and stream processing.
  
• RabbitMQ streams are suitable for scenarios where high data throughput and replay capabilities are crucial, such as real-time monitoring, IoT data ingestion, and analytics pipelines.

Although messages are persisted to disk with both quorum queues and streams, the storage and confirmation mechanisms of these two are different, leading to different data safety guarantees.

In short, for applications requiring strong consistency, high availability, and data safety, quorum queues are the better choice. They provide the necessary guarantees for critical operations. However, for applications needing high throughput, real-time processing, and the ability to replay messages, RabbitMQ streams are more suitable. They excel in handling large-scale data processing and analytics tasks.

Consequently, RabbitMQ streams could not and should not be a replacement for RabbitMQ queues or RabbitMQ quorum queues.

Performance Implications, Reliability, Scalability

When deploying RabbitMQ, whether you’re utilising streams or queues, it’s crucial to allocate adequate resources to ensure smooth operation. Our recommended minimum setup includes:

• A cluster of three nodes
• 4 vCPUs per node
• 4 GB of RAM per node
• SSD storage for optimal disk I/O

RabbitMQ streams are generally more resource-efficient than queues, primarily due to their disk-based data storage approach. This design choice means that streams have lower memory and CPU demands. Therefore, streams benefit significantly from high I/O rate disks, which can enhance their performance.

As mentioned, both RabbitMQ quorum queues and streams persist messages to disk, but they offer different levels of message reliability:

• Quorum Queues: Publisher confirms are issued only after messages are written and flushed to disk on at least a quorum of nodes. Quorum queues use fsync to ensure data is securely written to disk.
  
• Streams: Publisher confirms are issued once messages have been replicated to a quorum of stream replicas. Unlike quorum queues, streams depend on the operating system for syncing messages to disk, which means they lack direct control over this process. This reliance introduces a potential risk of data loss during an uncontrolled server shutdown, even if publisher confirms have been issued.

It’s worth noting that there are no guarantees for messages that haven’t been confirmed or if the publisher confirm mechanism isn’t used.

When it comes to ensuring data safety, quorum queues are the superior choice due to their explicit disk write control. However, if performance is your priority, streams may be the better option thanks to their efficient resource usage.

Among their many capabilities, RabbitMQ quorum queues and streams, both come with controls for managing replicas to enhance availability. For example, adding a replica is just as simple:

rabbitmq-streams add_replica [-p <vhost>] <stream-name> <node>

or

rabbitmq-queues add_member [-p <vhost>] <queue-name> <node>

For more operational details on managing streams and quorum queues, please refer to the official RabbitMQ documentation:

• Managing Streams: https://www.rabbitmq.com/docs/man/rabbitmq-streams.8
• Managing Quorum Queues: https://www.rabbitmq.com/docs/man/rabbitmq-queues.8

It’s important to note that replicas in both streams and quorum queues are managed explicitly by operators. This means that in environments like Kubernetes, it’s not advisable to add or remove RabbitMQ nodes automatically. Instead, it’s recommended to use the RabbitMQ Operator with a static cluster membership when possible. This approach allows for more controlled scaling and ensures that the message system remains stable and reliable.

The size of the cluster can also significantly impact the throughput of both quorum queues and streams. As a rule of thumb, the more replicas present, the lower the throughput tends to be. This is because of the increased demands of data replication and consensus achievement across multiple nodes.

Client Libraries for RabbitMQ Streams

RabbitMQ streams can be used similarly to RabbitMQ queues through the AMQP protocol. However, updates to some client libraries may be necessary to support new features, such as setting stream offsets.

Besides, to truly tap into the rich capabilities of RabbitMQ streams – such as enhanced performance and the single active consumer feature – it’s essential to utilise a specialised protocol: the binary protocol.

The detailed protocol documentation can be found at: https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbitmq_stream/docs/PROTOCOL.adoc

Here are some of the client libraries available for different programming languages:

• .NET: https://github.com/rabbitmq/rabbitmq-stream-dotnet-client
• Go: https://github.com/rabbitmq/rabbitmq-stream-go-client
• Java: https://github.com/rabbitmq/rabbitmq-stream-java-client

Conclusion

In conclusion, comprehending the functionalities of RabbitMQ streams and queues enables us to make informed decisions when selecting the optimal RabbitMQ solution. Each brings to the table distinct features that not only enhance system capabilities but also adapt seamlessly to a variety of use cases. By employing these wisely, we can ensure our systems are robust, scalable, and ready to meet the ever-evolving demands of applications.