The first version of RabbitMQ has been released in 2007. Back in these days, the goal was to provide a complete open source implementation of Advanced Message Queuing Protocol (AMQP), aiming at modern messaging needs such as high availability, high performance, scalability and security.
Nowadays, RabbitMQ is one of the most popular message brokers and can be found in several domains.
This article lights up core concepts and compares it with ActiveMQ Artemis and AWS SQS.
The integration of multiple applications requires some kind of communication. From a high-level perspective the kinds of communication can be grouped into direct and indirect communication. Direct communication means that a direct network connection will be established between sender and receiver of data. This happens when using HTTP, REST, GraphQL or grpc. Indirect communication flows via an infrastructural component between sender and receiver of data. A file on a shared filesystem is a simple example. An intermediate FTP server, a database or a messaging system are other examples. The main reason why these architectures are used is the provided degree of decoupling between sender and receiver. With direct communication, sender and receiver have to be available at time of communication. By introducing an intermediate component, data can be stored and delivered when the receiver is available. Modern messaging solutions provide a lot more features and one of them is RabbitMQ.
RabbitMQ provides both durable and non-durable queues. While durable queues survive a node restart, messages in non-durable queues will be lost. Topics, also known as Publish/Subscribe, are also supported by RabbitMQ.
With acknowledged messages, RabbitMQ provides an “at least once” delivery guarantee. If acknowledgements are disabled, e.g. to increase throughput, this will change to an “at most once” semantic.
The routing of messages to queues can be delayed by a specified amount of milliseconds. This will happen entirely inside RabbitMQ. The Time-To-Live property can be used to flag messages that become irrelevant to consumers after a specified amount of time. Messages that are not consumed within the Time-To-Live will be removed from RabbitMQ.
In the world of RabbitMQ, producers send messages to so-called exchanges and not to the target queues directly. Which queue a message will be routed to, depends on the routing key which links exchanges and queues. If the routing key of a message matches a routing key of a queue bound to that exchange, this message will be delivered to the queue.
For queues, it is also possible to use wildcards * and # on routing keys. In that case, a queue will get all messages which match that pattern.
Example:
Messages get published with the routing keys world.eu and world.na. A queue declared with the routing key world.* will receive all messages. Another queue with the routing key world.eu will only get those messages.
When a consumer is not able to process a message, it is possible to reject messages in order to do some error handling supported by RabbitMQ. Rejected messages go back to a position closer to the queue head.
When using RabbitMQ with AMQP 0-9-1, transactions are also provided. In the context of RabbitMQ, transactions are a batching feature on publishes, acknowledgements and rejections of messages. Creation of resources such as queues are not protected by transactions.
RabbitMQ comes with a built-in management UI. It provides information about consumers, queues, messages, users and the current routing configuration. Besides, basic monitoring is also included. For in-depth monitoring, version 3.8.0+, provides support for Prometheus and Grafana.
RabbitMQ supports AMQP 0-9-1, AMQP 1 (via plugin), MQTT and STOMP and clients exist for several programming languages. For Java clients, right now, there are multiple approaches.
Launch a local instance of rabbitmq with docker run --rm -p 5672:5672 -p 15672:15672 docker.io/rabbitmq:3.8.9-management. Port 5672 is for AMQP, 15672 exposes the management console.
Add the spring-boot-starter-amqp:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
Then create a message receiver
import org.springframework.amqp.rabbit.annotation.Queue;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;
@Component
public class Receiver {
@RabbitListener(queuesToDeclare = @Queue("my-queue"))
public void receiveMessage(String message) {
System.out.println("Received <" + message + ">");
}
}
And finally send and receive a message
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.boot.CommandLineRunner;
import org.springframework.stereotype.Component;
@Component
public class Runner implements CommandLineRunner {
private final RabbitTemplate rabbitTemplate;
public Runner(RabbitTemplate rabbitTemplate) {
this.rabbitTemplate = rabbitTemplate;
}
@Override
public void run(String... args) throws Exception {
System.out.println("Sending message...");
rabbitTemplate.convertAndSend("my-queue","Hello from RabbitMQ");
Thread.sleep(1000);
}
}
The log output should contain
Sending message...
Received <Hello from RabbitMQ!>
If you read the section about exchanges and routing keys carefully, you should be wondered why and how rabbitTemplate.convertAndSend("my-queue","Hello from RabbitMQ") works - as said before: producers send messages together with their routing key to exchanges and not to queues. However, the code example did not contain an exchange, but it seemed to contain a queue name. It works, because Spring AMQP has a few default behaviors:
1. The method convertAndSend(String routingKey, Object object) sends a message to the default exchange. This explains, why it was not necessary to set an exchange explicitly.
2. The receiver’s annotation @RabbitListener(queuesToDeclare = @Queue("my-queue")) created a queue with the name my-queue, but it also created a binding with the queue name as the routing key.
Just to conclude: The code sample created a queue called my-queue and bound it to the default exchange with the routing key my-queue. The producer sent its message to the default exchange with the routing key my-queue.
RabbitMQ provides two setups to increase availability: Active/Active and Active/Passive.
In Active/Active, HA is defined on two layers: on nodes and on queues. In a cluster that consists of multiple nodes, clients can connect to any of them. However, this does not mean that the queues are automatically high available as well. Every queue is managed by a single node, the queue master, and can be replicated to other nodes if configured. However, all operations on that queue will still be performed only by the queue master no matter which node a client connected to. When the queue master goes down, another node will take over to be the new queue master.
In case of network partitioning within the cluster, RabbitMQ can either retain consistency or availability. Consistency will be retained when pause_minority is configured. Instances on the minority side of the partition will pause themselves and wait until the partition is resolved. During this time, these instances are not available.
Availability will be retained when auto-heal or ignore is configured. When a partition is detected, all instances keep running. When the partition is resolved, the state of the majority side of the partition will be used as the new state for the whole cluster. Obviously, this can lead to loss of data.
In an Active/Passive setup, instances share persistent messages via the filesystem. If a node crashes, another node will initialize with the shared file system and take over. Non-persistent messages will be lost. This approach comes with the drawback of a delay when another node takes over. Besides, the Active/Passive setup seems to be more complex to set up. Combinations Active/Active and Active/Passive are also possible.
AWS, GCP and Azure provide ways to setup RabbitMQ clusters. While GCP and Azure offer it via their marketplaces, AWS also provides a native service that is part of AWS MQ.
For the operation on Kubernetes, there is an operator available. Key features are the provisioning of single-node and multi-node clusters, management of active vs. desired state and a set of monitoring tools based on Prometheus and Grafana. Upcoming versions will also provide rolling upgrades for RabbitMQ clusters.
In general, ActiveMQ Artemis can be seen as similar to RabbitMQ, because most features listed in this article are also supported. The following list summarizes features that are unique to ActiveMQ Artemis.
Regarding HA, ActiveMQ Artemis also provides both Active/Active and Active/Passive setups. In contrast to RabbitMQ, the Active/Passive configuration is not limited to the shared storage concept. There is also a replication mode, where the state is synchronized continuously between the Active and the Passive node. Compared to the Active/Active concept in RabbitMQ, Artemis follows an all-or-nothing-principle regarding replication - it is not possible to replicate just single queues.
Apart from that, I want to point out the good quality of the online documentation.
AWS SQS is focused on queues and doesn’t provide features regarding Publish/Subscribe or routing like in RabbitMQ or ActiveMQ Artemis. In order to do that, SQS can be combined with SNS. Features provided by SQS are standard queues (optimized for throughput), FIFO queues (retain ordering), delaying of single messages or entire queues. Messages in SQS are automatically deleted after at most 14 days. By setting smaller values (but at least 1 minute), the TTL feature provided by RabbitMQ and ActiveMQ Artemis can be implemented. SQS provides clients for the most common programming languages - including, but not limited to, Java, Python, Ruby, .NET, PHP and JavaScript. A JMS 1.1 client is also available. As of the time of writing this article, AWS does not provide a (mock) implementation for local testing.