RabbitMQ Federation – Part 3

In my last two blog posts, I covered how to deploy a single RabbitMQ node for testing or non HA queues and RabbitMQ Cluster for HA queues.

In this blog post, I’ll cover the latest and the most reliable scenario, with a slight twist. The basic idea behind queue federation is to deal with load-balancing messages across queues on different brokers. If you have a set of queues federated with each other, then producers can publish into them and consumers can consume from them from any location. This time we’ll use Google Compute. 

In this scenario we will also use a cluster of 2 nodes.  We need to create a wrapper box built on top of the RabbitMQ Cluster Node to create the federated cluster. We will call it RabbitMQ Federated Node.
The box wrapper has this variables:

Name                                     Value                                          TYPE
upstream                                BINDING TO                              RabbitMQ Cluster Node
rabbitmq                                 RabbitMQ                                   BOX

We need to create a post_configure Event with the following script to initialize or join an existing cluster. This is a great example of box composition, this box is adding the federation logic to the RabbitMQ Cluster Node box without having to modify the original box.


rabbitmqctl set_parameter federation-upstream {{ instance }} '{"uri":"amqp://{{ upstream.rabbitmq.username }}:{{ upstream.rabbitmq.password }}@{{ upstream.address.private }}:5672","expires":3600000}'
rabbitmqctl set_policy --apply-to exchanges federate-me "^amq\." '{"federation-upstream-set":"all"}'

Step 1: Deploying the Cluster in GCE

Click the “New Instance” button and select the “RabbitMQ Node for Cluster”:

In the next step, you will be asked to create a deployment profile, which lets you pick the configurations for deployment such as cloud provider, datacenter, etc.

Click on “New Profile.” Create a new deployment profile with default values.

  • Be sure to select Automatic in security groups. Using this option, ElasticBox will automatically create a security group configured with the rules needed by RabbitMQ.
  • For persistent queues, make sure you select an EBS backed image and instance combination.

Fill the COOKIE field with the one you want to be shared between all the RabbitMQ nodes of the cluster. We will use this convention for naming the profiles: –
i.e.: rabbitmq-prod-us-west-1, as the environment. Deploy it.

Step 2: Deploying the second node of the cluster in GCE

  • Before launching the second node, the master must be already deployed in GCE.
  • follow the same steps as the master, but select “RabbitMQ Federated Node” for the new instance.
  • Just prior to hitting “Deploy,” select the binding to the master node of RabbitMQ for building the cluster.
  • Deploy the second node using the same name for the environment as the zone profile you are using to deploy.
  • Repeat these steps if you need to add more nodes.

There you have it, a federated node of RabbitMQ Cluster.

This wraps up my 3-part RabbitMQ blog post series. Hope you enjoyed it.  Email me if you have any questions.

Categories: Cool Features & Tutorials