RabbitMQ clustering and high availability on Ubuntu EC2 servers


The following setup and configurations were performed on the production servers listed below:

  • mq-node1.example.com
  • mq-node2.example.com
  • mq-node3.example.com


# apt-get install rabbitmq-server

Hostname and /etc/hosts

On each server, the first priority of setup is hostname and /etc/hosts for internal IP addresses, which are used by RabbitMQ (Erlang) nodes. Use this script (set-hostname.sh) to set the hostname for the corresponding server with the command:

# ./set-hostname.sh -s mq-nodeN.example.com

Then, modify /etc/hosts so that it looks like this (e.g. on mq-node1.example.com): mq-node1.example.com mq-node1 mq-node2 mq-node3


Make sure port 5672 (AMQP) and 45000 (we’ll use this later) are open on each server.

Config files

On each server, do the following:

Edit /etc/rabbitmq/rabbitmq-env.conf and add the following (port configuration):

SERVER_START_ARGS="-kernel inet_dist_listen_min 45000 -kernel inet_dist_listen_max 45000"

Edit /etc/rabbitmq/rabbitmq.config and add the following (clustering configuration):

[{cluster_nodes, {['[email protected]', '[email protected]', '[email protected]'], disc}}]}].

Edit /var/lib/rabbitmq/.erlang.cookie and make sure they are the same on each server.

Reset and start nodes

The RabbitMQ nodes may be already in running state. To put new configuration in effect, we have to reset and restart: First, we reset and stop each node:

# rabbitmqctl stop_app; rabbitmqctl reset;  rabbitmqctl stop

Then, start each node in turn. Remember, do this IN TURN, not simultaneously:

# service rabbitmq start

Check status

Until then, the nodes should be up and running. Check whether the cluster has been successfully created:

# rabbitmqctl cluster_status

You should get the following result:

Cluster status of node '[email protected]' ...
[{nodes,[{disc,['[email protected]','[email protected]',
                '[email protected]']}]},
 {running_nodes,['[email protected]','[email protected]',
                 '[email protected]']},

If you don’t get this result, you may have to troubleshoot by checking logs at (/var/log/rabbitmq) inspecting each of the previous steps, and reset and start again.

Setup high availability (mirrored queues)

Now it’s time to setup mirrored queues. Run the following command:

# rabbitmqctl set_policy ha-all ".*" '{"ha-mode":"all"}'

Then, you can check policies:

# rabbitmqctl list_policies
# rabbitmqctl list_queues name policy

Enable plugins

# rabbitmq-plugins enable amqp_client
# rabbitmq-plugins enable mochiweb
# rabbitmq-plugins enable rabbitmq_management
# rabbitmq-plugins enable rabbitmq_management_agent
# rabbitmq-plugins enable webmachine
# rabbitmq-plugins enable rabbitmq_web_dispatch
# service rabbitmq-server restart

Enable rabbitmqadmin

rabbitmqadmin is an administration command for user and queue management. After rabbitmq_management plugin is enabled, it is present on the server. You can find it and create a symbolic link to it.

# updatedb
# locate rabbitmqadmin
# ln -s $(locate rabbitmqadmin) /usr/sbin/rabbitmqadmin

User management

# Add privileged user
rabbitmqctl add_user 'root' 'your_password'
rabbitmqctl set_user_tags root administrator
rabbitmqctl set_permissions root ".*" ".*" ".*"
# mq-read
rabbitmqctl add_user 'mq-read' 'your_password'
rabbitmqctl set_user_tags mq-read administrator
rabbitmqctl set_permissions mq-read '^$' '^$' ".*"
# worker
rabbitmqctl add_user worker 'your_password'
rabbitmqctl add_vhost /
rabbitmqctl set_permissions -p / worker ".*" ".*" ".*"

Purge queues

# rabbitmqadmin -u root -p 'your_password' purge queue name=QUEUE_NAME

Useful rabbitmqctl commands

# Check queues
rabbitmqctl list_queues

# Check master node
rabbitmqctl list_queues pid

# Check slaves
rabbitmqctl list_queues slave_pids
rabbitmqctl list_queues synchronised_slave_pids