Think I’ve got a fun one for you all here. I’m looking at clustering RabbitMQ across multiple hosts without having to explicitly tell each service to cluster with another. I’d like for me to add a host and scale up the RabbitMQ service which will automatically find another RabbitMQ container and cluster with it.
I understand with the new Rancher metadata stuff I can get information about other containers within a service, but I’m struggling with a DNS issue.
When a RabbitMQ server starts, it sets its “node name” to rabbit@HOSTNAME, which usually comes out as something like rabbit@9d3302e323c1 within Docker. When another server wants to cluster with that, it must refer to the first server with the same name.
This means that if rabbit@123 wants to cluster with rabbit@abc, abc needs to point to whatever IP is relevant.
As far as I’ve seen, you can’t change the hostname that RabbitMQ uses (it uses the HOSTNAME environment variable, but I assume changing this would cause some… oddities).
So, any ideas?
Feel free to bombard me with any questions; I’m sure I haven’t explained the situation as thoroughly as I could’ve!
Have a look at labels especially io.rancher.container.hostname_override: container_name.
It is useful when deploying containers one per host ie.: elasticsearch nodes.
Found that little gem earlier and the containers to indeed set the RabbitMQ “node name” to something like rabbit@MyStack_RabbitMQ_1 which is awesome!
Problem is, other containers within that same service can’t seem to access the container from that hostname; they can access it via the internal IP fine, just the hostname isn’t in their /etc/hosts file!
I’m assuming it’d be bad for me to be editing that on the fly?
What about defining external service for that stack with ip of that rabbitmq node?
That way services can reffer to it as rabbit in their external_links: [ rabbitmq ].
Apologies for the exceedingly late reply! Tried that too and couldn’t get the hostnames to match up still.
Looks like 0.46.0 might quell my woes with its new DNS features, though! If I get this set up I’ll post how (as I know there’s nothing worse than finding a forum thread on what you want to do and finding “Nevermind. Fixed!”)!
I created a docker image for running RabbitMQ with Rancher. It uses the Rancher Metadata Service to allow RabbitMQ to scale, restart and add back after a crash.
