So here is what I ended up doing.
We love Rancher so much we forget that our entire infrastructure does not have to be built with containers. First of all I should mention that I am using AWS. Since I have my GlusterFS environment already in place and my Rancher hosts are being created with an AutoScaling group on AWS the setup was actually quite easy.
This is my “User Data” for my Launch Configuration.
I created a registry.crt file which contains my wildcard certificate and all intermediaries. So its really a chain cert. (This is required by docker registry, but not documented in your instructions).
sudo docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher -e CATTLE_HOST_LABELS=RancherPool=yes rancher/agent:v1.0.1 http://rch.mydomain.com:8080/v1/scripts/D67D5110CBA5A0577D90:1463691600000:50adKAtfd1aWesWH9BgSdi9GBmU
sudo apt-get install -y glusterfs-client
sudo mkdir /app
sudo mount -t glusterfs glusterfs1.mydomain.com:vol1 /app
sudo aws s3 cp s3://my-s3-bucket-certs/portus/registry.crt /usr/local/share/ca- certificates/ca.crt
sudo service docker restart
sudo mkdir /etc/docker/certs.d/reg.mydomain.com:5000
sudo aws s3 cp s3://my-s3-bucket-certs/portus/registry.crt /etc/docker/certs.d/reg.mydomain.com:5000/ca.crt
docker login -u docker -p "password" -e firstname.lastname@example.org reg.mydomain.com:5000 ## Added after registry was working
My launch configuration does the following:
- installed the rancher agent and lables it RancherPool=yes
- installs the gluster client
- mounts a volume at /app. (I use /app/registry as the Storage directory.)
- installs the CA certificate to the hosts’ trusted CA store
- creates the proper directory for my new registry
- copies the certificate to that registry
- logs in to the registry
Now my GlusterFS Volume is mounted on every RancherPool Host.
So then I launched your stack.
But I did not want any part of the stack to run on anything other than the instances labeled RancherPool.
So I copied the the docker and rancher compose files from the running registry down to my local machine where I have Rancher-Compose setup. Then I deleted the registry from my Applications on Rancher.
I modified the docker-compose.yml by changing the following lines:
lb modified line in labels section:
lb added to labels section:
sslproxy' added to labels section: 'io.rancher.scheduler.affinity:host_label: rancherpool=yes' Underregistry’ added to labels section:
After running rancher-compose -p registry up, everything came up.
Now everything is running just as I want it to.
I logged into the GUI and created my users and namespaces.
Was able to push to the registry using one of my namespaces. And I was able to pull that image to another computer. At first it seems like the items were not showing up in the registry. The first one took a really long time. But it did eventually show up. Subsequent ones showed up much faster.
Now I am a happy camper!