Rancher upgrade from 1.1.4 to 1.2 is not safe, stacks becomes unusable

This is our experience in upgrading our Rancher 1.1.4 to 1.2

It can serve as a warning to those planning doing an upgrade. We list what went well and what went wrong for us for now. We intentionally did the upgrade on a “rancher staging” environment so we know how to handle the uograde when going in the production one.

Saturday afternoon we upgraded our RancherDev to version 1.2.0. When we upgraded the Rancher server each environment is blocked until you click a button to upgrade it. It takes several minutes depending on the number of stacks.

What happens during the upgrade?

All stacks are rewritten to docker-compose v. 2 format. You can still use version 1 when deploying with rancher-compose for simple things.

The load-balancer configuration has changed a lot after the upgrade. The Docker image is now rancher/lb-service-haproxy and the routing information has moved from docker-compose.yml to rancher-compose.yml and has gotten more verbose. Whereas Rancher 1.1 had two sections for defaults and global, there is now one section. You now have to declare in the text what section you want to amend. Rancher will merge your configuration into the haproxy.cfg on a per-section basis. See v. 1.2 load balancers documentation.

Some of the load balancers with a modified ‘defaults’ section did not upgrade correct. Rancher inserted syntax errors.

The Convoy storage pool is still supported, but there is a new way called Rancher persistent storage, which has a lot more features such as service-specific storage shares.
While the stacks are stable in Devel 1, a container in the Convoy NFS keeps crashing every couple of minutes or so.
Some stacks are now non-functional and cannot be restarted.

I will probably post more details here as we go.
Please share your experience here.

4 Likes

Since an upgrade of Rancher 1.1.4 to 1.2 is quite risky and will create downtime and non-functional stacks, which all happens at once, we decided the following more “lean upgrade strategy”.

We will not upgrade our production rancher server 1.1.4. Instead we will setup a new fresh rancher server 1.2 and we slowly re-register the hosts and re-launch the stacks on this new server.

2 Likes