Rancher v1.5.x Users: Update on the Automatic Upgrades of Network-Services

Announcements
1

In the last 24 hours, Rancher v1.5.x has had 2 automatic upgrades triggered that were not intentional. We wanted to recap what happened to keep you informed as well as provide information on how we plan to prevent this in the future.

When adding in catalog templates for Rancher v1.6 in the master branch of the Rancher catalog, the default version of the network-services infrastructure was updated to the version tested with Rancher v1.6.0.

The mandatory/automatic upgrade logic was relying on the default version in the config.yml of the network-services catalog templates to determine which version should be used. The flaw in that logic was that the default version is not actually compatible with Rancher v1.5.x. Upon discovering the root cause of why the upgrades were occurring to an incompatible version, we corrected the default version to the correct version that is compatible with v1.5.x. This update to the default version caused a subsequent automatic upgrade of network-services, but it should have brought users back to the correct version that is compatible with their version of v1.5.x

How Rancher will be preventing this in the future:

  1. We are going to make specific branches for Rancher catalog per Rancher minor version. From a user’s perspective, you will see no difference as the Rancher catalog will be enabled, but it will be using specific branches for the infrastructure stacks.
  2. In v1.6.0, we have turned off the automatic upgrade for network-services, except for the Rancher versions that truly require this upgrade, i.e. Rancher versions 1.2.x, 1.3.x, 1.4.x).
  3. In v1.6.1, we are changing the upgrade.manager setting to the following
  • upgrade.manager=mandatory: This will be the default value and will upgrade any services that are deemed critical by Rancher.
  • upgrade.manager=false: Never auto upgrade any services. If there is a critical upgrade that is not applied, then Rancher may stop working properly.
  • upgrade.manager=true: Auto upgrade all the infrastructure services whenever there is an updated version. Not recommended for production installs

Verify that you’re running the valid network-services template

  • Users running Rancher v1.5.x should have v0.1.2 of network-services, which uses network-manager:v0.6.6.
  • Users running Rancher v1.6.0 should have v0.2.0 of network-services, which uses network-manager:v0.7.0.

If you are not running the correct version, the easiest way to get your correct version is to delete the network-services stack and re-launch it through Catalog -> Library.