Rancher Release - v2.2.7 - Addresses Kubernetes CVE-2019-11247 and CVE-2019-11249 and Rancher CVE-2019-14435 and CVE-2019-14436

Release v2.2.7

Important notes

  • This release comes with the latest Kubernetes versions, i.e. v1.13.9, v1.14.5, v1.15.2, for Rancher launched Kubernetes clusters to address Kubernetes CVE-2019-11247 and CVE-2019-11249. Rancher recommends upgrading all Kubernetes clusters to these Kubernetes versions.

  • This release addresses some security vulnerabilities found in Rancher:

    • CVE-2019-14436 - Project owner privilege escalation - This vulnerability allows a member of a project that has access to edit role bindings to be able to assign themselves or others a cluster level role granting them admin access to that cluster. The issue was found and reported by Michal Lipinski at Nokia. [#22026]
    • CVE-2019-14435 - This vulnerability allows authenticated users to potentially extract otherwise private data out of IPs reachable from system service containers used by Rancher. This can include but not only limited to services such as cloud provider metadata services. Although Rancher allow users to configure whitelisted domains for system service access, this flaw can still be exploited by a carefully crafted HTTP request. The issue was found and reported by Matt Belisle and Alex Stevenson at Workiva. [#22025]

As a result, the following versions are now latest and stable:

Type Rancher Version Docker Tag Helm Repo Helm Chart Version
Latest v2.2.7 rancher/rancher:latest server-charts/latest v2.2.7
Stable v2.2.7 rancher/rancher:stable server-charts/stable v2.2.7

Please review our version documentation for more details on versioning and tagging conventions.

Features and Enhancements

  • Added support for Docker 19.03 [#21805]
  • Added ability to set the s3 backup path [#21696]

Major Bugs Fixed Since v2.2.6

  • Fixed an issue where Rancher launched Kubernetes clusters would get stuck in updating state when enabling the --authorization-mode=Webhook flag on the kubelet.
  • Fixed an issue where backups no longer had timestamps on them [#21697]
  • Fixed an issue where node provisioning wasn’t working when certs were used in the Rancher install [#21729]
  • Fixed an issue where AKS clusters were failing to deploy in the correct resource group and location [#21636]

Other notes

Certificate expiry on Rancher provisioned clusters

In Rancher 2.0 and 2.1, the auto generated certificates for Rancher provisioned clusters have 1 year of expiry. It means if you created a Rancher provisioned cluster about 1 year ago, you need to rotate the certificates, otherwise the cluster will go into a bad state when the certificate expires. In Rancher 2.2.x, the rotation can be performed from Rancher UI, more details are here.

Additional Steps Required for Air Gap Installations and Upgrades

In v2.2.0, we’ve introduced a “system catalog” for managing micro-services that Rancher deploys for certain features such as Global DNS, Alerts, and Monitoring. These additional steps are documented as part of air gap installation instructions.

Known Major Issues

  • Cluster alerting and logging can get stuck in Updating state after upgrading Rancher. Workaround steps are provided in the issue [21480]
  • Certificate rotate for Rancher provisioned clusters will not work for the clusters which certificates had expired on Rancher versions v2.0.13 and earlier on 2.0.x release line, and 2.1.8 or earlier on 2.1.x release line. The issue won’t exist if the certificates expired on later versions of Rancher. Steps to workaround can be found in comments to [20381]
  • Catalog app revisions are not visible to the regular user; as a result regular user is not able to rollback the app [20204]
  • Global DNS entries are not properly updated when a node that was hosting an associated ingress becomes unavailable. A records to the unavailable hosts will remain on the ingress and in the DNS entry [#18932]
  • If you have Rancher cluster with OpenStack cloud provider having LoadBalancer set, and the cluster was provisioned on version 2.2.3 or less, the upgrade to the Rancher version v2.2.4 and up will fail. Steps to mitigate can be found in the comment to [20699]

Versions

Images

  • rancher/rancher:v2.2.7
  • rancher/rancher-agent:v2.2.7

Tools

System Charts Branch - For air gap installs

  • system charts branch - release-v2.2 - This is the branch used to populate the catalog items required for tools such as monitoring, logging, alerting and global DNS. To be able to use these features in an air gap install, you will need to mirror the system-charts repository to a location in your network that Rancher can reach and configure Rancher to use that repository.

Kubernetes

Upgrades and Rollbacks

Rancher supports both upgrade and rollback starting with v2.0.2. Please note the version you would like to upgrade or rollback to change the Rancher version.

Due to the HA improvements introduced in the v2.1.0 release, the Rancher helm chart is the only supported method for installing or upgrading Rancher. Please use the Rancher helm chart to install HA Rancher. For details, see the HA Install - Installation Outline.

If you are currently using the RKE add-on install method, see Migrating from a RKE add-on install for details on how to move to using a helm chart.

Any upgrade from a version prior to v2.0.3, when scaling up workloads, new pods will be created [#14136] - In order to update scheduling rules for workloads [#13527], a new field was added to all workloads on update, which will cause any pods in workloads from previous versions to re-create.

Note: When rolling back, we are expecting you to rollback to the state at the time of your upgrade. Any changes post upgrade would not be reflected. In the case of rolling back using a Rancher single-node install, you must specify the exact version you want to change the Rancher version to, rather than using the default :latest tag.

Note: If you had the helm stable catalog enabled in v2.0.0, we’ve updated the catalog to start pointing directly to the Kubernetes helm repo instead of an internal repo. Please delete the custom catalog that is now showing up and re-enable the helm stable. [#13582]