Rancher Release v2.5.3

Announcements
1

Release v2.5.3

Only addresses a high cpu usage issue in v2.5.2 for upgraded clusters with Project Network Isolation Enabled #30048, #30052, #30089. Please note that Release notes remain the same as v2.5.2.
Note: This requires redeploying nginx-ingress-controller. Please refer to this comment for more details.

Important

The primary UI in Rancher since version 2.0 is now called Cluster Manager. Our new Cluster Explorer dashboard, experimentally released in Rancher 2.4, has graduated to GA status. There are new features only available in the new Cluster Explorer dashboard. Some of these new features are similar in functionality to existing features in the Cluster Manager and we will try to differentiate them based on where they are located in the UI.

Install/Upgrade Notes

  • Rancher install or upgrade must occur with Helm 3.2.x+ due to the changes with the latest cert-manager release. #29213
  • Rancher HA cluster should be upgraded to Kubernetes 1.17+ before installing Rancher 2.5.
  • If using a proxy in front of an air-gapped Rancher, you must pass additional parameters to NO_PROXY. #2725
  • The local cluster can no longer be turned off, which means all admins will have access to the local cluster. If you would like to restrict permissions to the local cluster, there is a new restricted-admin role that must be used. The access to local cluster can now be disabled by setting hide_local_cluster to true from the v3/settings API. #29325

Docker Install

  • When starting the Rancher Docker container, the privileged flag must be used. See the docs for more info
  • When installing in an air gap environment, you must supply a custom registries.yaml file to the Docker run command as shown in the k3s docs. If the registry has certs, then you will need to also supply those. #28969
  • There are UI issues around startup time #28800, #28798

Duplicated Features in Cluster Manager and Cluster Explorer

  • Only 1 version of the feature may be installed at any given time due to potentially conflicting CRDs.
  • Each feature should only be managed by the UI that it was deployed from.
  • If you have installed the feature in Cluster Manager, you must uninstall in Cluster Manager before attempting to install the new version in Cluster Explorer dashboard.

Kubernetes 1.19

  • For K8s 1.19 and newer, we recommend disabling firewalld as it has been found to be incompatible with various CNI plugins. #28840
  • Certain alerts in Cluster Manager are not working with k8s 1.19 as certain metrics have changed in Kubernetes 1.19 #29292

Deprecated Features

Feature Justification
Cluster Manager - Rancher Monitoring Monitoring in Cluster Manager UI has been replaced with a new monitoring chart available in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - Rancher Alerts and Notifiers Alerting and notifiers functionality is now directly integrated with a new monitoring chart available in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - Rancher Logging Functionality replaced with a new logging solution using a new logging chart available in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - MultiCluster Apps Deploying to multiple clusters is now recommended to be handled with Rancher Continuous Delivery powered by Fleet available in Cluster Explorer.
Cluster Manager - Kubernetes CIS 1.4 Scanning Kubernetes CIS 1.5+ benchmark scanning is now replaced with a new scan tool deployed with a cis benchmarks chart available in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - Rancher Pipelines Git-based deployment pipelines is now recommend to be handled with Rancher Continuous Delivery powered by Fleet available in Cluster Explorer.
Cluster Manager - Istio v1.5 The Istio project has ended support for Istio 1.5 and has recommended all users upgrade. Istio 1.7 is now available as an Istio chart in the Apps & Marketplace in Cluster Explorer.

Versions

The following versions are now latest and stable:

Type Rancher Version Docker Tag Helm Repo Helm Chart Version
Latest v2.5.3 rancher/rancher:latest server-charts/latest v2.5.3
Stable v2.5.3 rancher/rancher:stable server-charts/stable v2.5.3

Please review our version documentation for more details on versioning and tagging conventions.

Experimental Features

  • OPA Gatekeeper: Users can deploy and manage the updated GA version of OPA Gatekeeper through Rancher. Users must uninstall the first Rancher installed version OPA Gatekeeper before installing this new feature.

  • RancherD: A single binary installation of Rancher. Admins create 1 or 3 hosts, and start the RancherD binary to perform all the work of installing Rancher. Check out this blog article for more details.

Major Bugs Fixed Since v2.5.1

  • Local cluster will no longer run the cattle cluster-agent and node-agent. On upgrade to 2.5.2 these pods will get deleted. #29397
  • If you are using Rancher to manage other Rancher instances, you can now upgrade Rancher without facing elevated CPU, load or network issues. #29364.
  • Cluster Explorer’s Monitoring feature can now be installed on a K8s 1.16 cluster. #29395
  • Cluster Explorer’s Monitoring feature can now be installed in a hardened cluster using chart version 9.4.201 and up. #28536
  • Istio can now be installed with the Ingress Gateway disabled. #29383
  • Fixed a bug where Longhorn uninstallation was getting stalled. Longhorn#1820
  • Rancher now supports deploying an EKS cluster in an air gap environment. #29070
  • Fixed a bug where the Auto Replace feature of Node Pools wasn’t working as expected in 2.5.1. #29754
  • You can now run a forked build of the UI and set ui-index setting to local and force that to load. #29362
  • Fixed a bug where Launch kubectl feature wasn’t working as expected. #29511
  • Fixed a bug where Windows worker nodes could not join a cluster with a cloud provider enabled. #29782
  • Fixed a bug for the Rancher Terraform provider where node draining could not be disabled. Terraform#440
  • Fixed a bug where private Git repo in fleet didn’t work when adding http/ssh credential in dashboard. Fleet#134

Other notes

Known Major Issues

  • Cluster Manager’s Monitoring stack does not install on the local cluster if it is K3s #29328
  • The setting to hide local cluster can only be set through API after Rancher is installed. It cannot be set during install because of an issue with Rancher API stripping quotes off of helm values. #29325
  • When provisioning EKS clusters, Rancher currently relies on public endpoints to connect to the cluster. Disabling public access is not recommended if Rancher does not have network access, as it will affect Rancher’s ability to communicate with the cluster.
    #29907
  • When Project Network Isolation is turned on, upgrading from a previous Rancher version to 2.5.2 will cause an increase in CPU / Logging. Workaround is turn off PNI. #30052. Fix is tracked in #30045.

Cluster Explorer Feature Caveats and Upgrades

  • General
    • Not all new features are currently installable on a hardened cluster.
    • New features are expected to be deployed using the Helm3 CLI and not with the Rancher CLI
    • The new Logging and Monitoring features do not yet work with windows clusters. #28721 #28327
    • An error can be seen during cluster provisioning intermittently, and it recovers within a couple of minutes without any user intervention. #28836
  • Rancher Backup
    • When migrating to a cluster with the Rancher Backup feature, the server-url cannot be changed to a different location, it must continue to use the same URL.
    • Rancher Continuous Delivery (Fleet) is not handled during backup. Backup#46
  • Monitoring
    • There is a known issue with using Rancher Monitoring to monitor etcd nodes in clusters that use RancherOS hosts for etcd #29815. If your etcd plane only consists of RancherOS hosts, a workaround for this issue can be found here. However there is no existing workaround for clusters that use a mix of RancherOS and non-RancherOS hosts in their etcd plane.
    • To continue to support users who are using version 9.4.200 of the Rancher Monitoring chart, the default memory limits for k3s clusters set by the Dashboard UI have been increased to 2500Mi. This increased limit is not required for users who upgrade to chart version 9.4.201 but is recommended. #28787
    • Monitoring sometimes errors on installation because it can’t identify CRDs. #29171
  • OPA Gatekeeper (Experimental)
    • The first edition of OPA must be uninstalled before the new OPA features are installed. #29188

Enhancements

  • Admins can now choose to use the CentOS/RHEL 8 operating system with Docker CE for both, the local Rancher HA cluster and clusters provisioned by Rancher. Please note this is only available for Docker CE 19.03.13 and higher. To be able to use our Docker install script to install Docker CE 19.03.13, if the OS image you are using contains the package runc, it needs to be removed manually before you can use the install script, because the containerd package will conflict with this package. The requirement of installing the package iptables is handled in the install script. #23045
  • Stats will now aggregate resources of all nodes that do not have the following four taints instead of relying on the worker node-role label: #29139
  • Starting with version 9.4.201 of the Rancher Monitoring chart, k3s clusters will switch to using one PushProx exporter instead of three PushProx exporters so increased memory limits are no longer required for k3s clusters. #29445
  • Keycloak SAML provider now accepts a custom Entity ID field. #29212
  • Logs now include data about the source. #29410
  • UI images path can now be specified in the Rancher Helm chart. #29419
  • Istio installer automatically adds Istio dashboards to Grafana when using Cluster Explorer’s Monitoring feature. #28468

Versions

Images

  • rancher/rancher:v2.5.3
  • rancher/rancher-agent:v2.5.3

Tools

Kubernetes

Upgrades and Rollbacks

Rancher supports both upgrade and rollback. Please note the version you would like to upgrade or rollback to change the Rancher version. There are different rollback instructions for Rancher versions 2.5.0 or newer and for versions 2.4.x or earlier.

Important: When rolling back, we are expecting you to rollback to the state at the time of your upgrade. Any changes post upgrade would not be reflected.