Rancher Release v2.5.5

Announcements
1

Release v2.5.5

Rancher 2.5.5 is a mirror release of 2.5.4 to address a few minor issues:

  • The dashboard could crash if helm charts were missing certain annotations. link
  • The rancher helm chart was restricted to kubernetes version less than 1.20.0 link
  • Rancher’s cluster explorer now display only windows compatible charts on windows clusters

Important

The primary UI in Rancher since version 2.0 is now called Cluster Manager. Our new Cluster Explorer dashboard, experimentally released in Rancher 2.4, has graduated to GA status. There are new features only available in the new Cluster Explorer dashboard. Some of these new features are similar in functionality to existing features in the Cluster Manager and we will try to differentiate them based on where they are located in the UI.

Install/Upgrade Notes

  • Rancher install or upgrade must occur with Helm 3.2.x+ due to the changes with the latest cert-manager release. #29213
  • Rancher HA cluster should be upgraded to Kubernetes 1.17+ before installing Rancher 2.5.
  • If using a proxy in front of an air-gapped Rancher, you must pass additional parameters to NO_PROXY. #2725
  • The local cluster can no longer be turned off, which means all admins will have access to the local cluster. If you would like to restrict permissions to the local cluster, there is a new restricted-admin role that must be used. The access to local cluster can now be disabled by setting hide_local_cluster to true from the v3/settings API. #29325

Docker Install

  • When starting the Rancher Docker container, the privileged flag must be used. See the docs for more info
  • When installing in an air gap environment, you must supply a custom registries.yaml file to the Docker run command as shown in the k3s docs. If the registry has certs, then you will need to also supply those. #28969
  • There are UI issues around startup time #28800, #28798

Duplicated Features in Cluster Manager and Cluster Explorer

  • Only 1 version of the feature may be installed at any given time due to potentially conflicting CRDs.
  • Each feature should only be managed by the UI that it was deployed from.
  • If you have installed the feature in Cluster Manager, you must uninstall in Cluster Manager before attempting to install the new version in Cluster Explorer dashboard.

Kubernetes 1.19

  • For K8s 1.19 and newer, we recommend disabling firewalld as it has been found to be incompatible with various CNI plugins. #28840
  • Certain alerts in Cluster Manager are not working with k8s 1.19 as certain metrics have changed in Kubernetes 1.19 #29292

Deprecated Features

Feature Justification
Cluster Manager - Rancher Monitoring Monitoring in Cluster Manager UI has been replaced with a new monitoring chart available in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - Rancher Alerts and Notifiers Alerting and notifiers functionality is now directly integrated with a new monitoring chart available in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - Rancher Logging Functionality replaced with a new logging solution using a new logging chart available in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - MultiCluster Apps Deploying to multiple clusters is now recommended to be handled with Rancher Continuous Delivery powered by Fleet available in Cluster Explorer.
Cluster Manager - Kubernetes CIS 1.4 Scanning Kubernetes CIS 1.5+ benchmark scanning is now replaced with a new scan tool deployed with a cis benchmarks chart available in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - Rancher Pipelines Git-based deployment pipelines is now recommend to be handled with Rancher Continuous Delivery powered by Fleet available in Cluster Explorer.
Cluster Manager - Istio v1.5 The Istio project has ended support for Istio 1.5 and has recommended all users upgrade. Istio 1.7 is now available as an Istio chart in the Apps & Marketplace in Cluster Explorer.

Versions

The following versions are now latest and stable:

Type Rancher Version Docker Tag Helm Repo Helm Chart Version
Latest v2.5.5 rancher/rancher:latest server-charts/latest v2.5.5
Stable v2.5.5 rancher/rancher:stable server-charts/stable v2.5.5

Please review our version documentation for more details on versioning and tagging conventions.

Major Bug Fixes

  • Switched to xml-roundtrip-validator package to address xml vulnerabilities in go’s xml package - link
  • Fixed intermittent installation fails with cluster manager monitoring. #30188
  • Fixed cluster explorer monitoring installation errors on hardened clusters. #29980
  • Fixed error when execing into pods via kubectl shell. #29668
  • Rancher now supports startTLS when using OpenLDAP. #29153
  • Fixed error message in cluster manager monitoring when secret webhook-receiver is not found. #28954
  • Rancher pods are no long scheduled on dead nodes. #27734
  • Noes running Amazon Linux no longer fail to provision. #21648
  • Fixed cluster manager logging support for clusters with Windows Server SAC 1909 nodes and for clusters with Windows Server LTSC 2019 nodes #30279
  • Fixed cluster manager monitoring’s prometheus operator container startup error with nfs subpaths #29149

Other Notes

Known Major Issues

  • Istio deployments can fail if sidecar injection is enabled in a cluster with a PSP where FS Group is set to MustRunAs with a range of 1..1024 #29700
  • Cluster explorer logging may not capture all kubelet logs for cloud providers. #30383
  • Rotating encryption keys with a custom encryption can fail. #30539

Cluster Explorer Feature Caveats and Upgrades

  • General
    • Not all new features are currently installable on a hardened cluster.
    • New features are expected to be deployed using the Helm3 CLI and not with the Rancher CLI
    • The new Logging and Monitoring features do not yet work with windows clusters. #28721 #28327
    • An error can be seen during cluster provisioning intermittently, and it recovers within a couple of minutes without any user intervention. #28836
  • Rancher Backup
    • When migrating to a cluster with the Rancher Backup feature, the server-url cannot be changed to a different location, it must continue to use the same URL.
    • Rancher Continuous Delivery (Fleet) is not handled during backup. Backup#46
  • Monitoring
    • There is a known issue with using Rancher Monitoring to monitor etcd nodes in clusters that use RancherOS hosts for etcd #29815. If your etcd plane only consists of RancherOS hosts, a workaround for this issue can be found here. However there is no existing workaround for clusters that use a mix of RancherOS and non-RancherOS hosts in their etcd plane.
    • To continue to support users who are using version 9.4.200 of the Rancher Monitoring chart, the default memory limits for k3s clusters set by the Dashboard UI have been increased to 2500Mi. This increased limit is not required for users who upgrade to chart version 9.4.201 but is recommended. #28787
    • Monitoring sometimes errors on installation because it can’t identify CRDs. #29171

Enhancements

  • CIS v2 enhancements – Schedule/Alert, Custom Benchmark #30123
  • CIS 1.6 for RKE1 #29649
  • CIS 1.5 for RKE2 #29649
  • Istio enhancements - CNI, Jaeger ​support #29903 #27377
  • Cluster Explorer logging enhancements - Syslog support and GKE/AKS node logs #29892 #28574 #28573
  • Tolerations can now be configured per add-on and apply to Deployment resources. The configured tolerations will replace the existing tolerations so make sure you configure all the tolerations you need. This resolves an issue where add-on pods with wildcard tolerations (toleration that tolerates all taints) were not evicted and if manually deleted, could be scheduled to a NotReady node. To avoid unschedulable pods on upgrade, the default configuration will be the same (so with wildcard tolerations) so you will need to configure tolerations to override the default configuration. See the documentation for more information. #27734
  • RKE encryption keys can now be rotated from the rancher API #27735
  • The default nginx http backend can optionally be disabled. It is installed by default. #25590
  • Dashboard supports configuring additional alerting receivers - link
  • Dashboard now supports additional logging outputs - link
  • UI polishing in Cluster Explorer - link

Versions

Images

  • rancher/rancher:v2.5.5
  • rancher/rancher-agent:v2.5.5

Tools

Kubernetes

  • 1.19.6 (default)
  • 1.18.14
  • 1.17.16

Bug Fixes

  • Fixed increased CPU and memory usage #30652
  • Suppressed excessive “rejected by webhook” messages #30452
  • the reset-admin url is no longer a private ip #29162
  • Fixed panic when provisioning RKE clusters #30143
  • Resolved error when installing RKE on node running flatcar os #29841 #30326
  • Fixed error when creating hosted EKS V2 cluster with the same name as an existing EKS cluster #29549
  • Prevented existing clusters in eks from being updated when a cluster with the same name is created with rancher #29103
  • Suppressed constant deprecation log #28943 #29144
  • Fixed rule deployer errors when enabling alerts #29318
  • Fixed error when installing OPA gatekeeper v2 after v1 #29188
  • Fixed nil pointer error when provisioning rke clusters #30373
  • Stopped using deprecated networking resources #22487
  • Fixed RKE missing ingress on install #30405
  • Fixed issue when upgrading from rancher 2.5.2 #30434
  • Fixed panic when deleting a node that is currently being provisioned #30430
  • reset-password no longer causes a panic #29565
  • EKSv2 now works with private-only endpoint​s #29907" can be changed to "Removed the necessity of extra network configuration for private-only endpoint​s in EKSv2 #29907
  • added missing images to images.txt #30496
  • Numerous dashboard bug fixes - link
  • Numerous UI fixes - link

Upgrades and Rollbacks

Rancher supports both upgrade and rollback. Please note the version you would like to upgrade or rollback to change the Rancher version.

Please be aware that upon an upgrade to v2.3.0+, any edits to a Rancher launched Kubernetes cluster will cause all system components to restart due to added tolerations to Kubernetes system components. Plan accordingly.

Recent changes to cert-manager require an upgrade if you have an HA install of Rancher using self-signed certificates. If you are using cert-manager older than v0.9.1, please see the documentation on how to upgrade cert-manager.

Important: When rolling back, we are expecting you to rollback to the state at the time of your upgrade. Any changes post upgrade would not be reflected.