Rancher Release v2.5.8

Announcements
1

Release v2.5.8

Install/Upgrade Notes

  • Rancher install or upgrade must occur with Helm 3.2.x+ due to the changes with the latest cert-manager release. #29213
  • The local Kubernetes cluster for the Rancher server should be upgraded to Kubernetes 1.17+ before installing Rancher 2.5.
  • If using a proxy in front of an air gapped Rancher, you must pass additional parameters to NO_PROXY. #2725 Docs
  • The local cluster can no longer be turned off, which means all admins will have access to the local cluster. If you would like to restrict permissions to the local cluster, there is a new restricted-admin role that must be used. The access to local cluster can now be disabled by setting hide_local_cluster to true from the v3/settings API. #29325 Docs
  • For users upgrading from >=v2.4.4 to v2.5.x with clusters where ACI CNI is enabled, note that upgrading Rancher will result in automatic cluster reconciliation. This is applicable for Kubernetes versions v1.17.16-rancher1-1, v1.17.17-rancher1-1, v1.17.17-rancher2-1, v1.18.14-rancher1-1, v1.18.15-rancher1-1, v1.18.16-rancher1-1, and v1.18.17-rancher1-1. Please refer to the workaround BEFORE upgrading to v2.5.x. #32002
  • For users upgrading from <=v2.4.8 (<= RKE v1.1.6) to v2.4.12+ (RKE v1.1.13+)/v2.5.0+ (RKE v1.2.0+) , please note that Edit and save cluster (even with no changes or a trivial change like cluster name) will result in cluster reconciliation and upgrading kube-proxy on all nodes because of a change in kube-proxy binds. This only happens on the first edit and later edits shouldn’t affect the cluster.
    #32216
  • For installing or upgrading Rancher in an air gapped environment, please add the flag --no-hooks to the helm template command to skip rendering files for Helm’s hooks. #3226
  • There is currently a setting allowing users to configure the length of refresh time in cron format: eks-refresh-cron. That setting is now deprecated and has been migrated to a standard seconds format in a new setting: eks-refresh. If previously set, the migration will happen automatically. #31789
  • When upgrading <=v2.5.7 to >=v2.5.8, you may notice that in app & marketplace there is a fleet-agent release stuck at uninstalling. This is caused by migrating fleet-agent release name. It is safe to delete fleet-agent release as it is no longer used and it should not delete the real fleet-agent deployment since it has been migrated. #362

Docker Install

  • When starting the Rancher Docker container, the privileged flag must be used. See the docs for more info.
  • When installing in an air gapped environment, you must supply a custom registries.yaml file to the docker run command as shown in the K3s docs. If the registry has certs, then you will need to also supply those. #28969
  • There are UI issues around startup time. #28800 #28798

Kubernetes 1.19 + firewalld

  • For K8s 1.19 and newer, we recommend disabling firewalld as it has been found to be incompatible with various CNI plugins. #28840

Versions

Please refer to the README for latest and stable versions.

Please review our version documentation for more details on versioning and tagging conventions.

Enhancements

Monitoring Enhancements

  • Monitoring graphs have been added to the Cluster Explorer in the Rancher UI. #30126
  • A new RBAC role has been added to give users read-only access to monitoring components such as the Grafana dashboard. #31411 Docs
  • Persistent Grafana dashboards can now be created by adding a ConfigMap to any namespace, and a narrower permission named Manage Config Maps is required to create them. #31921 Docs
  • As of v2.5.8, users no longer need to manually annotate the cattle-dashboards namespace with helm.sh/resource-policy: "keep" to prevent it, and its associated resources, from being deleted when uninstalling the Monitoring chart. For users who are using Monitoring V2 v9.4.203 or below, the namespace still needs to be manually marked with the annotation helm.sh/resource-policy: "keep". #31769
  • You can now more easily set up SMS and Teams notifiers using our new rancher-alerting-drivers Helm chart that helps you install alerting drivers that are not natively supported by Prometheus. #29951
Windows Support for Monitoring V2
  • For metrics to be scraped from Windows nodes, each Windows node in the cluster must use wins v.0.1.0. For details on upgrading wins on Windows nodes, refer to this page. #31842
  • Metrics will be scraped from Windows nodes using windows_exporter (a community project previously named wmi_exporter when deployed in Monitoring V1). #31148 #31497
  • The Monitoring V2 application adds Linux nodeSelectors and tolerations throughout all monitoring components to ensure they are never deployed on a Windows host. #31498
  • We now allow a container to publish a port available on the host network as a container port. This allows privileged containers using wins to avoid exposing a port on the host itself when it only needs to be exposed on the container level. #13
  • As of v2.5.8, when Rancher provisions a new Windows cluster, the cluster is able to support wins upgrades and it is able to use the new Monitoring V2 chart with no changes. #31499

Enhanced GKE Lifecycle Management

Full lifecycle management has been brought to GKE clusters:

  • Shared VPCs are now supported.
  • We now support more options for configuring private GKE clusters. Note: This advanced setup can require more steps during the cluster provisioning process. For details, see this section.
  • We now support more configuration options for Rancher managed GKE clusters. For the full list of options, see the docs.
  • When provisioning a GKE cluster, you can now use reusable cloud credentials instead of using a service account token directly to create the cluster.
  • Greater management capabilities are now available for registered GKE clusters. The same configuration options are available for registered GKE clusters as for the GKE clusters created through the Rancher UI. Docs

Logging Enhancements

  • Upgraded the Banzai logging operator. #31644
  • Logging now works on SELinux enabled setups and has been tested on RHEL/CentOS 7 and 8. Note that If journald is configured in persistent mode on RKE2 cluster nodes you will need to set systemdLogPath to /var/log/journal. This is the default behavior in Ubuntu. #30949 #31309 Rancher RPM installation docs.
Windows Support for Logging V2
  • Added support for Windows clusters. Logs can now be collected from Windows Nodes. #28721

Istio Enhancements

  • Clusters with PSPs enabled are now supported. PSP templates have been added and can be enabled for Istio, Kiali, and Tracing. #30744
  • Added v1.9.3 and v1.8.5 for Istio, which both address a vulnerability. For more info see the security announcement. Only v1.9.3 supports an air gapped installation.

Backup Enhancements

  • Rancher Continuous Delivery (Fleet) and rancher-operator are now handled during backup. Backup#69

CIS Scan Enhancements

  • Support for CIS scans using the CIS 1.6 benchmark has been added for RKE2 and K3s clusters. #29649

Fleet Enhancements

  • All labels on Rancher clusters are available using global.fleet.clusterLabels.LABELNAME. These can now be accessed directly as variables. #152 Docs
  • Downstream Fleet agents now fall back to Linux nodes if Windows nodes are not yet available in Windows clusters. #324
  • Fleet now supports authentication with private Helm repositories. Note that SSH keys with passphrase are not supported. #120 Docs
  • Fleet now supports using webhook to receive changes from git. #252
  • Rancher agent and Fleet agent work behind a proxy. RFE tickets: #29993 and #25412

RKE Cluster Enhancements

Users can enable project network isolation on any RKE cluster if the CNI network plugin supports the enforcement of Kubernetes network policies. Previously, the Canal network plugin was required for project network isolation. #31338

Release Image Enhancements

  • Releases now include an annotated copy images.txt displaying the source of each image. #31663
  • Windows image digests are now included with releases. #25222

Security Fixes

  • A security issue was discovered in the current image of RANCHER/NGINX-INGRESS-CONTROLLER version 0.43.0-rancher1. This issue was assigned CVE-2021-3449 and CVE-2021-3540. To fix the issue, the openssl version of the image “nginx-0.43.0-rancher1” was updated from 1.1.1i to 1.1.1k. #538 #2522
  • Multiple critical and high CVEs were discovered in the rancher/library-nginx:1.19.2-alpine image. The image has since been updated to rancher/library-nginx:1.19.9-alpine and addresses the CVEs listed. #82 #440
    • Curl: CVE-2020-8231, CVE-2020-8285, CVE-2020-8169, CVE-2020-8286, and CVE-2020-8177
    • libxml2: CVE-2020-24977
    • freetype: CVE-2020-15999
    • musl: CVE-2020-28928

Major Bug Fixes

  • Fixed several UI issues.
  • Switched to deterministic naming strategy for ClusterRoleBinding and RoleBinding resources to mitigate duplicate issues. #29932
  • Uninstalling Rancher no longer leaves pods running in the cluster for apps like Fleet, Rancher-Operator or Rancher-Webhook. #30924
  • Logging in the Cluster Explorer when using RKE no longer creates large files in /var/lib/rancher/logging/rke. After upgrading, this file can be safely deleted. #31309
  • EKS contained a bug where certain regions failed during deployment when using default user data. #31612
  • When provisioning EC2 clusters, AMIs are now up to date. #31708
  • When provisioning RKE clusters with a private registry, the private registry is applied properly to all system images. #31726
  • Windows images for air gap are now available for charts maintained in rancher/charts. #32290
  • Deleting a nodepool with multiple nodes no longer incorrectly deletes only one node in the provider. #31765
  • After upgrading Rancher, new nodes are no longer stuck at the “waiting to register” stage. #31999
  • The private registry setting in RKE is now respected by kubectl/shell and Helm operation pods. #30735
  • GCS backups past retention now get deleted. #30565
  • Added NetworkPolicy resources to system project-based namespaces to allow for Hardened CIS Scan to pass when Project Network Isolation is enabled. 30211
  • Only delete Rancher-created NetworkPolicy resources when disabling Project Network Isolation. #30135
  • Domains in the AWS China region are now whitelisted to allow users to add and edit EKS clusters using cloud credentials associated with the AWS China region. #29666
  • Amazon EC2 nodes can now be automatically provisioned in the AWS Middle East region. #31980
  • PSP templates annotations are now properly applied to downstream cluster PSPs. Only annotations without names containing .cattle.io/ are applied. #22093

UI Updates

The primary UI in Rancher since v2.0 is now referred to as Cluster Manager in the UI. The new Cluster Explorer dashboard, experimentally released in Rancher 2.4, has graduated to GA status. There are new features only available in the new Cluster Explorer dashboard. There are some new features in the new UI with similar functionality as existing features in Cluster Manager, but differences in implementation may cause variation.

Duplicated Features in Cluster Manager and Cluster Explorer

  • Only one version of the feature may be installed at any given time due to potentially conflicting CRDs.
  • Each feature should only be managed by the UI that it was deployed from.
  • If you have installed the feature in Cluster Manager, you must uninstall it in Cluster Manager before attempting to install the new version in Cluster Explorer dashboard.

Other Notes

Deprecated Features

Feature Justification
Cluster Manager - Rancher Monitoring Monitoring in Cluster Manager UI has been replaced with a new monitoring chart available in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - Rancher Alerts and Notifiers Alerting and notifiers functionality is now directly integrated with a new monitoring chart available in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - Rancher Logging Functionality replaced with a new logging solution using a new logging chart available in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - MultiCluster Apps Deploying to multiple clusters is now recommended to be handled with Rancher Continuous Delivery powered by Fleet available in Cluster Explorer.
Cluster Manager - Kubernetes CIS 1.4 Scanning Kubernetes CIS 1.5+ benchmark scanning is now replaced with a new scan tool deployed with a cis benchmarks chart available in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - Rancher Pipelines Git-based deployment pipelines is now recommend to be handled with Rancher Continuous Delivery powered by Fleet available in Cluster Explorer.
Cluster Manager - Istio v1.5 The Istio project has ended support for Istio 1.5 and has recommended all users upgrade. Newer Istio versions are now available as a chart in the Apps & Marketplace in Cluster Explorer.
Cluster Manager - Provision Kubernetes v1.16 Clusters We have ended support for Kubernetes v1.16. Cluster Manager no longer provisions new v1.16 clusters. If you already have a v1.16 cluster, it is unaffected.

Known Major Issues

  • Logging (Cluster Explorer): Windows nodeAgents are not deleted when performing helm upgrade after disabling Windows logging on a Windows cluster. #32325
  • Rotating encryption keys with a custom encryption provider is not supported. #30539
  • Istio 1.5 is not supported in air gapped environments. Please note that the Istio project has ended support for Istio 1.5. Please see above in Deprecated Features.
  • In air gapped setups, the generated rancher-images.txt that is used to mirror images on private registries does not contain the images required to run Monitoring in Cluster Manager v0.1.x. Clusters running k8s 1.15 and below will need to upgrade their k8s versions and leverage Monitoring in Cluster Manager v0.2.x or upgrade to Monitoring in Cluster Explorer.
  • Importing a Kubernetes v1.21 cluster might not work properly. We are planning to add support for Kubernetes v1.21 in the future.
  • When registering EKS and GKE clusters via Terraform, all unmanaged fields must be set to empty in order to avoid overriding configuration in the registered cluster. #648
  • Deploying Monitoring V2 on a Windows cluster with win_prefix_path set requires users to deploy Rancher Wins Upgrader to restart wins on the hosts to start collecting metrics in Prometheus. #32535
  • Monitoring V2 fails to scrape ingress-nginx pods on any nodes except for the one Prometheus is deployed on if the security group used by worker nodes blocks incoming requests to port 10254. The workaround for this issue is to open up port 10254 on all hosts. #32563

Cluster Explorer Feature Caveats and Upgrades

  • General
    • Not all new features are currently installable on a hardened cluster.
    • New features are expected to be deployed using the Helm 3 CLI and not with the Rancher CLI.
  • Rancher Backup
    • When migrating to a cluster with the Rancher Backup feature, the server-url cannot be changed to a different location, it must continue to use the same URL.
  • Monitoring
    • Monitoring sometimes errors on installation because it can’t identify CRDs. #29171
  • Istio
    • When accessing tracing information for a service in the Kiali dashboard bundled with v1.9.3 and v1.8.5, attempting to change the display options may result in a persistent error for that service’s tracing information. We recommend using the Jaeger dashboard if you would like different details for a particular services tracing until this issue is resolved. The resolution for this issue can be found in #32330
    • Be aware that when upgrading from Istio 1.7.4 or earlier to any later version there may be connectivity issues. Upgrade notes #31811
    • Starting in v1.8.x, DNS is supported natively. This means the additional addon component istioCoreDNS is deprecated in v1.8.x and is not supported in v1.9x. If you are upgrading from v1.8.x to v1.9.x and you are using the istioCoreDNS addon, it is recommended that you disable it and switch to the natively supported DNS prior to upgrade. If you upgrade without disabling it, you will need to manually clean up your installation as it will not get removed automatically. #31761 #31265

Cluster Manager Feature Caveats and Upgrades

  • GKE
    • Basic authentication must be explicitly disabed in GCP before upgrading a GKE cluster to 1.19+ in Rancher. #32312
    • When creating GKE clusters in Terraform, the labels field cannot be empty, at least one label must be set #32553
  • EKS & GKE
    • When creating EKS and GKE clusters in Terraform, string fields cannot be set to empty. #32440

Versions within Rancher

Images

  • rancher/rancher:v2.5.8
  • rancher/rancher-agent:v2.5.8

Tools

Kubernetes

  • v1.20.6

Upgrades and Rollbacks

Rancher supports both upgrade and rollback. Please note the version you would like to upgrade or rollback to change the Rancher version.

Please be aware that upon an upgrade to v2.3.0+, any edits to a Rancher launched Kubernetes cluster will cause all system components to restart due to added tolerations to Kubernetes system components. Plan accordingly.

Recent changes to cert-manager require an upgrade if you have an HA install of Rancher using self-signed certificates. If you are using cert-manager older than v0.9.1, please see the documentation on how to upgrade cert-manager. Docs

Existing GKE clusters and imported clusters will continue to operate as-is. Only new creations and registered clusters will use the new full lifecycle management.

Important: When rolling back, we are expecting you to rollback to the state at the time of your upgrade. Any changes post upgrade would not be reflected.

1 Like