Rancher Release v2.6.8

[Jono-SUSE-Rancher]

Release v2.6.8

Rancher v2.6.8 is a mirror release of v2.6.7 to address the following issues:

Major Bug Fixes

• Fleet bundle pruning was too aggressive when GitJobs had multiple paths. This situation can lead to data loss. Note that this data loss only affects applications deployed through Fleet, including Longhorn. This has been fixed. Update to v2.6.8 or later to prevent potential data loss. See #933 for more details on the fix.
• Fixed an issue in which custom branding, color, and theme in Rancher Manager v2.6.7 was not correctly applied in the UI. See #6704 for more details.

Known Issue in RKE

• RKE v1.3.10-v1.3.13 in Rancher v2.6.7-v2.6.8 removes user addons during an upgrade before redeploying them. This causes issues with RKE1 cluster upgrades if user addons template has cattle-* or other system namespaces. See #38749 for more details.
• High CPU usage is observed on RKE1 v1.24.x clusters. Cluster could be a fresh installation or upgraded to v1.24.x. See #38816.

Known Issue in GKE

• Provisioning K8s 1.23+ GKE clusters with the default option of “Container-Optimized OS with Docker” for “Image Type” fails due to GKE no longer supporting Docker-based images per GKE documentation. Please use containerd-based images instead. See #38743.

It is important to review the Install/Upgrade Notes below before upgrading to any Rancher version.

In Rancher v2.6.4, the cluster-api module has been upgraded from v0.4.4 to v1.0.2 in which the apiVersion of CAPI CRDs are upgraded from cluster.x-k8s.io/v1alpha4 to cluster.x-k8s.io/v1beta1. This has the effect of causing rollbacks from Rancher v2.6.4 to any previous version of Rancher v2.6.x to fail because the previous version the CRDs needed to roll back are no longer available in v1beta1. To avoid this, the Rancher resource cleanup script should be run before the restore or rollback is attempted. This script can be found in the rancherlabs/support-tools repo and the usage of the script can be found in the backup-restore operator docs. In addition, when users roll back Rancher on the same cluster using the Rancher Backup and Restore app in 2.6.4+, the updated steps to create the Restore Custom Resource must be followed. See also #36803 for more details.

Security Fixes for Rancher Vulnerabilities

This release addresses three critical severity security issues found in Rancher:

• Fixed an issue where sensitive fields like passwords, API keys, and Rancher’s service account token were stored as plaintext on Kubernetes objects. Any user with read access to those objects in the Kubernetes API could retrieve the plaintext version of those sensitive data. For more information, see CVE-2021-36782.

• Improved the sanitization (removal) of credentials from cluster template answers. Failure to sanitize data can lead to plaintext storage and exposure of credentials, passwords, and API tokens. For more information, see CVE-2021-36783.

• Fixed an authorization logic flaw that allowed privilege escalation in downstream clusters through cluster role template binding (CRTB) and project role template binding (PRTB). For more information, see CVE-2022-31247.

For more details, see the Security Advisories page.

Features and Enhancements

Azure Active Directory API Migration

Microsoft has deprecated the Azure AD Graph API that Rancher had been using for authentication via Azure AD. A configuration update is necessary to make sure users can still use Rancher with Azure AD. See the docs and #29306 for details.

• Limitations

  • Attempts to log in will fail after rolling back a Docker install of Rancher if the following conditions have occurred:

    • Azure AD is enabled.
    • Before the rollback, admins committed to the Azure AD configuration update.

    This is because the Azure AD endpoints will not be rolled back if the rollback is not performed via the backup-restore operator. If you want to roll back Rancher to use the old Azure AD Graph API without using the backup-restore operator, follow this workaround to edit the AzureAD authconfig resource stored in the local cluster’s database. The old Azure AD Graph API endpoints will not be rolled back on a Rancher rollback. See #38025.

• Other
  • Multi-factor authentication (MFA) now works with the Azure AD auth provider. Some Rancher setups might have had MFA enabled in Azure from before, but Rancher wasn’t working with it correctly. Be aware that on upgrade, if MFA is enabled for the Azure app, Rancher will require additional verification. See #38028.
  • Before starting the migration process or enabling Azure AD for the first time in v2.6.7+, ensure that you add the Azure app registration’s permissions of type Application and NOT Delegated for Microsoft Graph. Otherwise, you may not be able to login to Azure AD. This issue will persist even after you disable/re-enable Azure AD and will require an hour wait, or manual deletion of a cache value to resolve.

Integration with Cloud Marketplaces

Rancher v2.6.7 introduces an integration allowing users to easily purchase support through the AWS marketplace for installation hosted on AWS/EKS. You must be running Rancher v2.6.7 or higher and have set up Rancher and it’s local cluster according to the prerequisites.

For details about the integration, refer to the Rancher documentation and #37495.

Note: If users are using the csp-adapter and the rancher backup-restore operator, they will need to upgrade the backup-restore operator to the latest version (v2.1.3) in order to ensure that the applications work together.

New in Rancher

• Support for Kubernetes v1.24 added.
• Support has ended for Kubernetes v1.18 and v1.19.
• Increased entropy of CSRF (cross-site request forgery) token. See #14 and #414
• Starting in v2.6.0, whenever a user requests a kubeconfig file, Rancher creates a newly-generated token instead of retrieving the old one. The token TTL is not configurable on these tokens, causing token cleanup to be a manual process. We’ve now added a new setting to allow users to change the TTL on kubeconfig tokens called kubeconfig-default-token-TTL-minutes. This setting has a default value of 0 to retain default behavior between Rancher versions. Rancher recommends that admins change this setting from its default to prevent unbound token creation. Note that this setting only applies to tokens generated for kubeconfigs when kubeconfig-generate-tokens is true, which is the default. When kubeconfig-generate-tokens is false, kubeconfig-token-ttl-minutes will be used for token TTL. This behavior is the same as previous versions of Rancher. The kubeconfig-token-ttl-minutes setting is now deprecated in favor of using kubeconfig-default-token-TTL-minutes in the future. See #37705.
• The Rancher chart now exposes the ingress.ingressClassName value, which allows setting the name of the ingress controller to be used with Rancher’s Ingress resource. This is relevant for Rancher clusters created with a provider other than RKE, since RKE automatically sets nginx as the ingress class name. By default, the value is an empty string because Rancher does not make assumptions about the type of ingress controller that runs in Rancher (nginx, Traefik, etc.). See #37971.
• Behavior Changes
  • The Kubernetes team has observed an increase in memory usage with Kubernetes v1.24. See the upstream changelog for details.
  • All Kubernetes 1.24 clusters will have cri-dockerd enabled by default which includes new and upgraded clusters. Users can apply the io.cattle.cluster.cridockerd.enable annotation on a cluster, and the annotation will override the default behavior. If the annotation is set to false and enable_cri_dockerd is set to true, the annotation will override the field/flag behavior and enable_cri_dockerd will be updated to false. Clusters will not provision correctly if cri-dockerd is disabled; this is expected unless a proper Docker runtime is provided. See #38160.

New in RKE1

• Resolved an issue for RKE clusters that prevented specifying more than one private registry in the YAML configuration. See #37658.
• Windows
  • Important: RKE1 Support for Windows will stop on September 1st 2022 due to upstream changes. See this article for more details.
  • A warning message has been added to inform users that Windows support is being deprecated for RKE1. See #5995.

New in RKE2

• New encryption key rotation feature added. See the docs and #35436.
• Windows
  • HostProcess containers are now supported in Kubernetes v1.24.1 and up. See #69.
• Behavior Changes
  • After an upgrade to Rancher v2.6.7, RKE2 provisioned clusters will briefly go into an Updating state with the message waiting for plan to be applied. This behavior is expected and has no adverse effects. See #38353.
• Known Issues
  • Encryption keys may fail to rotate when there are a large number (> 2000) of secrets. See #38283.
  • Users running RHEL/CentOS 7 should not install or upgrade to K3s/RKE2 v1.24.2 or v1.24.3 but should instead wait for K3s/RKE2 v1.24.4 to do so. See #5912.
  • The system-upgrade-controller Deployment may fail after Monitoring is enabled on an RKE2 v1.23 or v1.24 cluster with Windows nodes. See 38646.

New in the Rancher UI

• Removed monitoring dashboard “Rancher Internal State (Controllers)”. Most functionality from this dashboard has been replaced and can be found in the dashboard “Rancher Performance Debugging”. See #37274.
• ProjectHelmCharts has been added as a selectable resource from the helm.cattle.io API group when creating a new project/namespace role. See #5747.
• Added a Diagnostics page to allow users to gather data from their systems to append to any issues filed for Rancher. The Diagnostics page is accessible via the About page. See #6544.
• The Deployment creation screen has been improved and a new Pod creation view has been added. See #5734.
• When viewing the details of a GitRepo through Fleet, users can now get a graphical representation of the bundle deployments that came from that GitRepo. See #4680.
• Behavior Changes
  • Project owners and project members will no longer be able to see namespaces outside of the project(s) they have access to. This is to prevent a bad user experience, where some users could see namespaces that they could not use.
  • Project owners and project members will now be required to delete namespaces within a project when deleting the project. This is to prevent a situation where they would essentially be creating orphaned namespaces, which they would lose access to when they delete the project.

Major Bug Fixes

• User Preferences set by a drop-down component will now be applied correctly. Previously, updating a user preference in this manner would cause adverse effects such as the inability to view logs or the setting not taking effect. See #5984.
• Prior to v2.6.7, if S3 or other kinds of credentials were added to a cluster after it was already created, the reference to the secret containing the credentials was lost because the cluster status cannot be updated through the API. The references are now moved to the cluster Spec so that they can be updated after creation. To repair a cluster after a upgrade to v2.6.7, edit the cluster and change the etcd snapshot configuration back to local and save it, then edit again to configure S3 snapshots again. See #38215.
• Certificates with a CN exceeding 64 characters will not cause an error. See #37766.
• If the creation of the impersonation ClusterRoleBinding is interfered with or interrupted, users can now access the downstream cluster without experiencing unauthorized errors. See #37733.
• Resolved an issue where users that existed since Rancher v2.6.2 or earlier may start experiencing authorization errors upon upgrade to Rancher v2.6.5 when using kubectl with a downloaded kubeconfig for a downstream cluster. See #37894.
• Users with the role ‘Cluster Owners’ who are not also ‘Admins’ are now able to manage snapshots on RKE2 clusters. See #37630.
• A bug was found that overloaded the downstream Kubernetes API server when the Cluster Explorer dashboard is left open to a page for a downstream cluster for over 30 minutes and would start rapidly opening and closing watch requests perpetually. See #37627.
• Rancher server now generates a new token every time a kubeconfig is requested via the CLI. This token is then cached by the CLI on the local system and will not cause previously created tokens to become invalid. See #37245.
• Windows installation scripts are now successfully retrieved in proxied RKE2 downstream clusters. See #36574.
• The istiod-istio-system ValidatingWebhookConfiguration has been removed to allow Istio 1.11.x and higher to be installed in air-gapped environments. See #35742.
• Most API responses now set response headers to include Cache-Control: no-store which directs intermediate caches not to cache the response. Previously, some intermediate caches between the Rancher server and clients, including cluster agents, were configured to cache responses for the purpose of scalability and improved response time. In some cases this led to stale data getting inadvertently cached which would disrupt deployment of downstream clusters. See #35199.
• Project resource quotas are now correctly removed when deleted through the UI. See #35688.

Install/Upgrade Notes

• If you are installing Rancher for the first time, your environment must fulfill the installation requirements.
• The namespace where the local Fleet agent runs has been changed to cattle-fleet-local-system. This change does not impact GitOps workflows.

Upgrade Requirements

• Creating backups: We strongly recommend creating a backup before upgrading Rancher. To roll back Rancher after an upgrade, you must back up and restore Rancher to the previous Rancher version. Because Rancher will be restored to its state when a backup was created, any changes post upgrade will not be included after the restore. For more information, see the documentation on backing up Rancher.
• Helm version: Rancher install or upgrade must occur with Helm 3.2.x+ due to the changes with the latest cert-manager release. See #29213.
• Kubernetes version:
  • The local Kubernetes cluster for the Rancher server should be upgraded to Kubernetes 1.18+ before installing Rancher 2.6+.
• CNI requirements:
  • For Kubernetes v1.19 and newer, we recommend disabling firewalld as it has been found to be incompatible with various CNI plugins. See #28840.
  • If upgrading or installing to a Linux distribution which uses nf_tables as the backend packet filter, such as SLES 15, RHEL 8, Ubuntu 20.10, Debian 10, or newer, users should upgrade to RKE1 v1.19.2 or later to get Flannel version v0.13.0 that supports nf_tables. See Flannel #1317.
  • For users upgrading from >=v2.4.4 to v2.5.x with clusters where ACI CNI is enabled, note that upgrading Rancher will result in automatic cluster reconciliation. This is applicable for Kubernetes versions v1.17.16-rancher1-1, v1.17.17-rancher1-1, v1.17.17-rancher2-1, v1.18.14-rancher1-1, v1.18.15-rancher1-1, v1.18.16-rancher1-1, and v1.18.17-rancher1-1. Please refer to the workaround BEFORE upgrading to v2.5.x. See #32002.
• Requirements for air gapped environments:
  • For installing or upgrading Rancher in an air gapped environment, please add the flag --no-hooks to the helm template command to skip rendering files for Helm’s hooks. See #3226.
  • If using a proxy in front of an air gapped Rancher, you must pass additional parameters to NO_PROXY. See the documentation and related issue #2725.
• Cert-manager version requirements: Recent changes to cert-manager require an upgrade if you have a high-availability install of Rancher using self-signed certificates. If you are using cert-manager older than v0.9.1, please see the documentation on how to upgrade cert-manager. See documentation.
• Requirements for Docker installs:
  • When starting the Rancher Docker container, the privileged flag must be used. See documentation.
  • When installing in an air gapped environment, you must supply a custom registries.yaml file to the docker run command as shown in the K3s documentation. If the registry has certificates, then you will need to also supply those. See #28969.
  • When upgrading a Docker installation, a panic may occur in the container, which causes it to restart. After restarting, the container comes up and is working as expected. See #33685.

Rancher Behavior Changes

• Cert-Manager:
  • Rancher now supports cert-manager versions 1.6.2 and 1.7.1. We recommend v1.7.x because v 1.6.x will reach end-of-life on March 30, 2022. To read more, see the documentation.
  • When upgrading Rancher and cert-manager, you will need to use Option B: Reinstalling Rancher and cert-manager from the Rancher docs.
  • There are several versions of cert-manager which, due to their backwards incompatibility, are not recommended for use with Rancher. You can read more about which versions are affected by this issue in the cert-manager docs. As a result, only versions 1.6.2 and 1.7.1 are recommended for use at this time.
  • For instructions on upgrading cert-manager from version 1.5 to 1.6, see the relevant cert-manager docs.
  • For instructions on upgrading cert-manager from version 1.6 to 1.7, see the relevant cert-manager docs.
• Readiness and Liveness Check:
  • Users can now configure the Readiness Check and Liveness Check of coredns-autoscaler. See #24939.
• Legacy Features:
  • Users upgrading from Rancher <=v2.5.x will automatically have the --legacy feature flag enabled. New installations that require legacy features need to enable the flag on install or through the UI.
  • When workloads created using the legacy UI are deleted, the corresponding services are not automatically deleted. Users will need to manually remove these services. A message will be displayed notifying the user to manually delete the associated services when such a workload is deleted. See #34639.
• Library and Helm3-Library Catalogs:
  • Users will no longer be able to launch charts from the library and helm3-library catalogs, which are available through the legacy apps and multi-cluster-apps pages. Any existing legacy app that was deployed from a previous Rancher version will continue to be able to edit its currently deployed chart. Note that the Longhorn app will still be available from the library for new installs but will be removed in the next Rancher version. All users are recommended to deploy Longhorn from the Apps & Marketplace section of the Rancher UI instead of through the Legacy Apps pages.
• Local Cluster:
  • In older Rancher versions, the local cluster could be hidden to restrict admin access to the Rancher server’s local Kubernetes cluster, but that feature has been deprecated. The local Kubernetes cluster can no longer be hidden and all admins will have access to the local cluster. If you would like to restrict permissions to the local cluster, there is a new restricted-admin role that must be used. The access to local cluster can now be disabled by setting hide_local_cluster to true from the v3/settings API. See the documentation and #29325. For more information on upgrading from Rancher with a hidden local cluster, see the documentation.
• Upgrading the Rancher UI:
  • After upgrading to v2.6+, users will be automatically logged out of the old Rancher UI and must log in again to access Rancher and the new UI. See #34004.
• Fleet:
  • For users upgrading from v2.5.x to v2.6.x, note that Fleet will be enabled by default as it is required for operation in v2.6+. This will occur even if Fleet was disabled in v2.5.x. During the upgrade process, users will observe restarts of the rancher pods, which is expected. See #31044 and #32688.
  • Starting with Rancher v2.6.1, Fleet allows for two agents in the local cluster for scenarios where “Fleet is managing Fleet”. The true local agent runs in the new cattle-fleet-local-system namespace. The agent downstream from another Fleet management cluster runs in cattle-fleet-system, similar to the agent pure downstream clusters. See #34716 and #531.
• Editing and Saving Clusters:
  • For users upgrading from <=v2.4.8 (<= RKE v1.1.6) to v2.4.12+ (RKE v1.1.13+)/v2.5.0+ (RKE v1.2.0+) , please note that Edit and save cluster (even with no changes or a trivial change like cluster name) will result in cluster reconciliation and upgrading kube-proxy on all nodes because of a change in kube-proxy binds. This only happens on the first edit and later edits shouldn’t affect the cluster. See #32216.
• EKS Cluster:
  • There is currently a setting allowing users to configure the length of refresh time in cron format: eks-refresh-cron. That setting is now deprecated and has been migrated to a standard seconds format in a new setting: eks-refresh. If previously set, the migration will happen automatically. See #31789.
• System Components:
  • Please be aware that upon an upgrade to v2.3.0+, any edits to a Rancher launched Kubernetes cluster will cause all system components to restart due to added tolerations to Kubernetes system components. Plan accordingly.
• GKE and AKS Clusters:
  • Existing GKE and AKS clusters and imported clusters will continue to operate as-is. Only new creations and registered clusters will use the new full lifecycle management.
• Rolling Back Rancher:
  • The process to roll back Rancher has been updated for versions v2.5.0 and above. New steps require scaling Rancher down to 0 replica before restoring the backup. Please refer to the documentation for the new instructions.
• RBAC:
  • Due to the change of the provisioning framework, the Manage Nodes role will no longer be able to scale up/down machine pools. The user would need the ability to edit the cluster to manage the machine pools #34474.
• Azure Cloud Provider for RKE2:
  • For RKE2, the process to set up an Azure cloud provider is different than for RKE1 clusters. Users should refer to the documentation for the new instructions. See #34367 for original issue.
• Machines vs. Kube Nodes:
  • In previous versions, Rancher only displayed Nodes, but with v2.6, there are the concepts of machines and kube nodes. Kube nodes are the Kubernetes node objects and are only accessible if the Kubernetes API server is running and the cluster is active. Machines are the cluster’s machine object which defines what the cluster should be running.
• Rancher’s External IP Webhook:
  • In v1.22, upstream Kubernetes has enabled the admission controller to reject usage of external IPs. As such, the rancher-external-ip-webhook chart that was created as a workaround is no longer needed, and support for it is now capped to Kubernetes v1.21 and below. See #33893.
• Memory Limit for Legacy Monitoring:
  • The default value of the Prometheus memory limit in the legacy Rancher UI is now 2000Mi to prevent the pod from restarting due to a OOMKill. See #34850.
• Memory Limit for Monitoring:
  • The default value of the Prometheus memory limit in the new Rancher UI is now 3000Mi to prevent the pod from restarting due to a OOMKill. See #34850.
• **Snapshot

Versions

Please refer to the README for latest and stable versions.

Please review our version documentation for more details on versioning and tagging conventions.

Images

• rancher/rancher:v2.6.8

Tools

• CLI - v2.6.7
• RKE - v1.3.13

Kubernetes Versions

• v1.24.2 (Default)
• v1.23.8
• v1.22.11
• v1.21.14
• v1.20.15

Rancher Helm Chart Versions

Starting in 2.6.0, many of the Rancher Helm charts available in the Apps & Marketplace will start with a major version of 100. This was done to avoid simultaneous upstream changes and Rancher changes from causing conflicting version increments. This also brings us into compliance with semver, which is a requirement for newer versions of Helm. You can now see the upstream version of a chart in the build metadata, for example: 100.0.0+up2.1.0. See #32294.

Other Notes

Feature Flags

Feature flags introduced in 2.6.0 and the Harvester feature flag introduced in 2.6.1 are listed below for reference:

Feature Flag	Default Value	Description
harvester	true	Used to manage access to the Harvester list page where users can navigate directly to Harvester host clusters and have the ability to import them.
fleet	true	The previous fleet feature flag is now required to be enabled as the fleet capabilities are leveraged within the new provisioning framework. If you had this feature flag disabled in earlier versions, upon upgrading to Rancher, the flag will automatically be enabled.
gitops	true	If you want to hide the “Continuous Delivery” feature from your users, then please use the newly introduced gitops feature flag, which hides the ability to leverage Continuous Delivery.
rke2	true	Used to enable the ability to provision RKE2 clusters. By default, this feature flag is enabled, which allows users to attempt to provision these type of clusters.
legacy	false for new installs, true for upgrades	There are a set of features from previous versions that are slowly being phased out of Rancher for newer iterations of the feature. This is a mix of deprecated features as well as features that will eventually be moved to newer variations in Rancher. By default, this feature flag is disabled for new installations. If you are upgrading from a previous version, this feature flag would be enabled.
token-hashing	false	Used to enable new token-hashing feature. Once enabled, existing tokens will be hashed and all new tokens will be hashed automatically using the SHA256 algorithm. Once a token is hashed it cannot be undone. Once this feature flag is enabled it cannot be disabled.

Experimental Features

• Dual-stack and IPv6-only support for RKE1 clusters using the Flannel CNI will be experimental starting in v1.23.x. See the upstream Kubernetes docs. Dual-stack is not currently supported on Windows. See #165.

• RancherD was introduced as part of Rancher v2.5.4 through v2.5.10 as an experimental feature but is now deprecated. See #33423.

Legacy Features

Legacy features are features hidden behind the legacy feature flag, which are various features/functionality of Rancher that was available in previous releases. These are features that Rancher doesn’t intend for new users to consume, but if you have been using past versions of Rancher, you’ll still want to use this functionality.

When you first start 2.6, there is a card in the Home page that outlines the location of where these features are now located.

The deprecated features from v2.5 are now behind the legacy feature flag. Please review our deprecation policy for questions.

The following legacy features are no longer supported on Kubernetes v1.21+ clusters:

• Logging
• CIS Scans
• Istio 1.5
• Pipelines

The following legacy feature is no longer supported past Kubernetes v1.21+ clusters:

• Monitoring v1

Known Major Issues

• Kubernetes Cluster Distributions:
  • RKE:
    • Rotating encryption keys with a custom encryption provider is not supported. See #30539.
  • RKE1 - Windows:
    • OPA Gatekeeper gets stuck when uninstalled. See #37029.
  • RKE2:
    • Amazon ECR Private Registries are not functional. See #33920.
    • When provisioning using an RKE2 cluster template, the rootSize for AWS EC2 provisioners does not currently take an integer when it should, and an error is thrown. To work around this issue, wrap the EC2 rootSize in quotes. See Dashboard #3689.
    • RKE2 node driver cluster gets stuck in provisioning state after an upgrade to v2.6.4 and rollback to v2.6.3. See #36859.
    • RKE2 node driver cluster has its nodes redeployed when upgrading Rancher from v2.6.3 to v2.6.4. See #36627.
    • The communication between the ingress controller and the pods doesn’t work when you create an RKE2 cluster with Cilium as the CNI and activate project network isolation. See documentation and #34275.
  • RKE2 - Windows:
    • OPA Gatekeeper gets stuck when uninstalled. See #37029.
    • In v2.6.5, v1.21.x of RKE2 will remain experimental and unsupported for RKE2 Windows. End users should not use v1.21.x of RKE2 for any RKE2 cluster that will have Windows worker nodes. This is due to an upstream Calico bug that was not backported to the minor version of Calico (3.19.x) that is present in v1.21.x of RKE2. See #131.
    • CSI Proxy for Windows will now work in an air-gapped environment.
    • NodePorts do not work on Windows Server 2022 in RKE2 clusters due to a Windows kernel bug. See #159.
    • When upgrading Windows nodes in RKE2 clusters via the Rancher UI, Windows worker nodes will require a reboot after the upgrade is completed. See #37645.
  • AKS:
    • When editing or upgrading the AKS cluster, do not make changes from the Azure console or CLI at the same time. These actions must be done separately. See #33561.
    • Windows node pools are not currently supported. See #32586.
    • Azure Container Registry-based Helm charts cannot be added in Cluster Explorer, but do work in the Apps feature of Cluster Manager. Note that when using a Helm chart repository, the disableSameOriginCheck setting controls when credentials are attached to requests. See documentation and #34584 for more information.
  • GKE:
    • Basic authentication must be explicitly disabled in GCP before upgrading a GKE cluster to 1.19+ in Rancher. See #32312.
  • AWS:
    • On RHEL8.4 SELinux in AWS AMI, Kubernetes v1.22 fails to provision on AWS. As Rancher will not install RPMs on the nodes, users may work around this issue either by using AMI with this package already installed, or by installing AMI via cloud-init. Users will encounter this issue on upgrade to v1.22 as well. When upgrading to 1.22, users must manually upgrade/install the rancher-selinux package on all the nodes in the cluster, then upgrade the Kubernetes version. See #36509.
• Infrastructures:
  • vSphere:
    • PersistentVolumes are unable to mount to custom vSphere hardened clusters using CSI charts. See #35173.
  • Oracle:
    • Kubernetes 1.24 clusters fail to reach an Active state using Oracle Linux 8.4. See #38214.
• Harvester:
  • Upgrades from Harvester v0.3.0 are not supported.
  • Deploying Fleet to Harvester clusters is not yet supported. Clusters, whether Harvester or non-Harvester, imported using the Virtualization Management page will result in the cluster not being listed on the Continuous Delivery page. See #35049.
• Cluster Tools:
  • Fleet:
    • Multiple fleet-agent pods may be created and deleted during initial downstream agent deployment; rather than just one. This resolves itself quickly, but is unintentional behavior. See #33293.
  • Hardened clusters:
    • Not all cluster tools can currently be installed on a hardened cluster.
  • Rancher Backup:
    • When migrating to a cluster with the Rancher Backup feature, the server-url cannot be changed to a different location. It must continue to use the same URL.
    • When running a newer version of the rancher-backup app to restore a backup made with an older version of the app, the resourceSet named rancher-resource-set will be restored to an older version that might be different from the one defined in the current running rancher-backup app. The workaround is to edit the rancher-backup app to trigger a reconciliation. See #34495.
    • Because Kubernetes v1.22 drops the apiVersion apiextensions.k8s.io/v1beta1, trying to restore an existing backup file into a v1.22 cluster will fail because the backup file contains CRDs with the apiVersion v1beta1. There are two options to work around this issue: update the default resourceSet to collect the CRDs with the apiVersion v1, or update the default resourceSet and the client to use the new APIs internally. See documentation and #34154.
  • Monitoring:
    • Deploying Monitoring on a Windows cluster with win_prefix_path set requires users to deploy Rancher Wins Upgrader to restart wins on the hosts to start collecting metrics in Prometheus. See #32535.
  • Logging:
    • Windows nodeAgents are not deleted when performing helm upgrade after disabling Windows logging on a Windows cluster. See #32325.
  • Istio Versions:
    • Istio 1.12 and below do not work on Kubernetes 1.23 clusters. To use the Istio charts, please do not update to Kubernetes 1.23 until the next charts’ release.
    • Istio 1.5 is not supported in air-gapped environments. Please note that the Istio project has ended support for Istio 1.5.
    • Istio 1.9 support ended on October 8th, 2021.
    • Deprecated resources are not automatically removed and will cause errors during upgrades. Manual steps must be taken to migrate and/or cleanup resources before an upgrade is performed. See #34699.
    • Applications injecting Istio sidecars, fail on SELinux RHEL 8.4 enabled clusters. A temporary workaround for this issue is to run the following command on each cluster node before creating a cluster: mkdir -p /var/run/istio-cni && semanage fcontext -a -t container_file_t /var/run/istio-cni && restorecon -v /var/run/istio-cni. See #33291.
  • Legacy Monitoring:
    • The Grafana instance inside Cluster Manager’s Monitoring is not compatible with Kubernetes v1.21. To work around this issue, disable the BoundServiceAccountTokenVolume feature in Kubernetes v1.21 and above. Note that this workaround will be deprecated in Kubernetes v1.22. See #33465.
    • In air gapped setups, the generated rancher-images.txt that is used to mirror images on private registries does not contain the images required to run Legacy Monitoring which is compatible with Kubernetes v1.15 clusters. If you are running Kubernetes v1.15 clusters in an air gapped environment, and you want to either install Legacy Monitoring or upgrade Legacy Monitoring to the latest that is offered by Rancher for Kubernetes v1.15 clusters, you will need to take one of the following actions:
      • Upgrade the Kubernetes version so that you can use v0.2.x of the Monitoring application Helm chart.
      • Manually import the necessary images into your private registry for the Monitoring application to use.
    • When deploying any downstream cluster, Rancher logs errors that seem to be related to Monitoring even when Monitoring is not installed onto either cluster; specifically, Rancher logs that it failed on subscribe to the Prometheus CRs in the cluster because it is unable to get the resource prometheus.meta.k8s.io. These logs appear in a similar fashion for other Prometheus CRs (namely Alertmanager, ServiceMonitors, and PrometheusRules), but do not seem to cause any other major impact in functionality. See #32978.
    • Legacy Monitoring does not support Kubernetes v1.22 due to the feature-gates flag no longer being supported. See #35574.
    • After performing an upgrade to Rancher v2.6.3 from v2.6.2, the Legacy Monitoring custom metric endpoint stops working. To work around this issue, delete the service that is being targeted by the servicemonitor and allow it to be recreated; this will reload the pods that need to be targeted on a service sync. See #35790.
• Docker Installations:
  • UI issues may occur due to a longer startup time. User will receive an error message when launching Docker for the first time #28800, and user is directed to username/password screen when accessing the UI after a Docker install of Rancher. See #28798.
  • On a Docker install upgrade and rollback, Rancher logs will repeatedly display the messages “Updating workload ingress-nginx/nginx-ingress-controller” and “Updating service frontend with public endpoints”. Ingresses and clusters are functional and active, and logs resolve eventually. See #35798.
  • Rancher single node wont start on Apple M1 devices with Docker Desktop 4.3.0 or newer. See #35930.
• Rancher UI:
  • After installing an app from a partner chart repo, the partner chart will upgrade to feature charts if the chart also exists in the feature charts default repo. See #5655.
  • In some instances under Users and Authentication, no users are listed and clicking Create to create a new user does not display the entire form. To work around this when encountered, perform a hard refresh to be able to log back in. See #37531.
  • Deployment securityContext section is missing when a new workload is created. This prevents pods from starting when Pod Security Policy Support is enabled. See #4815.
• Legacy UI:
  • When using the Rancher v2.6 UI to add a new port of type ClusterIP to an existing Deployment created using the legacy UI, the new port will not be created upon saving. To work around this issue, repeat the procedure to add the port again. Users will notice the Service Type field will display as Do not create a service. Change this to ClusterIP and upon saving, the new port will be created successfully during this subsequent attempt. See #4280.