Downstream Kubelet certificate expired and not rotating with generate_serving_certificate set to true

Rancher Server Setup

  • Rancher version: 2.6.6
  • Installation option (Docker install/Helm Chart): Helm Chart
    • RKE1, 1.21.5
  • Proxy/Cert Details:

Information about the Cluster

  • Kubernetes version: 1.21.5
  • Cluster Type (Local/Downstream): Custom Downstream
    • Iv1.21.5-rancher1-1
    • Deployed HA Kubernetes with RKE

User Information

  • What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom): Admin

Describe the bug
Hello, Our Downstream cluster had every kubelet’s exposed on 10250 certificate expired on every node few weeks ago (roughly a year after setting up the cluster).

We’ve found out that we need to have generate_serving_certificate set to true with rke for rancher in order to add the kubelet certificate to rancher’s rotation (see https://github.com/rancher/rancher/issues/33164 ).

After doing so, rancher UI now has an option to rotate the “kubelet” service in “rotate certificates”. After doing so the kubelet is still using outdated certificates, while the containers did get restarted the crt (/var/lib/kubelet/pki/kubelet.crt) and key (/var/lib/kubelet/pki/kubelet.key) in question did not get regenerated and cert is still expired. Any thoughts on process would be responsible for this so I could find the logs?

Had to edit the downstream cluster as yaml and set generate_serving_certificate to true, after that the kubelet certificates rotated.