Rancher Update on docker Host lost all Config Data

jme · January 7, 2022, 8:51am

Hi,

I installed Rancher 2.6.2 on a single Docker Host with:

docker run -d --privileged --restart=unless-stopped -p 80:80 -p 443:443 -v /opt/rancher:/var/lib/rancher --name rancher rancher/rancher:stable --no-cacerts

My Setup is Apache Reverse Proxy (SSL Termination) > Docker Rancher Container Port 80 to access the UI. All was good and I configured my Rancher Vsphere K8 Cluster.

Now I tried to update to the latest 2.6.3 Docker Image and followed this guide:
Rancher Update Guide

But it seems that I lost all my configured Data. When I access the UI, I get:

It looks like this is your first time visiting Rancher; if you pre-set your own bootstrap password, enter it here. Otherwise a random one has been generated for you. To find it:



For a "docker run" installation:

My Update process was:

docker stop rancher
docker create --volumes-from rancher --name rancher-data rancher/rancher:stable
docker pull rancher/rancher:v2.6.3

### Option A
docker run -d --volumes-from rancher-data --restart=unless-stopped  -p 80:80 -p 443:443 --privileged  rancher/rancher:v2.6.3

and tried Option C

docker run -d --volumes-from rancher-data --restart=unless-stopped -p 80:80 -p 443:443  -v /etc/rancher/ssl/cert.pem:/etc/rancher/ssl/cert.pem -v /etc/rancher/ssl/key.key:/etc/rancher/ssl/key.pem --privileged rancher/rancher:v2.6.3 --no-cacerts

But with both Options no success. And all previous data are gone. I did a snapshot of the whole Docker VM, so I can revert.

But what I did wrong? Hope you can Help me.
Thank You
Jan

LOG:

2022/01/07 08:32:53 [INFO] Rancher version v2.6.3 (3c1d5fac3) is starting
2022/01/07 08:32:53 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
2022/01/07 08:32:53 [INFO] Listening on /tmp/log.sock
2022/01/07 08:32:53 [INFO] Waiting for k3s to start
2022/01/07 08:32:54 [INFO] Waiting for k3s to start
2022/01/07 08:32:55 [INFO] Waiting for k3s to start
2022/01/07 08:32:56 [INFO] Waiting for k3s to start
2022/01/07 08:32:57 [INFO] Waiting for server to become available: an error on the server ("apiserver not ready") has prevented the request from succeeding
2022/01/07 08:32:59 [INFO] Waiting for server to become available: an error on the server ("apiserver not ready") has prevented the request from succeeding
2022/01/07 08:33:11 [INFO] Running in single server mode, will not peer connections
2022/01/07 08:33:12 [INFO] Applying CRD features.management.cattle.io
2022/01/07 08:33:12 [INFO] Waiting for CRD features.management.cattle.io to become available
2022/01/07 08:33:12 [INFO] Done waiting for CRD features.management.cattle.io to become available
2022/01/07 08:33:14 [INFO] Applying CRD navlinks.ui.cattle.io
2022/01/07 08:33:14 [INFO] Applying CRD clusters.management.cattle.io
2022/01/07 08:33:14 [INFO] Applying CRD apiservices.management.cattle.io
2022/01/07 08:33:14 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
2022/01/07 08:33:14 [INFO] Applying CRD settings.management.cattle.io
2022/01/07 08:33:14 [INFO] Applying CRD preferences.management.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD features.management.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD clusterrepos.catalog.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD operations.catalog.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD apps.catalog.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD fleetworkspaces.management.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD bundles.fleet.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD clusters.fleet.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD managedcharts.management.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD clusters.provisioning.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD clusters.provisioning.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD rkeclusters.rke.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD rkebootstraps.rke.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD custommachines.rke.cattle.io
2022/01/07 08:33:15 [INFO] Applying CRD clusters.cluster.x-k8s.io
2022/01/07 08:33:15 [INFO] Applying CRD machinedeployments.cluster.x-k8s.io
2022/01/07 08:33:15 [INFO] Applying CRD machinehealthchecks.cluster.x-k8s.io
2022/01/07 08:33:15 [INFO] Applying CRD machines.cluster.x-k8s.io
2022/01/07 08:33:16 [INFO] Applying CRD machinesets.cluster.x-k8s.io
2022/01/07 08:33:16 [INFO] Waiting for CRD machinesets.cluster.x-k8s.io to become available
2022/01/07 08:33:16 [INFO] Done waiting for CRD machinesets.cluster.x-k8s.io to become available
2022/01/07 08:33:16 [INFO] Creating CRD authconfigs.management.cattle.io
2022/01/07 08:33:16 [INFO] Creating CRD groupmembers.management.cattle.io
2022/01/07 08:33:16 [INFO] Creating CRD groups.management.cattle.io
2022/01/07 08:33:16 [INFO] Creating CRD tokens.management.cattle.io
2022/01/07 08:33:16 [INFO] Creating CRD userattributes.management.cattle.io
2022/01/07 08:33:16 [INFO] Creating CRD users.management.cattle.io
2022/01/07 08:33:16 [INFO] Waiting for CRD users.management.cattle.io to become available
2022/01/07 08:33:17 [INFO] Done waiting for CRD users.management.cattle.io to become available
2022/01/07 08:33:17 [INFO] Waiting for CRD tokens.management.cattle.io to become available
2022/01/07 08:33:17 [INFO] Done waiting for CRD tokens.management.cattle.io to become available
2022/01/07 08:33:17 [INFO] Waiting for CRD userattributes.management.cattle.io to become available
2022/01/07 08:33:18 [INFO] Done waiting for CRD userattributes.management.cattle.io to become available
2022/01/07 08:33:19 [INFO] Creating CRD clusterroletemplatebindings.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD dynamicschemas.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD catalogs.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD etcdbackups.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD catalogtemplates.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD apps.project.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD apprevisions.project.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD catalogtemplateversions.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD globalrolebindings.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD globalroles.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD pipelineexecutions.project.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD clusteralerts.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD clusteralertgroups.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD pipelinesettings.project.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD kontainerdrivers.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD nodedrivers.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD clustercatalogs.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD pipelines.project.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD clusterloggings.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD nodepools.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD sourcecodecredentials.project.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD clusteralertrules.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD clustermonitorgraphs.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD sourcecodeproviderconfigs.project.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD nodetemplates.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD clusterscans.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD nodes.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD sourcecoderepositories.project.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD composeconfigs.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD podsecuritypolicytemplateprojectbindings.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD multiclusterapps.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD podsecuritypolicytemplates.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD multiclusterapprevisions.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD projectnetworkpolicies.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD monitormetrics.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD projectroletemplatebindings.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD notifiers.management.cattle.io
2022/01/07 08:33:19 [INFO] Waiting for CRD sourcecodeproviderconfigs.project.cattle.io to become available
2022/01/07 08:33:19 [INFO] Creating CRD projectalerts.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD projects.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD projectalertgroups.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD rkek8ssystemimages.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD projectcatalogs.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD projectloggings.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD rkek8sserviceoptions.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD projectalertrules.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD rkeaddons.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD projectmonitorgraphs.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD roletemplates.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD cisconfigs.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD samltokens.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD clustertemplates.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD cisbenchmarkversions.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD templates.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD clustertemplaterevisions.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD templateversions.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD templatecontents.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD globaldnses.management.cattle.io
2022/01/07 08:33:19 [INFO] Creating CRD globaldnsproviders.management.cattle.io
2022/01/07 08:33:19 [INFO] Waiting for CRD nodetemplates.management.cattle.io to become available
2022/01/07 08:33:19 [INFO] Waiting for CRD multiclusterapprevisions.management.cattle.io to become available
2022/01/07 08:33:19 [INFO] Done waiting for CRD sourcecodeproviderconfigs.project.cattle.io to become available
2022/01/07 08:33:19 [INFO] Waiting for CRD sourcecoderepositories.project.cattle.io to become available
2022/01/07 08:33:20 [INFO] Done waiting for CRD nodetemplates.management.cattle.io to become available
2022/01/07 08:33:20 [INFO] Waiting for CRD rkek8sserviceoptions.management.cattle.io to become available
2022/01/07 08:33:20 [INFO] Done waiting for CRD multiclusterapprevisions.management.cattle.io to become available
2022/01/07 08:33:20 [INFO] Waiting for CRD notifiers.management.cattle.io to become available
2022/01/07 08:33:20 [INFO] Done waiting for CRD sourcecoderepositories.project.cattle.io to become available
2022/01/07 08:33:20 [INFO] Waiting for CRD pipelineexecutions.project.cattle.io to become available
2022/01/07 08:33:20 [INFO] Done waiting for CRD rkek8sserviceoptions.management.cattle.io to become available
2022/01/07 08:33:20 [INFO] Waiting for CRD clustertemplaterevisions.management.cattle.io to become available
2022/01/07 08:33:20 [INFO] Done waiting for CRD notifiers.management.cattle.io to become available
2022/01/07 08:33:20 [INFO] Waiting for CRD projectloggings.management.cattle.io to become available
2022/01/07 08:33:21 [INFO] Done waiting for CRD pipelineexecutions.project.cattle.io to become available
2022/01/07 08:33:21 [INFO] Waiting for CRD pipelinesettings.project.cattle.io to become available
2022/01/07 08:33:21 [INFO] Done waiting for CRD projectloggings.management.cattle.io to become available
2022/01/07 08:33:21 [INFO] Waiting for CRD globaldnses.management.cattle.io to become available
2022/01/07 08:33:21 [INFO] Done waiting for CRD pipelinesettings.project.cattle.io to become available
2022/01/07 08:33:21 [INFO] Waiting for CRD pipelines.project.cattle.io to become available
2022/01/07 08:33:21 [INFO] Waiting for CRD clustertemplaterevisions.management.cattle.io to become available
2022/01/07 08:33:22 [INFO] Done waiting for CRD pipelines.project.cattle.io to become available
2022/01/07 08:33:22 [INFO] Waiting for CRD sourcecodecredentials.project.cattle.io to become available
2022/01/07 08:33:22 [INFO] Waiting for CRD clustertemplaterevisions.management.cattle.io to become available
2022/01/07 08:33:22 [INFO] Done waiting for CRD clustertemplaterevisions.management.cattle.io to become available
2022/01/07 08:33:22 [INFO] Waiting for CRD podsecuritypolicytemplates.management.cattle.io to become available
2022/01/07 08:33:22 [INFO] Waiting for CRD globaldnses.management.cattle.io to become available
2022/01/07 08:33:22 [INFO] Done waiting for CRD globaldnses.management.cattle.io to become available
2022/01/07 08:33:22 [INFO] Waiting for CRD multiclusterapps.management.cattle.io to become available
2022/01/07 08:33:22 [INFO] Done waiting for CRD sourcecodecredentials.project.cattle.io to become available
2022/01/07 08:33:22 [INFO] Done waiting for CRD podsecuritypolicytemplates.management.cattle.io to become available
2022/01/07 08:33:22 [INFO] Waiting for CRD rkek8ssystemimages.management.cattle.io to become available
2022/01/07 08:33:22 [INFO] Done waiting for CRD multiclusterapps.management.cattle.io to become available
2022/01/07 08:33:22 [INFO] Waiting for CRD projectcatalogs.management.cattle.io to become available
2022/01/07 08:33:23 [INFO] Done waiting for CRD rkek8ssystemimages.management.cattle.io to become available
2022/01/07 08:33:23 [INFO] Waiting for CRD nodepools.management.cattle.io to become available
2022/01/07 08:33:23 [INFO] Done waiting for CRD projectcatalogs.management.cattle.io to become available
2022/01/07 08:33:23 [INFO] Waiting for CRD clusterloggings.management.cattle.io to become available
2022/01/07 08:33:23 [INFO] Done waiting for CRD nodepools.management.cattle.io to become available
2022/01/07 08:33:23 [INFO] Waiting for CRD podsecuritypolicytemplateprojectbindings.management.cattle.io to become available
2022/01/07 08:33:23 [INFO] Done waiting for CRD clusterloggings.management.cattle.io to become available
2022/01/07 08:33:23 [INFO] Waiting for CRD projectalerts.management.cattle.io to become available
2022/01/07 08:33:24 [INFO] Done waiting for CRD podsecuritypolicytemplateprojectbindings.management.cattle.io to become available
2022/01/07 08:33:24 [INFO] Waiting for CRD projectnetworkpolicies.management.cattle.io to become available
2022/01/07 08:33:24 [INFO] Done waiting for CRD projectalerts.management.cattle.io to become available
2022/01/07 08:33:24 [INFO] Waiting for CRD projectmonitorgraphs.management.cattle.io to become available
2022/01/07 08:33:24 [INFO] Done waiting for CRD projectnetworkpolicies.management.cattle.io to become available
2022/01/07 08:33:24 [INFO] Waiting for CRD rkeaddons.management.cattle.io to become available
2022/01/07 08:33:24 [INFO] Done waiting for CRD projectmonitorgraphs.management.cattle.io to become available
2022/01/07 08:33:24 [INFO] Waiting for CRD templates.management.cattle.io to become available
2022/01/07 08:33:25 [INFO] Done waiting for CRD rkeaddons.management.cattle.io to become available
2022/01/07 08:33:25 [INFO] Waiting for CRD samltokens.management.cattle.io to become available
2022/01/07 08:33:25 [INFO] Done waiting for CRD templates.management.cattle.io to become available
2022/01/07 08:33:25 [INFO] Waiting for CRD templatecontents.management.cattle.io to become available
2022/01/07 08:33:25 [INFO] Done waiting for CRD samltokens.management.cattle.io to become available
2022/01/07 08:33:25 [INFO] Waiting for CRD nodes.management.cattle.io to become available
2022/01/07 08:33:25 [INFO] Done waiting for CRD templatecontents.management.cattle.io to become available
2022/01/07 08:33:25 [INFO] Waiting for CRD globaldnsproviders.management.cattle.io to become available
2022/01/07 08:33:26 [INFO] Done waiting for CRD nodes.management.cattle.io to become available
2022/01/07 08:33:26 [INFO] Waiting for CRD projects.management.cattle.io to become available
2022/01/07 08:33:26 [INFO] Done waiting for CRD globaldnsproviders.management.cattle.io to become available
2022/01/07 08:33:26 [INFO] Waiting for CRD clusteralertrules.management.cattle.io to become available
2022/01/07 08:33:26 [INFO] Done waiting for CRD projects.management.cattle.io to become available
2022/01/07 08:33:26 [INFO] Waiting for CRD nodedrivers.management.cattle.io to become available
2022/01/07 08:33:26 [INFO] Done waiting for CRD clusteralertrules.management.cattle.io to become available
2022/01/07 08:33:26 [INFO] Waiting for CRD clusterscans.management.cattle.io to become available
2022/01/07 08:33:27 [INFO] Done waiting for CRD nodedrivers.management.cattle.io to become available
2022/01/07 08:33:27 [INFO] Waiting for CRD projectroletemplatebindings.management.cattle.io to become available
2022/01/07 08:33:27 [INFO] Done waiting for CRD clusterscans.management.cattle.io to become available
2022/01/07 08:33:27 [INFO] Waiting for CRD cisbenchmarkversions.management.cattle.io to become available
2022/01/07 08:33:27 [INFO] Done waiting for CRD projectroletemplatebindings.management.cattle.io to become available
2022/01/07 08:33:27 [INFO] Waiting for CRD roletemplates.management.cattle.io to become available
2022/01/07 08:33:28 [INFO] Done waiting for CRD cisbenchmarkversions.management.cattle.io to become available
2022/01/07 08:33:28 [INFO] Waiting for CRD templateversions.management.cattle.io to become available
2022/01/07 08:33:28 [INFO] Done waiting for CRD roletemplates.management.cattle.io to become available
2022/01/07 08:33:28 [INFO] Waiting for CRD clustertemplates.management.cattle.io to become available
2022/01/07 08:33:28 [INFO] Done waiting for CRD templateversions.management.cattle.io to become available
2022/01/07 08:33:28 [INFO] Waiting for CRD projectalertgroups.management.cattle.io to become available
2022/01/07 08:33:28 [INFO] Done waiting for CRD clustertemplates.management.cattle.io to become available
2022/01/07 08:33:29 [INFO] Done waiting for CRD projectalertgroups.management.cattle.io to become available
2022/01/07 08:33:29 [INFO] Waiting for CRD projectalertrules.management.cattle.io to become available
2022/01/07 08:33:29 [INFO] Done waiting for CRD projectalertrules.management.cattle.io to become available
2022/01/07 08:33:29 [INFO] Waiting for CRD cisconfigs.management.cattle.io to become available
2022/01/07 08:33:30 [INFO] Done waiting for CRD cisconfigs.management.cattle.io to become available
2022/01/07 08:33:30 [INFO] Waiting for CRD clustermonitorgraphs.management.cattle.io to become available
2022/01/07 08:33:30 [INFO] Done waiting for CRD clustermonitorgraphs.management.cattle.io to become available
2022/01/07 08:33:30 [INFO] Waiting for CRD composeconfigs.management.cattle.io to become available
2022/01/07 08:33:31 [INFO] Done waiting for CRD composeconfigs.management.cattle.io to become available
2022/01/07 08:33:31 [INFO] Waiting for CRD monitormetrics.management.cattle.io to become available
2022/01/07 08:33:31 [INFO] Done waiting for CRD monitormetrics.management.cattle.io to become available
2022/01/07 08:33:38 [INFO] Catalog-cache removed 6 entries from disk
2022/01/07 08:33:44 [INFO] Starting API controllers
2022/01/07 08:33:44 [INFO] Starting project.cattle.io/v3, Kind=App controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding controller
2022/01/07 08:33:44 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=DynamicSchema controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplate controller
2022/01/07 08:33:44 [INFO] Starting apiextensions.k8s.io/v1, Kind=CustomResourceDefinition controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2022/01/07 08:33:44 [INFO] Starting provisioning.cattle.io/v1, Kind=Cluster controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=Preference controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=Catalog controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterAppRevision controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterApp controller
2022/01/07 08:33:44 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeCredential controller
2022/01/07 08:33:44 [INFO] Starting cluster.x-k8s.io/v1alpha4, Kind=Machine controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=ProjectLogging controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=ClusterRegistrationToken controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sServiceOption controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sSystemImage controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller
2022/01/07 08:33:44 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeRepository controller
2022/01/07 08:33:44 [INFO] Starting /v1, Kind=Secret controller
2022/01/07 08:33:44 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=NodeDriver controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=GlobalRole controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=AuthConfig controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=ClusterCatalog controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=CisConfig controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=RkeAddon controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplate controller
2022/01/07 08:33:44 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrap controller
2022/01/07 08:33:44 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=User controller
2022/01/07 08:33:44 [INFO] Starting project.cattle.io/v3, Kind=Pipeline controller
2022/01/07 08:33:44 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2022/01/07 08:33:44 [INFO] Starting catalog.cattle.io/v1, Kind=ClusterRepo controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
2022/01/07 08:33:44 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplate controller
2022/01/07 08:33:44 [INFO] Starting /v1, Kind=Namespace controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=CisBenchmarkVersion controller
2022/01/07 08:33:44 [INFO] Starting project.cattle.io/v3, Kind=PipelineExecution controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=Feature controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=APIService controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
2022/01/07 08:33:44 [INFO] Starting /v1, Kind=ConfigMap controller
2022/01/07 08:33:44 [INFO] Starting /v1, Kind=ServiceAccount controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=NodeTemplate controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=GlobalDns controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplateVersion controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=Project controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=KontainerDriver controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=Setting controller
2022/01/07 08:33:44 [INFO] Reconciling GlobalRoles
2022/01/07 08:33:44 [INFO] Creating clustertemplates-create
2022/01/07 08:33:44 [INFO] Creating clustertemplaterevisions-create
2022/01/07 08:33:44 [INFO] Creating view-rancher-metrics
2022/01/07 08:33:44 [INFO] Creating user
2022/01/07 08:33:44 [INFO] Creating user-base
2022/01/07 08:33:44 [INFO] Creating clusters-create
2022/01/07 08:33:44 [INFO] Creating kontainerdrivers-manage
2022/01/07 08:33:44 [INFO] Creating podsecuritypolicytemplates-manage
2022/01/07 08:33:44 [INFO] Creating nodedrivers-manage
2022/01/07 08:33:44 [INFO] Creating authn-manage
2022/01/07 08:33:44 [INFO] Creating admin
2022/01/07 08:33:44 [INFO] Creating catalogs-use
2022/01/07 08:33:44 [INFO] Creating users-manage
2022/01/07 08:33:44 [INFO] Creating settings-manage
2022/01/07 08:33:44 [INFO] Creating restricted-admin
2022/01/07 08:33:44 [INFO] Creating catalogs-manage
2022/01/07 08:33:44 [INFO] Creating roles-manage
2022/01/07 08:33:44 [INFO] 
2022/01/07 08:33:44 [INFO] -----------------------------------------
2022/01/07 08:33:44 [INFO] Welcome to Rancher
2022/01/07 08:33:44 [INFO] A bootstrap password has been generated for your admin user.
2022/01/07 08:33:44 [INFO] 
2022/01/07 08:33:44 [INFO] Bootstrap Password: w22s2264fptk6tctsc8p4rqfs9mx62zpdgvclwm54hpgh7gw9vqjpz
2022/01/07 08:33:44 [INFO] 
2022/01/07 08:33:44 [INFO] Use https://172.17.1.2/dashboard/?setup=w22s2264fptk6tctsc8p4rqfs9mx62zpdgvclwm54hpgh7gw9vqjpz to complete setup in the UI
2022/01/07 08:33:44 [INFO] -----------------------------------------
2022/01/07 08:33:44 [INFO] 
2022/01/07 08:33:44 [INFO] Creating features-manage
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=User controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2022/01/07 08:33:44 [INFO] Starting /v1, Kind=Secret controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2022/01/07 08:33:44 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2022/01/07 08:33:44 [INFO] Waiting for initial data to be populated
2022/01/07 08:33:44 [INFO] Reconciling RoleTemplates
2022/01/07 08:33:44 [INFO] Creating nodes-view
2022/01/07 08:33:44 [INFO] Created default admin user and binding
2022/01/07 08:33:44 [INFO] Creating clustercatalogs-view
2022/01/07 08:33:44 [INFO] Creating read-only
2022/01/07 08:33:44 [INFO] Creating configmaps-manage
2022/01/07 08:33:44 [INFO] Creating persistentvolumeclaims-view
2022/01/07 08:33:44 [INFO] Creating project-monitoring-readonly
2022/01/07 08:33:44 [INFO] Creating edit
2022/01/07 08:33:44 [INFO] Creating view
2022/01/07 08:33:44 [INFO] Creating storage-manage
2022/01/07 08:33:44 [INFO] Creating clustercatalogs-manage
2022/01/07 08:33:44 [INFO] Creating services-view
2022/01/07 08:33:44 [INFO] Steve auth startup complete
2022/01/07 08:33:44 [INFO] Creating persistentvolumeclaims-manage
2022/01/07 08:33:44 [INFO] Creating project-owner
2022/01/07 08:33:44 [INFO] Creating project-member
2022/01/07 08:33:44 [INFO] Creating monitoring-ui-view
2022/01/07 08:33:44 [INFO] Creating cluster-member
2022/01/07 08:33:44 [INFO] Creating clusterroletemplatebindings-manage
2022/01/07 08:33:44 [INFO] Creating navlinks-manage
2022/01/07 08:33:44 [INFO] Creating ingress-manage
2022/01/07 08:33:44 [INFO] Creating projectroletemplatebindings-view
2022/01/07 08:33:44 [INFO] Creating projects-create
2022/01/07 08:33:44 [INFO] Creating create-ns
2022/01/07 08:33:44 [INFO] Creating workloads-view
2022/01/07 08:33:44 [INFO] Creating services-manage
2022/01/07 08:33:44 [INFO] Creating secrets-view
2022/01/07 08:33:44 [INFO] Registering namespaceHandler for adding labels 
2022/01/07 08:33:44 [INFO] Creating projectcatalogs-view
2022/01/07 08:33:44 [INFO] Creating workloads-manage
2022/01/07 08:33:44 [INFO] Creating projectroletemplatebindings-manage
2022/01/07 08:33:44 [INFO] Creating cluster-admin
2022/01/07 08:33:44 [INFO] Creating admin
2022/01/07 08:33:44 [INFO] Creating nodes-manage
2022/01/07 08:33:44 [INFO] Creating ingress-view
2022/01/07 08:33:45 [INFO] Creating CRD prometheuses.monitoring.coreos.com
2022/01/07 08:33:45 [INFO] Creating configmaps-view
2022/01/07 08:33:45 [INFO] Creating serviceaccounts-manage
2022/01/07 08:33:45 [INFO] Creating CRD prometheusrules.monitoring.coreos.com
2022/01/07 08:33:45 [INFO] Creating CRD alertmanagers.monitoring.coreos.com
2022/01/07 08:33:45 [INFO] Creating serviceaccounts-view
2022/01/07 08:33:45 [INFO] Creating projectcatalogs-manage
2022/01/07 08:33:45 [INFO] Creating navlinks-view
2022/01/07 08:33:45 [INFO] Creating cluster-owner
2022/01/07 08:33:45 [INFO] Creating CRD servicemonitors.monitoring.coreos.com
2022/01/07 08:33:45 [INFO] Creating projects-view
2022/01/07 08:33:45 [INFO] Creating clusterroletemplatebindings-view
2022/01/07 08:33:45 [INFO] Creating backups-manage
2022/01/07 08:33:45 [INFO] Creating secrets-manage
2022/01/07 08:33:45 [INFO] Waiting for CRD alertmanagers.monitoring.coreos.com to become available
2022/01/07 08:33:45 [INFO] Done waiting for CRD alertmanagers.monitoring.coreos.com to become available
2022/01/07 08:33:45 [INFO] Waiting for CRD servicemonitors.monitoring.coreos.com to become available
2022/01/07 08:33:46 [INFO] Waiting for initial data to be populated
2022/01/07 08:33:46 [INFO] Done waiting for CRD servicemonitors.monitoring.coreos.com to become available
2022/01/07 08:33:46 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2022/01/07 08:33:47 [INFO] Detected v2.6.0 upgrade, forcing logout for all web users
2022/01/07 08:33:47 [INFO] Starting /v1, Kind=Node controller
2022/01/07 08:33:47 [INFO] Starting monitoring.coreos.com/v1, Kind=Prometheus controller
2022/01/07 08:33:47 [INFO] Starting rke.cattle.io/v1, Kind=CustomMachine controller
2022/01/07 08:33:47 [INFO] Starting admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration controller
2022/01/07 08:33:47 [INFO] Starting admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration controller
2022/01/07 08:33:47 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrapTemplate controller
2022/01/07 08:33:47 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [INFO] Starting apps/v1, Kind=StatefulSet controller
2022/01/07 08:33:47 [INFO] Starting management.cattle.io/v3, Kind=ManagedChart controller
2022/01/07 08:33:47 [INFO] Starting cluster.x-k8s.io/v1alpha4, Kind=Cluster controller
2022/01/07 08:33:47 [INFO] Starting fleet.cattle.io/v1alpha1, Kind=Bundle controller
2022/01/07 08:33:47 [INFO] Starting networking.k8s.io/v1, Kind=Ingress controller
2022/01/07 08:33:47 [INFO] Starting rke.cattle.io/v1, Kind=RKEControlPlane controller
2022/01/07 08:33:47 [INFO] Starting fleet.cattle.io/v1alpha1, Kind=Cluster controller
2022/01/07 08:33:47 [INFO] Starting apps/v1, Kind=Deployment controller
2022/01/07 08:33:47 [INFO] Starting catalog.cattle.io/v1, Kind=App controller
2022/01/07 08:33:47 [INFO] Starting cluster.x-k8s.io/v1alpha4, Kind=MachineDeployment controller
2022/01/07 08:33:47 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [INFO] Starting batch/v1beta1, Kind=CronJob controller
2022/01/07 08:33:47 [INFO] Starting /v1, Kind=Endpoints controller
2022/01/07 08:33:47 [INFO] Starting monitoring.coreos.com/v1, Kind=ServiceMonitor controller
2022/01/07 08:33:47 [INFO] Starting apps/v1, Kind=DaemonSet controller
2022/01/07 08:33:47 [INFO] Starting catalog.cattle.io/v1, Kind=Operation controller
2022/01/07 08:33:47 [INFO] Starting /v1, Kind=Service controller
2022/01/07 08:33:47 [INFO] Starting management.cattle.io/v3, Kind=FleetWorkspace controller
2022/01/07 08:33:47 [INFO] Starting apps/v1, Kind=ReplicaSet controller
2022/01/07 08:33:47 [INFO] Starting /v1, Kind=Pod controller
2022/01/07 08:33:47 [INFO] Starting /v1, Kind=ReplicationController controller
2022/01/07 08:33:47 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [INFO] Starting rke.cattle.io/v1, Kind=RKECluster controller
2022/01/07 08:33:47 [INFO] Starting batch/v1, Kind=Job controller
2022/01/07 08:33:47 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:47 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:48 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/01/07 08:33:48 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:48 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:48 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/01/07 08:33:48 [INFO] Waiting for initial data to be populated
2022/01/07 08:33:49 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/01/07 08:33:49 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:49 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:49 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/01/07 08:33:50 [INFO] adding kontainer driver rancherKubernetesEngine
2022/01/07 08:33:50 [INFO] adding kontainer driver googleKubernetesEngine
2022/01/07 08:33:50 [INFO] adding kontainer driver azureKubernetesService
2022/01/07 08:33:50 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/01/07 08:33:50 [INFO] adding kontainer driver amazonElasticContainerService
2022/01/07 08:33:50 [INFO] adding kontainer driver baiducloudcontainerengine
2022/01/07 08:33:50 [INFO] adding kontainer driver aliyunkubernetescontainerservice
2022/01/07 08:33:50 [INFO] adding kontainer driver tencentkubernetesengine
2022/01/07 08:33:50 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/01/07 08:33:50 [INFO] adding kontainer driver huaweicontainercloudengine
2022/01/07 08:33:50 [INFO] adding kontainer driver oraclecontainerengine
2022/01/07 08:33:50 [INFO] adding kontainer driver linodekubernetesengine
2022/01/07 08:33:50 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/01/07 08:33:50 [INFO] adding kontainer driver opentelekomcloudcontainerengine
2022/01/07 08:33:50 [INFO] Created cattle-global-data namespace
2022/01/07 08:33:50 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/01/07 08:33:51 [INFO] Created cattle-global-nt namespace
2022/01/07 08:33:51 [INFO] Creating node driver pinganyunecs
2022/01/07 08:33:51 [INFO] Listening on :443
2022/01/07 08:33:51 [INFO] Listening on :80
2022/01/07 08:33:51 [INFO] Creating node driver aliyunecs
2022/01/07 08:33:51 [INFO] Creating node driver amazonec2
2022/01/07 08:33:51 [ERROR] error syncing 'server-url': handler apiservice-settings: secrets "tls-rancher-internal-ca" not found, requeuing
2022/01/07 08:33:51 [INFO] Creating node driver azure
2022/01/07 08:33:51 [INFO] Creating node driver cloudca
2022/01/07 08:33:51 [INFO] Listening on :444
2022/01/07 08:33:51 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2022-01-07 08:33:51 +0000 UTC notAfter=2023-01-07 08:33:51 +0000 UTC
2022/01/07 08:33:51 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/01/07 08:33:51 [INFO] Creating node driver cloudscale
2022/01/07 08:33:51 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2022-01-07 08:33:51 +0000 UTC notAfter=2023-01-07 08:33:51 +0000 UTC
2022/01/07 08:33:51 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca,O=dynamiclistener-org: notBefore=2022-01-07 08:33:51 +0000 UTC notAfter=2023-01-07 08:33:51 +0000 UTC
2022/01/07 08:33:51 [INFO] Creating new TLS secret for tls-rancher-internal (count: 1): map[listener.cattle.io/cn-10.43.57.81:10.43.57.81 listener.cattle.io/fingerprint:SHA1=70DB7526EE13DFFC95E4283F33C9ED9BEF525DCA]
2022/01/07 08:33:51 [ERROR] Failed to handling tunnel request from 192.168.35.72:44116: response 401: failed authentication
2022/01/07 08:33:51 [ERROR] Failed to handling tunnel request from 192.168.35.72:44114: response 401: failed authentication
2022/01/07 08:33:51 [INFO] Active TLS secret tls-rancher-internal (ver=1098) (count 1): map[listener.cattle.io/cn-10.43.57.81:10.43.57.81 listener.cattle.io/fingerprint:SHA1=70DB7526EE13DFFC95E4283F33C9ED9BEF525DCA]
2022/01/07 08:33:51 [INFO] Creating node driver digitalocean
2022/01/07 08:33:51 [INFO] Creating node driver exoscale
2022/01/07 08:33:51 [INFO] Creating node driver google
2022/01/07 08:33:51 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2022/01/07 08:33:52 [INFO] Starting /v1, Kind=Secret controller
2022/01/07 08:33:52 [INFO] Creating node driver harvester
2022/01/07 08:33:52 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/01/07 08:33:52 [INFO] Creating node driver linode
2022/01/07 08:33:52 [INFO] Creating node driver oci
2022/01/07 08:33:52 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:52 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:52 [INFO] Creating node driver openstack
2022/01/07 08:33:52 [INFO] Creating node driver otc
2022/01/07 08:33:52 [INFO] Creating node driver packet
2022/01/07 08:33:53 [INFO] Creating node driver pnap
2022/01/07 08:33:53 [ERROR] error syncing 'fleet-local/local': handler cluster-create: server url is missing, can't generate kubeconfig for fleet import cluster, requeuing
2022/01/07 08:33:53 [INFO] Creating node driver rackspace
2022/01/07 08:33:53 [INFO] Creating node driver softlayer
2022/01/07 08:33:53 [INFO] Creating node driver vmwarevsphere
2022/01/07 08:33:54 [INFO] Starting catalog controller
2022/01/07 08:33:54 [INFO] Starting project-level catalog controller
2022/01/07 08:33:54 [INFO] Starting cluster-level catalog controller
2022/01/07 08:33:54 [ERROR] error syncing 'fleet-local/local': handler workspace-backport-cluster: fleetworkspaces.management.cattle.io "fleet-local" not found, requeuing
2022/01/07 08:33:54 [ERROR] Failed to handling tunnel request from 192.168.35.72:44118: response 400: cluster not found
2022/01/07 08:33:54 [ERROR] Failed to handling tunnel request from 192.168.35.72:44120: response 400: cluster not found
2022/01/07 08:33:54 [ERROR] Failed to handling tunnel request from 192.168.35.72:44122: response 400: cluster not found
2022/01/07 08:33:54 [ERROR] Failed to handling tunnel request from 192.168.35.72:44124: response 400: cluster not found
2022/01/07 08:33:55 [ERROR] Failed to handling tunnel request from 192.168.35.72:44128: response 400: cluster not found
2022/01/07 08:33:55 [ERROR] Failed to handling tunnel request from 192.168.35.72:44126: response 400: cluster not found
2022/01/07 08:33:55 [ERROR] Failed to handling tunnel request from 192.168.35.72:44130: response 400: cluster not found
2022/01/07 08:33:55 [ERROR] Failed to handling tunnel request from 192.168.35.72:44132: response 400: cluster not found
2022/01/07 08:33:55 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertRule controller
2022/01/07 08:33:55 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=APIService
2022/01/07 08:33:55 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKECluster
2022/01/07 08:33:55 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=Event
2022/01/07 08:33:55 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding
2022/01/07 08:33:55 [INFO] Watching metadata for k3s.cattle.io/v1, Kind=Addon
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=Namespace
2022/01/07 08:33:55 [INFO] Watching metadata for events.k8s.io/v1, Kind=Event
2022/01/07 08:33:55 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph
2022/01/07 08:33:55 [INFO] Watching metadata for batch/v1, Kind=Job
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=ConfigMap
2022/01/07 08:33:55 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSIDriver
2022/01/07 08:33:55 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=PodTemplate
2022/01/07 08:33:55 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration
2022/01/07 08:33:55 [INFO] Watching metadata for rke.cattle.io/v1, Kind=CustomMachine
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan
2022/01/07 08:33:55 [INFO] Watching metadata for helm.cattle.io/v1, Kind=HelmChartConfig
2022/01/07 08:33:55 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEBootstrapTemplate
2022/01/07 08:33:55 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=LimitRange
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=User
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting
2022/01/07 08:33:55 [INFO] Starting management.cattle.io/v3, Kind=ClusterScan controller
2022/01/07 08:33:55 [INFO] Starting management.cattle.io/v3, Kind=EtcdBackup controller
2022/01/07 08:33:55 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2022/01/07 08:33:55 [INFO] Refreshing driverMetadata in 1440 minutes
2022/01/07 08:33:55 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-clustertemplaterevisions-create for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember
2022/01/07 08:33:55 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha4, Kind=Cluster
2022/01/07 08:33:55 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=ResourceQuota
2022/01/07 08:33:55 [INFO] Watching metadata for apps/v1, Kind=DaemonSet
2022/01/07 08:33:55 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule
2022/01/07 08:33:55 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha4, Kind=MachineDeployment
2022/01/07 08:33:55 [INFO] Watching metadata for storage.k8s.io/v1beta1, Kind=CSIStorageCapacity
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
2022/01/07 08:33:55 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=FleetWorkspace
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=ServiceAccount
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=Pod
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog
2022/01/07 08:33:55 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha4, Kind=Machine
2022/01/07 08:33:55 [INFO] Watching metadata for project.cattle.io/v3, Kind=App
2022/01/07 08:33:55 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
2022/01/07 08:33:55 [INFO] Watching metadata for batch/v1, Kind=CronJob
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision
2022/01/07 08:33:55 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Cluster
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=PersistentVolume
2022/01/07 08:33:55 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha4, Kind=MachineHealthCheck
2022/01/07 08:33:55 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=App
2022/01/07 08:33:55 [INFO] Watching metadata for provisioning.cattle.io/v1, Kind=Cluster
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema
2022/01/07 08:33:55 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group
2022/01/07 08:33:55 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution
2022/01/07 08:33:55 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha4, Kind=MachineSet
2022/01/07 08:33:55 [INFO] Watching metadata for apps/v1, Kind=Deployment
2022/01/07 08:33:55 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding
2022/01/07 08:33:55 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
2022/01/07 08:33:55 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule
2022/01/07 08:33:55 [INFO] Watching metadata for flowcontrol.apiserver.k8s.io/v1beta1, Kind=PriorityLevelConfiguration
2022/01/07 08:33:55 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=ClusterRepo
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging
2022/01/07 08:33:55 [INFO] Watching metadata for networking.k8s.io/v1, Kind=Ingress
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken
2022/01/07 08:33:55 [INFO] Watching metadata for flowcontrol.apiserver.k8s.io/v1beta1, Kind=FlowSchema
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=Endpoints
2022/01/07 08:33:55 [INFO] Watching metadata for node.k8s.io/v1, Kind=RuntimeClass
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup
2022/01/07 08:33:55 [INFO] Watching metadata for networking.k8s.io/v1, Kind=IngressClass
2022/01/07 08:33:55 [INFO] Watching metadata for ui.cattle.io/v1, Kind=NavLink
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog
2022/01/07 08:33:55 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEBootstrap
2022/01/07 08:33:55 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=ReplicationController
2022/01/07 08:33:55 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential
2022/01/07 08:33:55 [INFO] Watching metadata for policy/v1, Kind=PodDisruptionBudget
2022/01/07 08:33:55 [INFO] Watching metadata for autoscaling/v2beta2, Kind=HorizontalPodAutoscaler
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier
2022/01/07 08:33:55 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig
2022/01/07 08:33:55 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass
2022/01/07 08:33:55 [INFO] Watching metadata for apps/v1, Kind=StatefulSet
2022/01/07 08:33:55 [INFO] Watching metadata for helm.cattle.io/v1, Kind=HelmChart
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=Service
2022/01/07 08:33:55 [INFO] Watching metadata for discovery.k8s.io/v1, Kind=EndpointSlice
2022/01/07 08:33:55 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy
2022/01/07 08:33:55 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=ManagedChart
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon
2022/01/07 08:33:55 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Bundle
2022/01/07 08:33:55 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline
2022/01/07 08:33:55 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=Operation
2022/01/07 08:33:55 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver
2022/01/07 08:33:55 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=Secret
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp
2022/01/07 08:33:55 [INFO] Watching metadata for /v1, Kind=Node
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature
2022/01/07 08:33:55 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole
2022/01/07 08:33:55 [INFO] Watching metadata for certificates.k8s.io/v1, Kind=CertificateSigningRequest
2022/01/07 08:33:55 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision
2022/01/07 08:33:55 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEControlPlane
2022/01/07 08:33:55 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion
2022/01/07 08:33:55 [INFO] Starting management.cattle.io/v3, Kind=ComposeConfig controller
2022/01/07 08:33:55 [INFO] Starting management.cattle.io/v3, Kind=GlobalDnsProvider controller
2022/01/07 08:33:55 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertGroup controller
2022/01/07 08:33:55 [INFO] [mgmt-auth-crtb-controller] Creating role/clusterRole local-clusterowner
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-settings-manage for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-clusters-create for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-nodedrivers-manage for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-features-manage for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-user-base for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-podsecuritypolicytemplates-manage for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-user for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-clustertemplates-create for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-authn-manage for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-users-manage for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-kontainerdrivers-manage for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-catalogs-use for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-restricted-admin for corresponding GlobalRole
2022/01/07 08:33:55 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-roles-manage for corresponding GlobalRole
2022/01/07 08:33:56 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-view-rancher-metrics for corresponding GlobalRole
2022/01/07 08:33:56 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json
2022/01/07 08:33:56 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
2022/01/07 08:33:56 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding globalrolebinding-qrr5c
2022/01/07 08:33:56 [INFO] [mgmt-auth-grb-controller] Creating clusterRoleBinding for globalRoleBinding globalrolebinding-qrr5c for user user-tvj4n with role cattle-globalrole-admin
2022/01/07 08:33:56 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-catalogs-manage for corresponding GlobalRole
2022/01/07 08:33:56 [INFO] [mgmt-auth-gr-controller] Creating clusterRole cattle-globalrole-admin for corresponding GlobalRole
2022/01/07 08:33:56 [INFO] create kontainerdriver amazonelasticcontainerservice
2022/01/07 08:33:56 [INFO] create kontainerdriver aliyunkubernetescontainerservice
2022/01/07 08:33:56 [INFO] create kontainerdriver rancherkubernetesengine
2022/01/07 08:33:56 [INFO] create kontainerdriver linodekubernetesengine
2022/01/07 08:33:56 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment
2022/01/07 08:33:56 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck
2022/01/07 08:33:56 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=Machine
2022/01/07 08:33:56 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=Cluster
2022/01/07 08:33:56 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineSet
2022/01/07 08:33:56 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment
2022/01/07 08:33:56 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck
2022/01/07 08:33:56 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=Machine
2022/01/07 08:33:56 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=Cluster
2022/01/07 08:33:56 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=MachineSet
2022/01/07 08:33:56 [INFO] Starting rke.cattle.io/v1, Kind=RKEControlPlane controller
2022/01/07 08:33:56 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrapTemplate controller
2022/01/07 08:33:56 [INFO] Starting rke.cattle.io/v1, Kind=CustomMachine controller
2022/01/07 08:33:56 [INFO] Starting rke.cattle.io/v1, Kind=RKECluster controller
2022/01/07 08:33:56 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrap controller
2022/01/07 08:33:56 [INFO] Starting provisioning.cattle.io/v1, Kind=Cluster controller
2022/01/07 08:33:56 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment: failed to wait for caches to sync
2022/01/07 08:33:56 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck: failed to wait for caches to sync
2022/01/07 08:33:56 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=Machine: failed to wait for caches to sync
2022/01/07 08:33:56 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=Cluster: failed to wait for caches to sync
2022/01/07 08:33:56 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=MachineSet: failed to wait for caches to sync
2022/01/07 08:33:56 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=Machine
2022/01/07 08:33:56 [ERROR] Failed to handling tunnel request from 192.168.35.72:44136: response 401: failed authentication
2022/01/07 08:33:56 [ERROR] Failed to handling tunnel request from 192.168.35.72:44138: response 401: failed authentication
2022/01/07 08:33:56 [INFO] create kontainerdriver oraclecontainerengine
2022/01/07 08:33:56 [INFO] Rancher startup complete
2022/01/07 08:33:56 [INFO] checking configmap cattle-system/admincreated to determine if orphan bindings cleanup needs to run
2022/01/07 08:33:56 [INFO] checking configmap cattle-system/admincreated to determine if duplicate bindings cleanup needs to run
2022/01/07 08:33:56 [INFO] create kontainerdriver opentelekomcloudcontainerengine
2022/01/07 08:33:56 [INFO] create kontainerdriver tencentkubernetesengine
2022/01/07 08:33:56 [INFO] [mgmt-auth-crtb-controller] Creating clusterRoleBinding for membership in cluster local for subject u-mo773yttt4
2022/01/07 08:33:56 [INFO] create kontainerdriver googlekubernetesengine
2022/01/07 08:33:56 [INFO] create kontainerdriver baiducloudcontainerengine
2022/01/07 08:33:56 [INFO] create kontainerdriver azurekubernetesservice
2022/01/07 08:33:56 [INFO] [mgmt-cluster-rbac-delete] Creating Default project for cluster local
2022/01/07 08:33:56 [INFO] create kontainerdriver huaweicontainercloudengine
2022/01/07 08:33:56 [ERROR] Failed to find system chart fleet will try again in 5 seconds: configmaps "" not found
2022/01/07 08:33:57 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/01/07 08:33:57 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace local
2022/01/07 08:33:57 [INFO] bindings cleanup needed, waiting 3 minutes before starting
2022/01/07 08:33:57 [INFO] [clean-orphan-bindings] cleaning up orphaned bindings
2022/01/07 08:33:57 [INFO] [clean-orphan-bindings] checking for orphaned rolebindings
2022/01/07 08:33:57 [INFO] [mgmt-project-rbac-create] Creating namespace p-nsdv7
2022/01/07 08:33:57 [INFO] create kontainerdriver linodekubernetesengine
2022/01/07 08:33:57 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment
2022/01/07 08:33:57 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=Machine controller
2022/01/07 08:33:57 [INFO] create kontainerdriver oraclecontainerengine
2022/01/07 08:33:57 [INFO] update kontainerdriver aliyunkubernetescontainerservice
2022/01/07 08:33:57 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:57 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:33:57 [INFO] [mgmt-cluster-rbac-delete] Creating System project for cluster local
2022/01/07 08:33:57 [INFO] update kontainerdriver amazonelasticcontainerservice
2022/01/07 08:33:57 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-mo773yttt4 with role cluster-owner in namespace local
2022/01/07 08:33:57 [INFO] update kontainerdriver oraclecontainerengine
2022/01/07 08:33:57 [INFO] update kontainerdriver linodekubernetesengine
2022/01/07 08:33:57 [INFO] create kontainerdriver googlekubernetesengine
2022/01/07 08:33:57 [INFO] create kontainerdriver huaweicontainercloudengine
2022/01/07 08:33:57 [INFO] update kontainerdriver oraclecontainerengine
2022/01/07 08:33:57 [INFO] update kontainerdriver linodekubernetesengine
2022/01/07 08:33:57 [INFO] create kontainerdriver azurekubernetesservice
2022/01/07 08:33:57 [INFO] create kontainerdriver rancherkubernetesengine
2022/01/07 08:33:57 [INFO] create kontainerdriver opentelekomcloudcontainerengine
2022/01/07 08:33:57 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck
2022/01/07 08:33:57 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=Cluster
2022/01/07 08:33:57 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineSet
2022/01/07 08:33:57 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment controller
2022/01/07 08:33:57 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/01/07 08:33:57 [INFO] create kontainerdriver tencentkubernetesengine
2022/01/07 08:33:57 [INFO] [mgmt-project-rbac-create] Creating namespace p-gvjv8
2022/01/07 08:33:57 [INFO] [mgmt-cluster-rbac-delete] Updating cluster local
2022/01/07 08:33:57 [INFO] update kontainerdriver opentelekomcloudcontainerengine
2022/01/07 08:33:57 [INFO] update kontainerdriver googlekubernetesengine
2022/01/07 08:33:57 [INFO] update kontainerdriver opentelekomcloudcontainerengine
2022/01/07 08:33:57 [INFO] update kontainerdriver huaweicontainercloudengine
2022/01/07 08:33:57 [INFO] update kontainerdriver azurekubernetesservice
2022/01/07 08:33:57 [INFO] update kontainerdriver huaweicontainercloudengine
2022/01/07 08:33:57 [INFO] update kontainerdriver baiducloudcontainerengine
2022/01/07 08:33:58 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-nsdv7
2022/01/07 08:33:58 [INFO] update kontainerdriver rancherkubernetesengine
2022/01/07 08:33:58 [INFO] update kontainerdriver rancherkubernetesengine
2022/01/07 08:33:58 [WARNING] Project p-nsdv7 has no creatorId annotation. Cannot add creator as owner
2022/01/07 08:33:58 [INFO] [mgmt-project-rbac-create] Updating project p-nsdv7
2022/01/07 08:33:58 [ERROR] failed to sync cache for cluster.x-k8s.io/v1alpha3, Kind=Cluster
2022/01/07 08:33:58 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck controller
2022/01/07 08:33:58 [ERROR] failed to start controller for cluster.x-k8s.io/v1alpha3, Kind=Cluster: failed to wait for caches to sync
2022/01/07 08:33:58 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=MachineSet controller
2022/01/07 08:33:58 [INFO] update kontainerdriver tencentkubernetesengine
2022/01/07 08:33:58 [INFO] update kontainerdriver tencentkubernetesengine
2022/01/07 08:33:58 [WARNING] Cluster local has no creatorId annotation. Cannot add creator as owner
2022/01/07 08:33:58 [INFO] [mgmt-cluster-rbac-delete] Updating cluster local
2022/01/07 08:33:58 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/01/07 08:33:58 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=Cluster
2022/01/07 08:33:58 [INFO] successfully cleaned up orphan bindings
2022/01/07 08:33:58 [WARNING] Project p-gvjv8 has no creatorId annotation. Cannot add creator as owner
2022/01/07 08:33:58 [INFO] [mgmt-project-rbac-create] Updating project p-gvjv8
2022/01/07 08:33:58 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-mo773yttt4 with role cluster-owner in namespace p-nsdv7
2022/01/07 08:33:58 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=Cluster controller
2022/01/07 08:33:58 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/01/07 08:33:58 [INFO] [mgmt-auth-crtb-controller] Creating role cluster-owner in namespace p-gvjv8
2022/01/07 08:33:59 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-mo773yttt4 with role cluster-owner in namespace p-gvjv8
2022/01/07 08:33:59 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/01/07 08:33:59 [WARNING] Cluster local has no creatorId annotation. Cannot add creator as owner
2022/01/07 08:33:59 [INFO] [mgmt-cluster-rbac-delete] Updating cluster local
2022/01/07 08:33:59 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/01/07 08:34:00 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/01/07 08:34:00 [INFO] Updating global catalog helm3-library
2022/01/07 08:34:00 [INFO] Helm icon error: open management-state/catalog-cache/f341cfdfa521a9aa2b993cb34b26bb91b2d173ef1a7df8d41b8921b0e4f82788/charts/mariadb/mariadb-stack-220x234.png: no such file or directory
2022/01/07 08:34:00 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/01/07 08:34:01 [INFO] uploading azureConfig to nodeconfig schema
2022/01/07 08:34:01 [INFO] uploading azureConfig to nodetemplateconfig schema
2022/01/07 08:34:01 [INFO] uploading vmwarevsphereConfig to nodeconfig schema
2022/01/07 08:34:01 [INFO] uploading vmwarevsphereConfig to nodetemplateconfig schema
2022/01/07 08:34:01 [INFO] uploading linodeConfig to nodeconfig schema
2022/01/07 08:34:01 [INFO] uploading linodeConfig to nodetemplateconfig schema
2022/01/07 08:34:01 [INFO] uploading amazonec2Config to nodeconfig schema
2022/01/07 08:34:01 [ERROR] Failed to handling tunnel request from 192.168.35.72:44150: response 401: failed authentication
2022/01/07 08:34:01 [ERROR] Failed to handling tunnel request from 192.168.35.72:44148: response 401: failed authentication
2022/01/07 08:34:01 [INFO] uploading amazonec2Config to nodetemplateconfig schema
2022/01/07 08:34:01 [INFO] uploading harvesterConfig to nodeconfig schema
2022/01/07 08:34:02 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=AzureConfig
2022/01/07 08:34:02 [INFO] uploading harvesterConfig to nodetemplateconfig schema
2022/01/07 08:34:02 [INFO] uploading digitaloceanConfig to nodeconfig schema
2022/01/07 08:34:02 [INFO] Starting /v1, Kind=ConfigMap controller
2022/01/07 08:34:02 [INFO] uploading digitaloceanConfig to nodetemplateconfig schema
2022/01/07 08:34:02 [ERROR] error syncing 'git-webhook': handler apiservice: failed to create cattle-fleet-system/stv-aggregation /v1, Kind=Secret for apiservice git-webhook: namespaces "cattle-fleet-system" not found, requeuing
2022/01/07 08:34:02 [INFO] uploading azurecredentialConfig to credentialconfig schema
2022/01/07 08:34:02 [INFO] Starting rke-machine-config.cattle.io/v1, Kind=AzureConfig controller
2022/01/07 08:34:02 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=AzureMachine
2022/01/07 08:34:02 [INFO] Starting /v1, Kind=ConfigMap controller
2022/01/07 08:34:02 [INFO] Registering istio for cluster "local"
2022/01/07 08:34:02 [INFO] Starting cluster controllers for local
2022/01/07 08:34:02 [INFO] uploading googlecredentialConfig to credentialconfig schema
2022/01/07 08:34:02 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:38673
2022/01/07 08:34:02 [INFO] kontainerdriver amazonelasticcontainerservice stopped
2022/01/07 08:34:03 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:41659
2022/01/07 08:34:03 [INFO] kontainerdriver azurekubernetesservice stopped
2022/01/07 08:34:03 [INFO] uploading vmwarevspherecredentialConfig to credentialconfig schema
2022/01/07 08:34:03 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:34073
2022/01/07 08:34:03 [INFO] kontainerdriver googlekubernetesengine stopped
2022/01/07 08:34:03 [INFO] Starting management.cattle.io/v3, Kind=ClusterLogging controller
2022/01/07 08:34:03 [INFO] Starting management.cattle.io/v3, Kind=Notifier controller
2022/01/07 08:34:03 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlert controller
2022/01/07 08:34:03 [INFO] Starting management.cattle.io/v3, Kind=SamlToken controller
2022/01/07 08:34:03 [INFO] uploading linodecredentialConfig to credentialconfig schema
2022/01/07 08:34:03 [INFO] Starting rke-machine.cattle.io/v1, Kind=AzureMachine controller
2022/01/07 08:34:03 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=AzureMachineTemplate
2022/01/07 08:34:03 [INFO] uploading amazonec2credentialConfig to credentialconfig schema
2022/01/07 08:34:04 [INFO] uploading harvestercredentialConfig to credentialconfig schema
2022/01/07 08:34:04 [INFO] Starting cluster agent for local [owner=true]
2022/01/07 08:34:04 [INFO] Starting /v1, Kind=ServiceAccount controller
2022/01/07 08:34:04 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2022/01/07 08:34:04 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2022/01/07 08:34:04 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2022/01/07 08:34:04 [INFO] Starting /v1, Kind=ResourceQuota controller
2022/01/07 08:34:04 [INFO] Starting /v1, Kind=LimitRange controller
2022/01/07 08:34:04 [INFO] Starting /v1, Kind=Node controller
2022/01/07 08:34:04 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2022/01/07 08:34:04 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2022/01/07 08:34:04 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2022/01/07 08:34:04 [INFO] Starting /v1, Kind=Namespace controller
2022/01/07 08:34:04 [INFO] Starting /v1, Kind=Secret controller
2022/01/07 08:34:04 [INFO] uploading digitaloceancredentialConfig to credentialconfig schema
2022/01/07 08:34:04 [INFO] Creating clusterRole for roleTemplate Cluster Owner (cluster-owner).
2022/01/07 08:34:04 [INFO] Starting rke-machine.cattle.io/v1, Kind=AzureMachineTemplate controller
2022/01/07 08:34:04 [INFO] Updating global catalog library
2022/01/07 08:34:04 [ERROR] Failed to handling tunnel request from 192.168.35.72:44154: response 400: cluster not found
2022/01/07 08:34:04 [INFO] Catalog sync done. 11 templates created, 0 templates updated, 0 templates deleted, 0 templates failed
2022/01/07 08:34:04 [INFO] Created machine for node [local-node]
2022/01/07 08:34:04 [INFO] Creating clusterRoleBinding User u-mo773yttt4 Role cluster-owner
2022/01/07 08:34:04 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=Amazonec2Config
2022/01/07 08:34:04 [ERROR] Failed to handling tunnel request from 192.168.35.72:44156: response 400: cluster not found
2022/01/07 08:34:04 [ERROR] Failed to handling tunnel request from 192.168.35.72:44158: response 400: cluster not found
2022/01/07 08:34:04 [ERROR] Failed to handling tunnel request from 192.168.35.72:44160: response 400: cluster not found
2022/01/07 08:34:05 [ERROR] Failed to handling tunnel request from 192.168.35.72:44164: response 400: cluster not found
2022/01/07 08:34:05 [ERROR] Failed to handling tunnel request from 192.168.35.72:44162: response 400: cluster not found
2022/01/07 08:34:05 [ERROR] Failed to handling tunnel request from 192.168.35.72:44166: response 400: cluster not found
2022/01/07 08:34:05 [ERROR] Failed to handling tunnel request from 192.168.35.72:44168: response 400: cluster not found
2022/01/07 08:34:05 [INFO] Creating user for principal system://local
2022/01/07 08:34:05 [INFO] Starting rke-machine-config.cattle.io/v1, Kind=Amazonec2Config controller
2022/01/07 08:34:05 [INFO] update kontainerdriver amazonelasticcontainerservice
2022/01/07 08:34:06 [INFO] Updating global catalog system-library
2022/01/07 08:34:06 [ERROR] error syncing 'nodes-view': handler auth-prov-v2-roletemplate: no matches for kind "HarvesterConfig" in version "rke-machine-config.cattle.io/v1", requeuing
2022/01/07 08:34:06 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=VmwarevsphereConfig
2022/01/07 08:34:06 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=LinodeConfig
2022/01/07 08:34:06 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=LinodeMachine
2022/01/07 08:34:06 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=VmwarevsphereMachineTemplate
2022/01/07 08:34:06 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=VmwarevsphereMachine
2022/01/07 08:34:06 [ERROR] error syncing 'azurekubernetesservice': handler mgmt-kontainer-driver-lifecycle: dynamicschemas.management.cattle.io "cluster" already exists, requeuing
2022/01/07 08:34:06 [INFO] update kontainerdriver azurekubernetesservice
2022/01/07 08:34:06 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-nsdv7 to namespace=default
2022/01/07 08:34:06 [INFO] uploading googlecredentialConfig to credentialconfig schema
2022/01/07 08:34:06 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-data, err=Operation cannot be fulfilled on namespaces "cattle-global-data": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:06 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-fleet-system, err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:06 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "default": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:06 [INFO] uploading linodecredentialConfig to credentialconfig schema
2022/01/07 08:34:06 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-system, err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:06 [INFO] update kontainerdriver googlekubernetesengine
2022/01/07 08:34:06 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-global-nt, err=Operation cannot be fulfilled on namespaces "cattle-global-nt": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:06 [INFO] uploading harvestercredentialConfig to credentialconfig schema
2022/01/07 08:34:06 [INFO] Creating globalRoleBindings for u-b4qkhsnliz
2022/01/07 08:34:06 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=cattle-fleet-system
2022/01/07 08:34:06 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-node-lease, err=Operation cannot be fulfilled on namespaces "kube-node-lease": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:06 [ERROR] Failed to handling tunnel request from 192.168.35.72:44170: response 401: failed authentication
2022/01/07 08:34:06 [ERROR] Failed to handling tunnel request from 192.168.35.72:44172: response 401: failed authentication
2022/01/07 08:34:06 [INFO] uploading linodecredentialConfig to credentialconfig schema
2022/01/07 08:34:06 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=cattle-global-nt
2022/01/07 08:34:06 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:06 [INFO] uploading linodecredentialConfig to credentialconfig schema
2022/01/07 08:34:06 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-global-nt": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:06 [INFO] Creating new GlobalRoleBinding for GlobalRoleBinding grb-cxwtg
2022/01/07 08:34:06 [INFO] [mgmt-auth-grb-controller] Creating clusterRoleBinding for globalRoleBinding grb-cxwtg for user u-b4qkhsnliz with role cattle-globalrole-user
2022/01/07 08:34:07 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=kube-public
2022/01/07 08:34:07 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=cattle-impersonation-system
2022/01/07 08:34:07 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:07 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-nsdv7 to namespace=default
2022/01/07 08:34:07 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-public, err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:07 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=cattle-fleet-system
2022/01/07 08:34:07 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-impersonation-system, err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:07 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-gvjv8, err=Operation cannot be fulfilled on namespaces "p-gvjv8": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:07 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=cattle-system
2022/01/07 08:34:07 [INFO] Starting rke-machine-config.cattle.io/v1, Kind=VmwarevsphereConfig controller
2022/01/07 08:34:07 [INFO] Starting rke-machine-config.cattle.io/v1, Kind=LinodeConfig controller
2022/01/07 08:34:07 [INFO] Starting rke-machine.cattle.io/v1, Kind=LinodeMachine controller
2022/01/07 08:34:07 [INFO] Starting rke-machine.cattle.io/v1, Kind=VmwarevsphereMachineTemplate controller
2022/01/07 08:34:07 [INFO] Starting rke-machine.cattle.io/v1, Kind=VmwarevsphereMachine controller
2022/01/07 08:34:07 [INFO] [mgmt-auth-crtb-controller] Creating clusterRoleBinding for membership in cluster local for subject u-b4qkhsnliz
2022/01/07 08:34:07 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-fleet-system": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:07 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:07 [ERROR] error syncing 'mutating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:34:07 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=HarvesterConfig
2022/01/07 08:34:07 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: services "webhook-service" not found, requeuing
2022/01/07 08:34:07 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=kube-public
2022/01/07 08:34:07 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-b4qkhsnliz with role cluster-owner in namespace local
2022/01/07 08:34:07 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=kube-node-lease
2022/01/07 08:34:07 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=fleet-default
2022/01/07 08:34:07 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "fleet-default": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:07 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-nsdv7, err=Operation cannot be fulfilled on namespaces "p-nsdv7": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:07 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-node-lease": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:07 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=cattle-global-nt
2022/01/07 08:34:08 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-b4qkhsnliz with role cluster-owner in namespace p-nsdv7
2022/01/07 08:34:08 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=fleet-local, err=Operation cannot be fulfilled on namespaces "fleet-local": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:08 [INFO] Active TLS secret tls-rancher-internal (ver=1855) (count 2): map[field.cattle.io/projectId:local:p-gvjv8 listener.cattle.io/cn-10.43.57.81:10.43.57.81 listener.cattle.io/fingerprint:SHA1=70DB7526EE13DFFC95E4283F33C9ED9BEF525DCA]
2022/01/07 08:34:08 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=cattle-global-data
2022/01/07 08:34:08 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=fleet-local
2022/01/07 08:34:08 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-global-nt": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:08 [INFO] [mgmt-auth-crtb-controller] Creating roleBinding for subject u-b4qkhsnliz with role cluster-owner in namespace p-gvjv8
2022/01/07 08:34:08 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "fleet-local": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:08 [INFO] Starting rke-machine-config.cattle.io/v1, Kind=HarvesterConfig controller
2022/01/07 08:34:08 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=LinodeMachineTemplate
2022/01/07 08:34:08 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-global-data": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:08 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-gvjv8, err=Operation cannot be fulfilled on namespaces "p-gvjv8": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:08 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-public, err=Operation cannot be fulfilled on namespaces "kube-public": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:08 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=cattle-fleet-system
2022/01/07 08:34:08 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=kube-node-lease, err=Operation cannot be fulfilled on namespaces "kube-node-lease": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:08 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=kube-node-lease
2022/01/07 08:34:08 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-nsdv7, err=Operation cannot be fulfilled on namespaces "p-nsdv7": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:08 [INFO] Starting rke-machine.cattle.io/v1, Kind=LinodeMachineTemplate controller
2022/01/07 08:34:08 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=Amazonec2Machine
2022/01/07 08:34:08 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=HarvesterMachine
2022/01/07 08:34:08 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=HarvesterMachineTemplate
2022/01/07 08:34:08 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=Amazonec2MachineTemplate
2022/01/07 08:34:09 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=cattle-impersonation-system, err=Operation cannot be fulfilled on namespaces "cattle-impersonation-system": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:09 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-node-lease": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:09 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=cattle-global-nt
2022/01/07 08:34:09 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=fleet-default
2022/01/07 08:34:09 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-nsdv7, err=Operation cannot be fulfilled on namespaces "p-nsdv7": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:09 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "fleet-default": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:09 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=cattle-global-data
2022/01/07 08:34:09 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=fleet-local
2022/01/07 08:34:09 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "fleet-local": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:09 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-nsdv7, err=Operation cannot be fulfilled on namespaces "p-nsdv7": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:09 [INFO] Creating clusterRoleBinding User u-b4qkhsnliz Role cluster-owner
2022/01/07 08:34:09 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=kube-node-lease
2022/01/07 08:34:09 [ERROR] defaultSvcAccountHandler: Sync: error handling default ServiceAccount of namespace key=p-nsdv7, err=Operation cannot be fulfilled on namespaces "p-nsdv7": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:09 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=fleet-default
2022/01/07 08:34:09 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "kube-node-lease": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:09 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=cattle-system
2022/01/07 08:34:09 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:09 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=fleet-local
2022/01/07 08:34:09 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=kube-node-lease
2022/01/07 08:34:10 [INFO] Starting rke-machine.cattle.io/v1, Kind=Amazonec2Machine controller
2022/01/07 08:34:10 [INFO] Starting rke-machine.cattle.io/v1, Kind=HarvesterMachine controller
2022/01/07 08:34:10 [INFO] Starting rke-machine.cattle.io/v1, Kind=HarvesterMachineTemplate controller
2022/01/07 08:34:10 [INFO] Starting rke-machine.cattle.io/v1, Kind=Amazonec2MachineTemplate controller
2022/01/07 08:34:10 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig
2022/01/07 08:34:10 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate
2022/01/07 08:34:10 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine
2022/01/07 08:34:10 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label field.cattle.io/projectId=p-gvjv8 to namespace=cattle-system
2022/01/07 08:34:11 [ERROR] namespaceHandler: Sync: error adding project id label to namespace err=Operation cannot be fulfilled on namespaces "cattle-system": the object has been modified; please apply your changes to the latest version and try again
2022/01/07 08:34:11 [INFO] Starting rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig controller
2022/01/07 08:34:11 [INFO] Starting rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate controller
2022/01/07 08:34:11 [INFO] Starting rke-machine.cattle.io/v1, Kind=DigitaloceanMachine controller
2022/01/07 08:34:11 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:33143
2022/01/07 08:34:11 [INFO] kontainerdriver amazonelasticcontainerservice stopped
2022/01/07 08:34:11 [INFO] dynamic schema for kontainerdriver amazonelasticcontainerservice updating
2022/01/07 08:34:11 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:46785
2022/01/07 08:34:11 [INFO] kontainerdriver azurekubernetesservice stopped
2022/01/07 08:34:11 [INFO] dynamic schema for kontainerdriver azurekubernetesservice updating
2022/01/07 08:34:11 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:37961
2022/01/07 08:34:11 [INFO] kontainerdriver googlekubernetesengine stopped
........
.....
...

anandr781 · January 10, 2022, 3:56am

Pls check out the release notes of 2.6.3 it carries a warning that an upgrade might wipe out .

https://github.com/rancher/rancher/releases/tag/v2.6.3

But it also specifies a possible work around .

jme · January 10, 2022, 6:45am

@anandr781
Okay, thank you I didn’t see the obvious. It works.

Topic		Replies	Views
Restarted Rancher Server - Lost Configuration Rancher 1.x	4	3141	October 24, 2016
Restart rancher/server lose all hosts and stacks Rancher 1.x	3	2963	October 6, 2016
Trouble with upgrading to v0.31.0 Rancher 1.x	3	915	August 6, 2015
Rancher 2.xdocker install single node Rancher	3	408	June 18, 2021
I lost everything! After I updated the rancher to 2.6.4 Rancher	1	534	May 5, 2022

Rancher Update on docker Host lost all Config Data

Related Topics