Uninstalled traefik, rancher no longer works

bramnet · August 28, 2022, 3:42am

So, I did a big no-no… I uninstalled traefik from my cluster, and now trying to reach rancher returns a 404 page not found. I had helm reinstall traefik, no change. I checked the logs for the rancher pod, the only major error I can see that might be the issue is “Error during subscribe write tcp 10.42.0.60:80->10.42.2.138:36430: write: no route to host” I’ve tried googling around, couldn’t find any good tips to try and resolve the situation.

shlomi · August 28, 2022, 6:06am

Hi,
Can you provide log from the traefik and rancher pods?

bramnet · August 29, 2022, 12:35am

I was busy with personal stuff most of the morning and afternoon, sorry for the delay. I had trouble running the logs command with the pod, only showing one message since it was live for so long. I scaled the deployment to 0 replicas and then to 1 to force it to reload, and I’m not getting the same error message I said in my original post
Here’s the log for the rancher pod:

2022/08/29 00:23:44 [INFO] Rancher version v2.6.7 (e5c6f0f6a) is starting
2022/08/29 00:23:44 [INFO] Listening on /tmp/log.sock
2022/08/29 00:23:44 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
2022/08/29 00:23:44 [INFO] Running in clustered mode with ID 10.42.3.243, monitoring endpoint cattle-system/rancher
2022/08/29 00:23:46 [INFO] Applying CRD features.management.cattle.io
2022/08/29 00:23:51 [INFO] Applying CRD navlinks.ui.cattle.io
2022/08/29 00:23:51 [INFO] Applying CRD clusters.management.cattle.io
2022/08/29 00:23:52 [INFO] Applying CRD apiservices.management.cattle.io
2022/08/29 00:23:52 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
2022/08/29 00:23:53 [INFO] Applying CRD settings.management.cattle.io
2022/08/29 00:23:53 [INFO] Applying CRD preferences.management.cattle.io
2022/08/29 00:23:54 [INFO] Applying CRD features.management.cattle.io
2022/08/29 00:23:54 [INFO] Applying CRD clusterrepos.catalog.cattle.io
2022/08/29 00:23:54 [INFO] Applying CRD operations.catalog.cattle.io
2022/08/29 00:23:55 [INFO] Applying CRD apps.catalog.cattle.io
2022/08/29 00:23:56 [INFO] Applying CRD fleetworkspaces.management.cattle.io
2022/08/29 00:23:56 [INFO] Applying CRD managedcharts.management.cattle.io
2022/08/29 00:23:57 [INFO] Applying CRD clusters.provisioning.cattle.io
2022/08/29 00:23:57 [INFO] Applying CRD clusters.provisioning.cattle.io
2022/08/29 00:23:58 [INFO] Applying CRD rkeclusters.rke.cattle.io
2022/08/29 00:23:58 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2022/08/29 00:23:59 [INFO] Applying CRD rkebootstraps.rke.cattle.io
2022/08/29 00:23:59 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
2022/08/29 00:24:00 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2022/08/29 00:24:00 [INFO] Applying CRD custommachines.rke.cattle.io
2022/08/29 00:24:01 [INFO] Applying CRD etcdsnapshots.rke.cattle.io
2022/08/29 00:24:01 [INFO] Applying CRD clusters.cluster.x-k8s.io
2022/08/29 00:24:02 [INFO] Applying CRD machinedeployments.cluster.x-k8s.io
2022/08/29 00:24:02 [INFO] Applying CRD machinehealthchecks.cluster.x-k8s.io
2022/08/29 00:24:03 [INFO] Applying CRD machines.cluster.x-k8s.io
2022/08/29 00:24:04 [INFO] Applying CRD machinesets.cluster.x-k8s.io
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=GlobalDns controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplate controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=CisConfig controller
I0829 00:24:46.318218      33 leaderelection.go:248] attempting to acquire leader lease kube-system/cattle-controllers...
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=Endpoints controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Preference controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=KontainerDriver controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterAppRevision controller
2022/08/29 00:24:46 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeCredential controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Project controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller
2022/08/29 00:24:46 [INFO] Starting project.cattle.io/v3, Kind=Pipeline controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=Namespace controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=ConfigMap controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ClusterCatalog controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sSystemImage controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ClusterRegistrationToken controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=ServiceAccount controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplate controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2022/08/29 00:24:46 [INFO] Starting project.cattle.io/v3, Kind=App controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=Secret controller
2022/08/29 00:24:46 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrap controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=NodeDriver controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Feature controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=CisBenchmarkVersion controller
2022/08/29 00:24:46 [INFO] Starting API controllers
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Catalog controller
2022/08/29 00:24:46 [INFO] Starting cluster controllers for local
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=NodeTemplate controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterApp controller
2022/08/29 00:24:46 [INFO] Starting apiextensions.k8s.io/v1, Kind=CustomResourceDefinition controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=RkeAddon controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=AuthConfig controller
2022/08/29 00:24:46 [INFO] Starting provisioning.cattle.io/v1, Kind=Cluster controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=GlobalRole controller
2022/08/29 00:24:46 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeRepository controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=DynamicSchema controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ProjectLogging controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplateVersion controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Setting controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=APIService controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=User controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplate controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
2022/08/29 00:24:46 [INFO] Starting project.cattle.io/v3, Kind=PipelineExecution controller
2022/08/29 00:24:46 [INFO] Starting cluster.x-k8s.io/v1beta1, Kind=Machine controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sServiceOption controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
2022/08/29 00:24:46 [INFO] Starting catalog.cattle.io/v1, Kind=ClusterRepo controller
2022/08/29 00:24:46 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=Secret controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=ServiceAccount controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=Namespace controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2022/08/29 00:24:46 [INFO] Starting cluster agent for local [owner=false]
2022/08/29 00:24:47 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2022/08/29 00:24:47 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2022/08/29 00:24:47 [INFO] Starting management.cattle.io/v3, Kind=User controller
2022/08/29 00:24:47 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2022/08/29 00:24:47 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2022/08/29 00:24:47 [INFO] Starting /v1, Kind=Secret controller
2022/08/29 00:24:47 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2022/08/29 00:24:47 [INFO] Active TLS secret cattle-system/serving-cert (ver=37806133) (count 8): map[field.cattle.io/projectId:local:p-pvcv6 listener.cattle.io/cn-10.42.0.60:10.42.0.60 listener.cattle.io/cn-10.42.3.242:10.42.3.242 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.1.88.sslip.io:192.168.1.88.sslip.io listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=629B6E1EA3957C0E4B3CC941DC81B84BA26D39F1]
2022/08/29 00:24:47 [INFO] Listening on :80
2022/08/29 00:24:47 [INFO] Listening on :443
2022/08/29 00:24:47 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca@1661641448,O=dynamiclistener-org: notBefore=2022-08-27 23:04:08 +0000 UTC notAfter=2023-08-29 00:24:47 +0000 UTC
2022/08/29 00:24:47 [INFO] Updating TLS secret for cattle-system/serving-cert (count: 9): map[field.cattle.io/projectId:local:p-pvcv6 listener.cattle.io/cn-10.42.0.60:10.42.0.60 listener.cattle.io/cn-10.42.3.242:10.42.3.242 listener.cattle.io/cn-10.42.3.243:10.42.3.243 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.1.88.sslip.io:192.168.1.88.sslip.io listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=1EFC00ECE6FBE9E546BFD9688745B1AFD6B7AC16]
2022/08/29 00:24:47 [INFO] Active TLS secret cattle-system/serving-cert (ver=37812352) (count 9): map[field.cattle.io/projectId:local:p-pvcv6 listener.cattle.io/cn-10.42.0.60:10.42.0.60 listener.cattle.io/cn-10.42.3.242:10.42.3.242 listener.cattle.io/cn-10.42.3.243:10.42.3.243 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.1.88.sslip.io:192.168.1.88.sslip.io listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=1EFC00ECE6FBE9E546BFD9688745B1AFD6B7AC16]
2022/08/29 00:24:47 [INFO] Active TLS secret cattle-system/tls-rancher-internal (ver=37067875) (count 2): map[field.cattle.io/projectId:local:p-pvcv6 listener.cattle.io/cn-10.43.234.219:10.43.234.219 listener.cattle.io/fingerprint:SHA1=C32EBD9E64C1A462D0FFADF30BF4448DA9B03619]
2022/08/29 00:24:47 [INFO] Listening on :444
2022/08/29 00:24:48 [INFO] Starting /v1, Kind=Secret controller
2022/08/29 00:24:48 [INFO] Updating TLS secret for cattle-system/tls-rancher-internal (count: 2): map[field.cattle.io/projectId:local:p-pvcv6 listener.cattle.io/cn-10.43.234.219:10.43.234.219 listener.cattle.io/fingerprint:SHA1=C32EBD9E64C1A462D0FFADF30BF4448DA9B03619]
2022/08/29 00:24:48 [INFO] Updating TLS secret for cattle-system/serving-cert (count: 9): map[field.cattle.io/projectId:local:p-pvcv6 listener.cattle.io/cn-10.42.0.60:10.42.0.60 listener.cattle.io/cn-10.42.3.242:10.42.3.242 listener.cattle.io/cn-10.42.3.243:10.42.3.243 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.1.88.sslip.io:192.168.1.88.sslip.io listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=1EFC00ECE6FBE9E546BFD9688745B1AFD6B7AC16]
W0829 00:24:50.824568      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging
2022/08/29 00:24:51 [INFO] Watching metadata for helm.cattle.io/v1, Kind=HelmChart
2022/08/29 00:24:51 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEBootstrap
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=ServiceAccount
2022/08/29 00:24:51 [INFO] Watching metadata for k8s.nginx.org/v1, Kind=Policy
2022/08/29 00:24:51 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=MachineSet
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ManagedChart
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Node
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Pod
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=ConfigMap
2022/08/29 00:24:51 [INFO] Watching metadata for batch/v1, Kind=CronJob
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember
2022/08/29 00:24:51 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=InstanceManager
2022/08/29 00:24:51 [INFO] Watching metadata for certificates.k8s.io/v1, Kind=CertificateSigningRequest
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=BackupVolume
2022/08/29 00:24:51 [INFO] Watching metadata for cert-manager.io/v1, Kind=Certificate
2022/08/29 00:24:51 [INFO] Watching metadata for node.k8s.io/v1, Kind=RuntimeClass
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepoRestriction
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group
2022/08/29 00:24:51 [INFO] Watching metadata for k3s.cattle.io/v1, Kind=Addon
2022/08/29 00:24:51 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSIStorageCapacity
2022/08/29 00:24:51 [INFO] Watching metadata for kubevirt.io/v1, Kind=KubeVirt
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=IngressRouteUDP
2022/08/29 00:24:51 [INFO] Watching metadata for cert-manager.io/v1, Kind=ClusterIssuer
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ImageScan
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistrationToken
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=ResourceQuota
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterGroup
2022/08/29 00:24:51 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=ReplicationController
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=BackingImage
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=APIService
2022/08/29 00:24:51 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=Machine
2022/08/29 00:24:51 [INFO] Watching metadata for flowcontrol.apiserver.k8s.io/v1beta2, Kind=FlowSchema
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=RancherUserNotification
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Namespace
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=ShareManager
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PodMonitor
2022/08/29 00:24:51 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSIDriver
2022/08/29 00:24:51 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=App
2022/08/29 00:24:51 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEBootstrapTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver
2022/08/29 00:24:51 [INFO] Watching metadata for flowcontrol.apiserver.k8s.io/v1beta2, Kind=PriorityLevelConfiguration
2022/08/29 00:24:51 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=ClusterRepo
2022/08/29 00:24:51 [INFO] Watching metadata for k8s.nginx.org/v1, Kind=VirtualServer
2022/08/29 00:24:51 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Order
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepo
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph
2022/08/29 00:24:51 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=MachineHealthCheck
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
2022/08/29 00:24:51 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=FleetWorkspace
2022/08/29 00:24:51 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=MiddlewareTCP
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Endpoints
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=User
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision
2022/08/29 00:24:51 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
2022/08/29 00:24:51 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Cluster
2022/08/29 00:24:51 [INFO] Watching metadata for cert-manager.io/v1, Kind=CertificateRequest
2022/08/29 00:24:51 [INFO] Watching metadata for ui.cattle.io/v1, Kind=NavLink
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding
2022/08/29 00:24:51 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=Cluster
2022/08/29 00:24:51 [INFO] Watching metadata for events.k8s.io/v1, Kind=Event
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
2022/08/29 00:24:51 [INFO] Watching metadata for batch/v1, Kind=Job
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=TLSStore
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Replica
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager
2022/08/29 00:24:51 [INFO] Watching metadata for provisioning.cattle.io/v1, Kind=Cluster
2022/08/29 00:24:51 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision
2022/08/29 00:24:51 [INFO] Watching metadata for apps/v1, Kind=DaemonSet
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=LimitRange
2022/08/29 00:24:51 [INFO] Watching metadata for apps/v1, Kind=Deployment
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Setting
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy
2022/08/29 00:24:51 [INFO] Watching metadata for k8s.nginx.org/v1, Kind=VirtualServerRoute
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
2022/08/29 00:24:51 [INFO] Watching metadata for rke.cattle.io/v1, Kind=ETCDSnapshot
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Snapshot
2022/08/29 00:24:51 [INFO] Watching metadata for k8s.nginx.org/v1alpha1, Kind=TransportServer
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster
2022/08/29 00:24:51 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=Operation
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=TraefikService
2022/08/29 00:24:51 [INFO] Watching metadata for apps/v1, Kind=StatefulSet
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token
2022/08/29 00:24:51 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease
2022/08/29 00:24:51 [INFO] Watching metadata for appprotectdos.f5.com/v1beta1, Kind=APDosPolicy
2022/08/29 00:24:51 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass
2022/08/29 00:24:51 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=MachineDeployment
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Probe
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=ServersTransport
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule
2022/08/29 00:24:51 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Service
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=Middleware
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Node
2022/08/29 00:24:51 [INFO] Watching metadata for appprotect.f5.com/v1beta1, Kind=APPolicy
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent
2022/08/29 00:24:51 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEControlPlane
2022/08/29 00:24:51 [INFO] Watching metadata for appprotect.f5.com/v1beta1, Kind=APUserSig
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature
2022/08/29 00:24:51 [INFO] Watching metadata for helm.cattle.io/v1, Kind=HelmChartConfig
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Engine
2022/08/29 00:24:51 [INFO] Watching metadata for appprotectdos.f5.com/v1beta1, Kind=DosProtectedResource
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns
2022/08/29 00:24:51 [INFO] Watching metadata for discovery.k8s.io/v1, Kind=EndpointSlice
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=PodTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template
2022/08/29 00:24:51 [INFO] Watching metadata for rke.cattle.io/v1, Kind=CustomMachine
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Orphan
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=BackingImageManager
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ThanosRuler
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1alpha1, Kind=AlertmanagerConfig
2022/08/29 00:24:51 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Backup
2022/08/29 00:24:51 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleDeployment
2022/08/29 00:24:51 [INFO] Watching metadata for policy/v1, Kind=PodDisruptionBudget
2022/08/29 00:24:51 [INFO] Watching metadata for networking.k8s.io/v1, Kind=IngressClass
2022/08/29 00:24:51 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Challenge
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=IngressRouteTCP
2022/08/29 00:24:51 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Content
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Volume
2022/08/29 00:24:51 [INFO] Watching metadata for gitjob.cattle.io/v1, Kind=GitJob
2022/08/29 00:24:51 [INFO] Watching metadata for networking.k8s.io/v1, Kind=Ingress
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository
2022/08/29 00:24:51 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=BackingImageDataSource
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=RecurringJob
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=PersistentVolume
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption
2022/08/29 00:24:51 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKECluster
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool
2022/08/29 00:24:51 [INFO] Watching metadata for k8s.nginx.org/v1alpha1, Kind=GlobalConfiguration
2022/08/29 00:24:51 [INFO] Watching metadata for appprotectdos.f5.com/v1beta1, Kind=APDosLogConf
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=EngineImage
2022/08/29 00:24:51 [INFO] Watching metadata for appprotect.f5.com/v1beta1, Kind=APLogConf
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Secret
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=TLSOption
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken
2022/08/29 00:24:51 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Event
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Bundle
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion
2022/08/29 00:24:51 [INFO] Watching metadata for upgrade.cattle.io/v1, Kind=Plan
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=App
2022/08/29 00:24:51 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role
2022/08/29 00:24:51 [INFO] Watching metadata for autoscaling/v2, Kind=HorizontalPodAutoscaler
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleNamespaceMapping
2022/08/29 00:24:51 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup
2022/08/29 00:24:51 [INFO] Watching metadata for cert-manager.io/v1, Kind=Issuer
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=IngressRoute
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=BackupTarget
W0829 00:24:52.056355      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0829 00:24:52.149133      33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
I0829 00:25:33.735384      33 leaderelection.go:258] successfully acquired lease kube-system/cattle-controllers
2022/08/29 00:25:33 [INFO] Stopping cluster agent for local
2022/08/29 00:25:33 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2022/08/29 00:25:33 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
2022/08/29 00:25:33 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
2022/08/29 00:25:33 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding workers
2022/08/29 00:25:33 [INFO] Shutting down /v1, Kind=Namespace workers
2022/08/29 00:25:33 [INFO] Shutting down /v1, Kind=ServiceAccount workers
2022/08/29 00:25:33 [INFO] Shutting down /v1, Kind=Secret workers
2022/08/29 00:25:33 [INFO] Registering istio for cluster "local"
2022/08/29 00:25:33 [INFO] Starting cluster controllers for local
2022/08/29 00:25:33 [INFO] Reconciling GlobalRoles
2022/08/29 00:25:33 [INFO] Starting management.cattle.io/v3, Kind=ClusterLogging controller
2022/08/29 00:25:33 [INFO] Starting rke.cattle.io/v1, Kind=ETCDSnapshot controller
2022/08/29 00:25:33 [INFO] Starting management.cattle.io/v3, Kind=Notifier controller
2022/08/29 00:25:33 [INFO] Starting management.cattle.io/v3, Kind=ClusterScan controller
2022/08/29 00:25:33 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlert controller
2022/08/29 00:25:33 [INFO] Reconciling RoleTemplates
2022/08/29 00:25:35 [INFO] Steve auth startup complete
2022/08/29 00:25:35 [INFO] Registering namespaceHandler for adding labels
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=ResourceQuota controller
2022/08/29 00:25:36 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=ConfigMap controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=ServiceAccount controller
2022/08/29 00:25:36 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=Namespace controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=LimitRange controller
2022/08/29 00:25:36 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2022/08/29 00:25:36 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2022/08/29 00:25:36 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=Node controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=Secret controller
2022/08/29 00:25:36 [INFO] Starting cluster agent for local [owner=true]
2022/08/29 00:25:36 [INFO] Starting cluster controllers for local
2022/08/29 00:25:36 [INFO] adding kontainer driver rancherKubernetesEngine
2022/08/29 00:25:36 [INFO] adding kontainer driver googleKubernetesEngine
2022/08/29 00:25:36 [INFO] adding kontainer driver azureKubernetesService
2022/08/29 00:25:36 [INFO] Starting fleet.cattle.io/v1alpha1, Kind=Bundle controller
2022/08/29 00:25:36 [INFO] Starting batch/v1, Kind=Job controller
2022/08/29 00:25:36 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrapTemplate controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=ReplicationController controller
2022/08/29 00:25:36 [INFO] Starting batch/v1beta1, Kind=CronJob controller
2022/08/29 00:25:36 [INFO] Starting rke.cattle.io/v1, Kind=CustomMachine controller
2022/08/29 00:25:36 [INFO] adding kontainer driver amazonElasticContainerService
2022/08/29 00:25:36 [INFO] Starting management.cattle.io/v3, Kind=FleetWorkspace controller
2022/08/29 00:25:36 [INFO] Starting networking.k8s.io/v1, Kind=Ingress controller
2022/08/29 00:25:36 [INFO] Starting rke.cattle.io/v1, Kind=RKECluster controller
2022/08/29 00:25:36 [INFO] Starting fleet.cattle.io/v1alpha1, Kind=Cluster controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=Service controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=Node controller
2022/08/29 00:25:36 [INFO] Starting admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=StatefulSet controller
2022/08/29 00:25:36 [INFO] Starting catalog.cattle.io/v1, Kind=Operation controller
2022/08/29 00:25:36 [INFO] Starting management.cattle.io/v3, Kind=ManagedChart controller
2022/08/29 00:25:36 [INFO] Starting cluster.x-k8s.io/v1beta1, Kind=MachineDeployment controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=DaemonSet controller
2022/08/29 00:25:36 [INFO] Starting rke.cattle.io/v1, Kind=RKEControlPlane controller
2022/08/29 00:25:36 [INFO] Starting cluster.x-k8s.io/v1beta1, Kind=Cluster controller
2022/08/29 00:25:36 [INFO] Starting catalog.cattle.io/v1, Kind=App controller
2022/08/29 00:25:36 [INFO] Starting admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=ReplicaSet controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=Deployment controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=Pod controller
2022/08/29 00:25:36 [INFO] adding kontainer driver baiducloudcontainerengine
2022/08/29 00:25:36 [INFO] adding kontainer driver aliyunkubernetescontainerservice
2022/08/29 00:25:36 [INFO] adding kontainer driver tencentkubernetesengine
2022/08/29 00:25:36 [INFO] adding kontainer driver huaweicontainercloudengine
2022/08/29 00:25:36 [INFO] adding kontainer driver oraclecontainerengine
2022/08/29 00:25:36 [INFO] adding kontainer driver linodekubernetesengine
2022/08/29 00:25:36 [INFO] adding kontainer driver opentelekomcloudcontainerengine
2022/08/29 00:25:36 [INFO] Updating node driver cloudscale
2022/08/29 00:25:36 [INFO] Starting batch/v1, Kind=Job controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=ReplicaSet controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=ReplicationController controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=StatefulSet controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=Deployment controller
2022/08/29 00:25:36 [INFO] Starting batch/v1beta1, Kind=CronJob controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=DaemonSet controller
2022/08/29 00:25:36 [INFO] Starting catalog controller
2022/08/29 00:25:36 [INFO] Starting project-level catalog controller
2022/08/29 00:25:36 [INFO] Starting cluster-level catalog controller
2022/08/29 00:25:37 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=Machine
2022/08/29 00:25:37 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2022/08/29 00:25:37 [INFO] Refreshing driverMetadata in 1440 minutes
2022/08/29 00:25:37 [INFO] update kontainerdriver tencentkubernetesengine
2022/08/29 00:25:37 [INFO] update kontainerdriver rancherkubernetesengine
2022/08/29 00:25:37 [INFO] update kontainerdriver oraclecontainerengine
2022/08/29 00:25:37 [INFO] update kontainerdriver aliyunkubernetescontainerservice
2022/08/29 00:25:37 [INFO] update kontainerdriver opentelekomcloudcontainerengine
2022/08/29 00:25:37 [INFO] update kontainerdriver huaweicontainercloudengine
2022/08/29 00:25:37 [INFO] update kontainerdriver amazonelasticcontainerservice
2022/08/29 00:25:37 [INFO] update kontainerdriver azurekubernetesservice
2022/08/29 00:25:37 [INFO] update kontainerdriver baiducloudcontainerengine
2022/08/29 00:25:37 [INFO] update kontainerdriver linodekubernetesengine
2022/08/29 00:25:37 [INFO] update kontainerdriver googlekubernetesengine
2022/08/29 00:25:37 [INFO] Starting management.cattle.io/v3, Kind=EtcdBackup controller
2022/08/29 00:25:37 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertRule controller
2022/08/29 00:25:37 [INFO] Starting management.cattle.io/v3, Kind=ComposeConfig controller
2022/08/29 00:25:37 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertGroup controller
2022/08/29 00:25:37 [INFO] Starting management.cattle.io/v3, Kind=GlobalDnsProvider controller
2022/08/29 00:25:37 [INFO] getReleaseConfig: reloading config for k3s
2022/08/29 00:25:37 [INFO] getReleaseConfig: reloading config for rke2
2022/08/29 00:25:37 [INFO] Rancher startup complete
2022/08/29 00:25:37 [INFO] checking configmap cattle-system/admincreated to determine if duplicate bindings cleanup needs to run
2022/08/29 00:25:37 [INFO] checking configmap cattle-system/admincreated to determine if orphan bindings cleanup needs to run
2022/08/29 00:25:37 [INFO] [CleanupOrphanBindingsDone] orphan bindings cleanup has already run, skipping
2022/08/29 00:25:37 [INFO] checking configmap cattle-system/admincreated to determine if orphan bindings cleanup needs to run
2022/08/29 00:25:37 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=Machine controller
2022/08/29 00:25:37 [INFO] duplicate bindings cleanup has already run, skipping
2022/08/29 00:25:37 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment
2022/08/29 00:25:37 [INFO] [clean-catalog-orphan-bindings] cleaning up orphaned catalog bindings
2022/08/29 00:25:37 [INFO] [clean-catalog-orphan-bindings] Processing 2 rolebindings
2022/08/29 00:25:37 [INFO] [clean-catalog-orphan-bindings] Deleting orphaned role global-catalog
2022/08/29 00:25:37 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment controller
2022/08/29 00:25:37 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=Cluster
2022/08/29 00:25:37 [WARNING] [clean-catalog-orphan-bindings] Error when deleting role global-catalog, roles.rbac.authorization.k8s.io "global-catalog" not found
2022/08/29 00:25:37 [WARNING] [CleanupOrphanCatalogBindingsDone] error during orphan binding cleanup: roles.rbac.authorization.k8s.io "global-catalog" not found
2022/08/29 00:25:37 [ERROR] failed to cleanup orphan catalog bindings
2022/08/29 00:25:37 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=Cluster controller
2022/08/29 00:25:37 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck
2022/08/29 00:25:37 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineSet
2022/08/29 00:25:37 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck controller
2022/08/29 00:25:37 [INFO] uploading digitaloceanConfig to nodeconfig schema
2022/08/29 00:25:37 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=MachineSet controller
2022/08/29 00:25:37 [INFO] Starting provisioning.cattle.io/v1, Kind=Cluster controller
2022/08/29 00:25:37 [INFO] Starting rke.cattle.io/v1, Kind=ETCDSnapshot controller
2022/08/29 00:25:37 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrap controller
2022/08/29 00:25:37 [INFO] Starting rke.cattle.io/v1, Kind=RKECluster controller
2022/08/29 00:25:37 [INFO] Starting rke.cattle.io/v1, Kind=RKEControlPlane controller
2022/08/29 00:25:37 [INFO] Starting rke.cattle.io/v1, Kind=CustomMachine controller
2022/08/29 00:25:37 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrapTemplate controller
2022/08/29 00:25:37 [INFO] uploading digitaloceanConfig to nodetemplateconfig schema
2022/08/29 00:25:38 [INFO] uploading digitaloceancredentialConfig to credentialconfig schema
2022/08/29 00:25:38 [INFO] uploading googlecredentialConfig to credentialconfig schema
2022/08/29 00:25:38 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2022/08/29 00:25:38 [INFO] Starting monitoring.coreos.com/v1, Kind=ServiceMonitor controller
2022/08/29 00:25:38 [INFO] Starting monitoring.coreos.com/v1, Kind=Prometheus controller
2022/08/29 00:25:38 [INFO] Starting management.cattle.io/v3, Kind=SamlToken controller
2022/08/29 00:25:40 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json
2022/08/29 00:25:40 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
2022/08/29 00:25:40 [INFO] Refreshing driverMetadata in 1440 minutes
2022/08/29 00:25:40 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig
2022/08/29 00:25:40 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine
2022/08/29 00:25:40 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate
2022/08/29 00:25:41 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json
2022/08/29 00:25:41 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
2022/08/29 00:25:41 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine
2022/08/29 00:25:41 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate
2022/08/29 00:25:41 [INFO] Starting rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig controller
2022/08/29 00:25:41 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine: failed to wait for caches to sync
2022/08/29 00:25:41 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate: failed to wait for caches to sync
E0829 00:25:41.884438      33 gvks.go:69] failed to sync schemas: failed to sync cache for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine
2022/08/29 00:25:41 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine
2022/08/29 00:25:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate
2022/08/29 00:25:42 [INFO] Starting rke-machine.cattle.io/v1, Kind=DigitaloceanMachine controller
2022/08/29 00:25:42 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:38321
2022/08/29 00:25:42 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:33845
2022/08/29 00:25:42 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:43789
2022/08/29 00:25:42 [INFO] Creating token for user user-pksfj
2022/08/29 00:25:42 [INFO] kontainerdriver azurekubernetesservice stopped
2022/08/29 00:25:42 [INFO] dynamic schema for kontainerdriver azurekubernetesservice updating
2022/08/29 00:25:42 [INFO] kontainerdriver googlekubernetesengine stopped
2022/08/29 00:25:42 [INFO] dynamic schema for kontainerdriver googlekubernetesengine updating
2022/08/29 00:25:42 [INFO] kontainerdriver amazonelasticcontainerservice stopped
2022/08/29 00:25:42 [INFO] dynamic schema for kontainerdriver amazonelasticcontainerservice updating
2022/08/29 00:25:42 [INFO] Starting rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate controller
time="2022-08-29 00:25:42" level=info msg="Telemetry Client v0.5.20"
time="2022-08-29 00:25:42" level=info msg="Listening on 0.0.0.0:8114"
time="2022-08-29 00:25:42" level=info msg="Collecting anonymous data from https://localhost:443/v3"
2022/08/29 00:25:43 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json in [0x6ea3898]
2022/08/29 00:25:43 [INFO] getReleaseConfig: reloading config for k3s
2022/08/29 00:25:43 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json in [0x6ea3898]
2022/08/29 00:25:43 [INFO] getReleaseConfig: reloading config for rke2
2022/08/29 00:25:47 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate
2022/08/29 00:25:47 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine
2022/08/29 00:25:47 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig
2022/08/29 00:25:47 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json in [0x6ea3898]
2022/08/29 00:25:47 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json in [0x6ea3898]

And here it is for traefik:

time="2022-08-28T03:07:22Z" level=info msg="Configuration loaded from flags."
time="2022-08-29T00:05:58Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" ingress=rancher namespace=cattle-system serviceName=rancher providerName=kubernetes
time="2022-08-29T00:09:52Z" level=error msg="Skipping service: no endpoints found" ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes
time="2022-08-29T00:11:07Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher
time="2022-08-29T00:23:23Z" level=error msg="Skipping service: no endpoints found" ingress=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" namespace=cattle-system serviceName=rancher providerName=kubernetes
time="2022-08-29T00:23:42Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2022-08-29T00:24:47Z" level=error msg="Skipping service: no endpoints found" namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher

If there’s a flag I can use to retrieve older logs, I can try that. I tried searching online, and the best I could find was -p which it looks like the poster I found was misinformed as this tries to retrieve logs of a previous pod/resource.

shlomi · August 29, 2022, 6:29am

@bramnet from the above rancher logs it looks OK, the previous log, in my opinion not related to your issue.
Traefik logs looking interesting,
can you please:
change traefik log to debug and post your service definition (ingress and service of rancher)

bramnet · August 29, 2022, 3:27pm

Okay, I think I set up the log to debug, describe says the environment variable for “log” is set to “DEBUG,” which according to traefik’s documentation is how to do it. I had to look up how to set it with yaml since I’m so used to Rancher’s UI.
I googled around for service definition, I think you mean the yaml for the services. I’m still new (half a year) so common terminology might be lost on me. Here we go.
Traefik:

apiVersion: v1
kind: Service
metadata:
  annotations:
    field.cattle.io/publicEndpoints: '[{"addresses":["192.168.1.88","192.168.1.90","192.168.1.91"],"port":80,"protocol":"TCP","serviceName":"default:traefik","allNodes":false},{"addresses":["192.168.1.88","192.168.1.90","192.168.1.91"],"port":443,"protocol":"TCP","serviceName":"default:traefik","allNodes":false}]'
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2022-08-28T03:07:06Z"
  finalizers:
  - svccontroller.k3s.cattle.io/daemonset
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-10.24.0
  name: traefik
  namespace: default
  resourceVersion: "37185722"
  uid: 87ca4a14-7e32-44c3-8161-d0961571d5b6
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.43.249.85
  clusterIPs:
  - 10.43.249.85
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: web
    nodePort: 31454
    port: 80
    protocol: TCP
    targetPort: web
  - name: websecure
    nodePort: 32579
    port: 443
    protocol: TCP
    targetPort: websecure
  selector:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 192.168.1.88
    - ip: 192.168.1.90
    - ip: 192.168.1.91

And Rancher:

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: rancher
    meta.helm.sh/release-namespace: cattle-system
  creationTimestamp: "2022-08-27T22:59:02Z"
  labels:
    app: rancher
    app.kubernetes.io/managed-by: Helm
    chart: rancher-2.6.7
    heritage: Helm
    release: rancher
  name: rancher
  namespace: cattle-system
  resourceVersion: "37065670"
  uid: 495f20a3-d8a3-496d-8fa2-9a0d26543f75
spec:
  clusterIP: 10.43.234.219
  clusterIPs:
  - 10.43.234.219
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https-internal
    port: 443
    protocol: TCP
    targetPort: 444
  selector:
    app: rancher
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

shlomi · August 31, 2022, 5:12am

@bramnet sorry for the late replay ,
Can you post the output of above commands :

curl -v rancher:80
kubectl get endpoints rancher -n cattle-system
kubectl get svc -n cattle-system

bramnet · August 31, 2022, 5:17am

Not a problem.
Curl

*   Trying 23.202.231.169:80...
* Connected to rancher (23.202.231.169) port 80 (#0)
> GET / HTTP/1.1
> Host: rancher
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Correlation-Id: aaaaaaaa
< Date: Wed, 31 Aug 2022 05:15:55 GMT
< Content-Length: 0
< Connection: close
<
* Closing connection 0

Kubectl get endpoints:

NAME      ENDPOINTS                        AGE
rancher   10.42.3.243:80,10.42.3.243:444   3d6h

kubectl get svc:

NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
rancher           ClusterIP   10.43.234.219   <none>        80/TCP,443/TCP   3d6h
webhook-service   ClusterIP   10.43.44.210    <none>        443/TCP          3d6h
rancher-webhook   ClusterIP   10.43.106.31    <none>        443/TCP          3d6h

shlomi · August 31, 2022, 5:28pm

@bramnet did you configure IngressRotue or ingress?

bramnet · August 31, 2022, 8:12pm

I’ll be honest, I have no idea. Is there a command I could run that would make it evident if I did either? I have a vague recollection of trying to configure ingress, but I’m only 20% sure of it.

bramnet · September 1, 2022, 1:36am

After doing some more research, I think I figured I have ingress… I think.
I put in “kubectl get ingress -n cattle-system” and got this returned

NAME      CLASS    HOSTS                   ADDRESS                                  PORTS     AGE
rancher   <none>   192.168.1.88.sslip.io   192.168.1.88,192.168.1.90,192.168.1.91   80, 443   4d2h

Not sure if this helps. I also had it describe it, this was the return.

Name:             rancher
Labels:           app=rancher
                  app.kubernetes.io/managed-by=Helm
                  chart=rancher-2.6.7
                  heritage=Helm
                  release=rancher
Namespace:        cattle-system
Address:          192.168.1.88,192.168.1.90,192.168.1.91
Ingress Class:    <none>
Default backend:  <default>
TLS:
  tls-rancher-ingress terminates 192.168.1.88.sslip.io
Rules:
  Host                   Path  Backends
  ----                   ----  --------
  192.168.1.88.sslip.io
                            rancher:80 (10.42.3.243:80)
Annotations:             cert-manager.io/issuer: rancher
                         cert-manager.io/issuer-kind: Issuer
                         field.cattle.io/publicEndpoints:
                           [{"addresses":["192.168.1.88","192.168.1.90","192.168.1.91"],"port":443,"protocol":"HTTPS","serviceName":"cattle-system:rancher","ingressN...
                         meta.helm.sh/release-name: rancher
                         meta.helm.sh/release-namespace: cattle-system
                         nginx.ingress.kubernetes.io/proxy-connect-timeout: 30
                         nginx.ingress.kubernetes.io/proxy-read-timeout: 1800
                         nginx.ingress.kubernetes.io/proxy-send-timeout: 1800
Events:                  <none>

When I reinstalled traefik, I reinstalled using helm. If I now need to add rancher to traefik (somehow) in order for this to work, that might be what I’m missing.

Topic		Replies	Views
Cannot get Rancher GUI to show up Rancher 2.x	1	506	October 4, 2020
Install failed, Then it went downhill. Rancher 2.6.7 Rancher 2.x	2	362	September 5, 2022
Tutorial : Traefik 2.0 - Baremetal - RancherOS - Hetzner - OVH Rancher 2.x	3	5914	December 18, 2019
Rancher on K3s with HAProxy LB - Backend down, 404 Rancher 2.x	2	360	March 12, 2024
Traefik Internal Server Error Rancher 2.x	0	421	September 20, 2022

Uninstalled traefik, rancher no longer works

Related Topics