So, I did a big no-no… I uninstalled traefik from my cluster, and now trying to reach rancher returns a 404 page not found. I had helm reinstall traefik, no change. I checked the logs for the rancher pod, the only major error I can see that might be the issue is “Error during subscribe write tcp 10.42.0.60:80->10.42.2.138:36430: write: no route to host” I’ve tried googling around, couldn’t find any good tips to try and resolve the situation.
Hi,
Can you provide log from the traefik and rancher pods?
I was busy with personal stuff most of the morning and afternoon, sorry for the delay. I had trouble running the logs command with the pod, only showing one message since it was live for so long. I scaled the deployment to 0 replicas and then to 1 to force it to reload, and I’m not getting the same error message I said in my original post
Here’s the log for the rancher pod:
2022/08/29 00:23:44 [INFO] Rancher version v2.6.7 (e5c6f0f6a) is starting
2022/08/29 00:23:44 [INFO] Listening on /tmp/log.sock
2022/08/29 00:23:44 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
2022/08/29 00:23:44 [INFO] Running in clustered mode with ID 10.42.3.243, monitoring endpoint cattle-system/rancher
2022/08/29 00:23:46 [INFO] Applying CRD features.management.cattle.io
2022/08/29 00:23:51 [INFO] Applying CRD navlinks.ui.cattle.io
2022/08/29 00:23:51 [INFO] Applying CRD clusters.management.cattle.io
2022/08/29 00:23:52 [INFO] Applying CRD apiservices.management.cattle.io
2022/08/29 00:23:52 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
2022/08/29 00:23:53 [INFO] Applying CRD settings.management.cattle.io
2022/08/29 00:23:53 [INFO] Applying CRD preferences.management.cattle.io
2022/08/29 00:23:54 [INFO] Applying CRD features.management.cattle.io
2022/08/29 00:23:54 [INFO] Applying CRD clusterrepos.catalog.cattle.io
2022/08/29 00:23:54 [INFO] Applying CRD operations.catalog.cattle.io
2022/08/29 00:23:55 [INFO] Applying CRD apps.catalog.cattle.io
2022/08/29 00:23:56 [INFO] Applying CRD fleetworkspaces.management.cattle.io
2022/08/29 00:23:56 [INFO] Applying CRD managedcharts.management.cattle.io
2022/08/29 00:23:57 [INFO] Applying CRD clusters.provisioning.cattle.io
2022/08/29 00:23:57 [INFO] Applying CRD clusters.provisioning.cattle.io
2022/08/29 00:23:58 [INFO] Applying CRD rkeclusters.rke.cattle.io
2022/08/29 00:23:58 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2022/08/29 00:23:59 [INFO] Applying CRD rkebootstraps.rke.cattle.io
2022/08/29 00:23:59 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
2022/08/29 00:24:00 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2022/08/29 00:24:00 [INFO] Applying CRD custommachines.rke.cattle.io
2022/08/29 00:24:01 [INFO] Applying CRD etcdsnapshots.rke.cattle.io
2022/08/29 00:24:01 [INFO] Applying CRD clusters.cluster.x-k8s.io
2022/08/29 00:24:02 [INFO] Applying CRD machinedeployments.cluster.x-k8s.io
2022/08/29 00:24:02 [INFO] Applying CRD machinehealthchecks.cluster.x-k8s.io
2022/08/29 00:24:03 [INFO] Applying CRD machines.cluster.x-k8s.io
2022/08/29 00:24:04 [INFO] Applying CRD machinesets.cluster.x-k8s.io
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=GlobalDns controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplate controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=CisConfig controller
I0829 00:24:46.318218 33 leaderelection.go:248] attempting to acquire leader lease kube-system/cattle-controllers...
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=Endpoints controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Preference controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=KontainerDriver controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterAppRevision controller
2022/08/29 00:24:46 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeCredential controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Project controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller
2022/08/29 00:24:46 [INFO] Starting project.cattle.io/v3, Kind=Pipeline controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=Namespace controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=ConfigMap controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ClusterCatalog controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sSystemImage controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ClusterRegistrationToken controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=ServiceAccount controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplate controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2022/08/29 00:24:46 [INFO] Starting project.cattle.io/v3, Kind=App controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=Secret controller
2022/08/29 00:24:46 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrap controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=NodeDriver controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Feature controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=CisBenchmarkVersion controller
2022/08/29 00:24:46 [INFO] Starting API controllers
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Catalog controller
2022/08/29 00:24:46 [INFO] Starting cluster controllers for local
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=NodeTemplate controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterApp controller
2022/08/29 00:24:46 [INFO] Starting apiextensions.k8s.io/v1, Kind=CustomResourceDefinition controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=RkeAddon controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=AuthConfig controller
2022/08/29 00:24:46 [INFO] Starting provisioning.cattle.io/v1, Kind=Cluster controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=GlobalRole controller
2022/08/29 00:24:46 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeRepository controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=DynamicSchema controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ProjectLogging controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplateVersion controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Setting controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=APIService controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=User controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplate controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
2022/08/29 00:24:46 [INFO] Starting project.cattle.io/v3, Kind=PipelineExecution controller
2022/08/29 00:24:46 [INFO] Starting cluster.x-k8s.io/v1beta1, Kind=Machine controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sServiceOption controller
2022/08/29 00:24:46 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
2022/08/29 00:24:46 [INFO] Starting catalog.cattle.io/v1, Kind=ClusterRepo controller
2022/08/29 00:24:46 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=Secret controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=ServiceAccount controller
2022/08/29 00:24:46 [INFO] Starting /v1, Kind=Namespace controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2022/08/29 00:24:46 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2022/08/29 00:24:46 [INFO] Starting cluster agent for local [owner=false]
2022/08/29 00:24:47 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2022/08/29 00:24:47 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2022/08/29 00:24:47 [INFO] Starting management.cattle.io/v3, Kind=User controller
2022/08/29 00:24:47 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2022/08/29 00:24:47 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2022/08/29 00:24:47 [INFO] Starting /v1, Kind=Secret controller
2022/08/29 00:24:47 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2022/08/29 00:24:47 [INFO] Active TLS secret cattle-system/serving-cert (ver=37806133) (count 8): map[field.cattle.io/projectId:local:p-pvcv6 listener.cattle.io/cn-10.42.0.60:10.42.0.60 listener.cattle.io/cn-10.42.3.242:10.42.3.242 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.1.88.sslip.io:192.168.1.88.sslip.io listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=629B6E1EA3957C0E4B3CC941DC81B84BA26D39F1]
2022/08/29 00:24:47 [INFO] Listening on :80
2022/08/29 00:24:47 [INFO] Listening on :443
2022/08/29 00:24:47 [INFO] certificate CN=dynamic,O=dynamic signed by CN=dynamiclistener-ca@1661641448,O=dynamiclistener-org: notBefore=2022-08-27 23:04:08 +0000 UTC notAfter=2023-08-29 00:24:47 +0000 UTC
2022/08/29 00:24:47 [INFO] Updating TLS secret for cattle-system/serving-cert (count: 9): map[field.cattle.io/projectId:local:p-pvcv6 listener.cattle.io/cn-10.42.0.60:10.42.0.60 listener.cattle.io/cn-10.42.3.242:10.42.3.242 listener.cattle.io/cn-10.42.3.243:10.42.3.243 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.1.88.sslip.io:192.168.1.88.sslip.io listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=1EFC00ECE6FBE9E546BFD9688745B1AFD6B7AC16]
2022/08/29 00:24:47 [INFO] Active TLS secret cattle-system/serving-cert (ver=37812352) (count 9): map[field.cattle.io/projectId:local:p-pvcv6 listener.cattle.io/cn-10.42.0.60:10.42.0.60 listener.cattle.io/cn-10.42.3.242:10.42.3.242 listener.cattle.io/cn-10.42.3.243:10.42.3.243 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.1.88.sslip.io:192.168.1.88.sslip.io listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=1EFC00ECE6FBE9E546BFD9688745B1AFD6B7AC16]
2022/08/29 00:24:47 [INFO] Active TLS secret cattle-system/tls-rancher-internal (ver=37067875) (count 2): map[field.cattle.io/projectId:local:p-pvcv6 listener.cattle.io/cn-10.43.234.219:10.43.234.219 listener.cattle.io/fingerprint:SHA1=C32EBD9E64C1A462D0FFADF30BF4448DA9B03619]
2022/08/29 00:24:47 [INFO] Listening on :444
2022/08/29 00:24:48 [INFO] Starting /v1, Kind=Secret controller
2022/08/29 00:24:48 [INFO] Updating TLS secret for cattle-system/tls-rancher-internal (count: 2): map[field.cattle.io/projectId:local:p-pvcv6 listener.cattle.io/cn-10.43.234.219:10.43.234.219 listener.cattle.io/fingerprint:SHA1=C32EBD9E64C1A462D0FFADF30BF4448DA9B03619]
2022/08/29 00:24:48 [INFO] Updating TLS secret for cattle-system/serving-cert (count: 9): map[field.cattle.io/projectId:local:p-pvcv6 listener.cattle.io/cn-10.42.0.60:10.42.0.60 listener.cattle.io/cn-10.42.3.242:10.42.3.242 listener.cattle.io/cn-10.42.3.243:10.42.3.243 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-192.168.1.88.sslip.io:192.168.1.88.sslip.io listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/fingerprint:SHA1=1EFC00ECE6FBE9E546BFD9688745B1AFD6B7AC16]
W0829 00:24:50.824568 33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging
2022/08/29 00:24:51 [INFO] Watching metadata for helm.cattle.io/v1, Kind=HelmChart
2022/08/29 00:24:51 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEBootstrap
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=ServiceAccount
2022/08/29 00:24:51 [INFO] Watching metadata for k8s.nginx.org/v1, Kind=Policy
2022/08/29 00:24:51 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=MachineSet
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ManagedChart
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Node
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Pod
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=ConfigMap
2022/08/29 00:24:51 [INFO] Watching metadata for batch/v1, Kind=CronJob
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember
2022/08/29 00:24:51 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=InstanceManager
2022/08/29 00:24:51 [INFO] Watching metadata for certificates.k8s.io/v1, Kind=CertificateSigningRequest
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=BackupVolume
2022/08/29 00:24:51 [INFO] Watching metadata for cert-manager.io/v1, Kind=Certificate
2022/08/29 00:24:51 [INFO] Watching metadata for node.k8s.io/v1, Kind=RuntimeClass
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepoRestriction
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group
2022/08/29 00:24:51 [INFO] Watching metadata for k3s.cattle.io/v1, Kind=Addon
2022/08/29 00:24:51 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSIStorageCapacity
2022/08/29 00:24:51 [INFO] Watching metadata for kubevirt.io/v1, Kind=KubeVirt
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=IngressRouteUDP
2022/08/29 00:24:51 [INFO] Watching metadata for cert-manager.io/v1, Kind=ClusterIssuer
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ImageScan
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistrationToken
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=ResourceQuota
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterGroup
2022/08/29 00:24:51 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=ReplicationController
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=BackingImage
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=APIService
2022/08/29 00:24:51 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=Machine
2022/08/29 00:24:51 [INFO] Watching metadata for flowcontrol.apiserver.k8s.io/v1beta2, Kind=FlowSchema
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=RancherUserNotification
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Namespace
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=ShareManager
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PodMonitor
2022/08/29 00:24:51 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSIDriver
2022/08/29 00:24:51 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=App
2022/08/29 00:24:51 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEBootstrapTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver
2022/08/29 00:24:51 [INFO] Watching metadata for flowcontrol.apiserver.k8s.io/v1beta2, Kind=PriorityLevelConfiguration
2022/08/29 00:24:51 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=ClusterRepo
2022/08/29 00:24:51 [INFO] Watching metadata for k8s.nginx.org/v1, Kind=VirtualServer
2022/08/29 00:24:51 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Order
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepo
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph
2022/08/29 00:24:51 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=MachineHealthCheck
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
2022/08/29 00:24:51 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=FleetWorkspace
2022/08/29 00:24:51 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=MiddlewareTCP
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Endpoints
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=User
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision
2022/08/29 00:24:51 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
2022/08/29 00:24:51 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Cluster
2022/08/29 00:24:51 [INFO] Watching metadata for cert-manager.io/v1, Kind=CertificateRequest
2022/08/29 00:24:51 [INFO] Watching metadata for ui.cattle.io/v1, Kind=NavLink
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding
2022/08/29 00:24:51 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=Cluster
2022/08/29 00:24:51 [INFO] Watching metadata for events.k8s.io/v1, Kind=Event
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
2022/08/29 00:24:51 [INFO] Watching metadata for batch/v1, Kind=Job
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=TLSStore
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Replica
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager
2022/08/29 00:24:51 [INFO] Watching metadata for provisioning.cattle.io/v1, Kind=Cluster
2022/08/29 00:24:51 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision
2022/08/29 00:24:51 [INFO] Watching metadata for apps/v1, Kind=DaemonSet
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=LimitRange
2022/08/29 00:24:51 [INFO] Watching metadata for apps/v1, Kind=Deployment
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Setting
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy
2022/08/29 00:24:51 [INFO] Watching metadata for k8s.nginx.org/v1, Kind=VirtualServerRoute
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
2022/08/29 00:24:51 [INFO] Watching metadata for rke.cattle.io/v1, Kind=ETCDSnapshot
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Snapshot
2022/08/29 00:24:51 [INFO] Watching metadata for k8s.nginx.org/v1alpha1, Kind=TransportServer
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster
2022/08/29 00:24:51 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=Operation
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=TraefikService
2022/08/29 00:24:51 [INFO] Watching metadata for apps/v1, Kind=StatefulSet
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token
2022/08/29 00:24:51 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease
2022/08/29 00:24:51 [INFO] Watching metadata for appprotectdos.f5.com/v1beta1, Kind=APDosPolicy
2022/08/29 00:24:51 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass
2022/08/29 00:24:51 [INFO] Watching metadata for cluster.x-k8s.io/v1beta1, Kind=MachineDeployment
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Probe
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=ServersTransport
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule
2022/08/29 00:24:51 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Service
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=Middleware
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Node
2022/08/29 00:24:51 [INFO] Watching metadata for appprotect.f5.com/v1beta1, Kind=APPolicy
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent
2022/08/29 00:24:51 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKEControlPlane
2022/08/29 00:24:51 [INFO] Watching metadata for appprotect.f5.com/v1beta1, Kind=APUserSig
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature
2022/08/29 00:24:51 [INFO] Watching metadata for helm.cattle.io/v1, Kind=HelmChartConfig
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Engine
2022/08/29 00:24:51 [INFO] Watching metadata for appprotectdos.f5.com/v1beta1, Kind=DosProtectedResource
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns
2022/08/29 00:24:51 [INFO] Watching metadata for discovery.k8s.io/v1, Kind=EndpointSlice
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=PodTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template
2022/08/29 00:24:51 [INFO] Watching metadata for rke.cattle.io/v1, Kind=CustomMachine
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Orphan
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=BackingImageManager
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ThanosRuler
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1alpha1, Kind=AlertmanagerConfig
2022/08/29 00:24:51 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Backup
2022/08/29 00:24:51 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleDeployment
2022/08/29 00:24:51 [INFO] Watching metadata for policy/v1, Kind=PodDisruptionBudget
2022/08/29 00:24:51 [INFO] Watching metadata for networking.k8s.io/v1, Kind=IngressClass
2022/08/29 00:24:51 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Challenge
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=IngressRouteTCP
2022/08/29 00:24:51 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Content
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=Volume
2022/08/29 00:24:51 [INFO] Watching metadata for gitjob.cattle.io/v1, Kind=GitJob
2022/08/29 00:24:51 [INFO] Watching metadata for networking.k8s.io/v1, Kind=Ingress
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository
2022/08/29 00:24:51 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=BackingImageDataSource
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=RecurringJob
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=PersistentVolume
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption
2022/08/29 00:24:51 [INFO] Watching metadata for rke.cattle.io/v1, Kind=RKECluster
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool
2022/08/29 00:24:51 [INFO] Watching metadata for k8s.nginx.org/v1alpha1, Kind=GlobalConfiguration
2022/08/29 00:24:51 [INFO] Watching metadata for appprotectdos.f5.com/v1beta1, Kind=APDosLogConf
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=EngineImage
2022/08/29 00:24:51 [INFO] Watching metadata for appprotect.f5.com/v1beta1, Kind=APLogConf
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Secret
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=TLSOption
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken
2022/08/29 00:24:51 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=Event
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Bundle
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion
2022/08/29 00:24:51 [INFO] Watching metadata for upgrade.cattle.io/v1, Kind=Plan
2022/08/29 00:24:51 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus
2022/08/29 00:24:51 [INFO] Watching metadata for project.cattle.io/v3, Kind=App
2022/08/29 00:24:51 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role
2022/08/29 00:24:51 [INFO] Watching metadata for autoscaling/v2, Kind=HorizontalPodAutoscaler
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding
2022/08/29 00:24:51 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim
2022/08/29 00:24:51 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleNamespaceMapping
2022/08/29 00:24:51 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup
2022/08/29 00:24:51 [INFO] Watching metadata for cert-manager.io/v1, Kind=Issuer
2022/08/29 00:24:51 [INFO] Watching metadata for traefik.containo.us/v1alpha1, Kind=IngressRoute
2022/08/29 00:24:51 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding
2022/08/29 00:24:51 [INFO] Watching metadata for longhorn.io/v1beta2, Kind=BackupTarget
W0829 00:24:52.056355 33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0829 00:24:52.149133 33 warnings.go:80] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
I0829 00:25:33.735384 33 leaderelection.go:258] successfully acquired lease kube-system/cattle-controllers
2022/08/29 00:25:33 [INFO] Stopping cluster agent for local
2022/08/29 00:25:33 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2022/08/29 00:25:33 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
2022/08/29 00:25:33 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
2022/08/29 00:25:33 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding workers
2022/08/29 00:25:33 [INFO] Shutting down /v1, Kind=Namespace workers
2022/08/29 00:25:33 [INFO] Shutting down /v1, Kind=ServiceAccount workers
2022/08/29 00:25:33 [INFO] Shutting down /v1, Kind=Secret workers
2022/08/29 00:25:33 [INFO] Registering istio for cluster "local"
2022/08/29 00:25:33 [INFO] Starting cluster controllers for local
2022/08/29 00:25:33 [INFO] Reconciling GlobalRoles
2022/08/29 00:25:33 [INFO] Starting management.cattle.io/v3, Kind=ClusterLogging controller
2022/08/29 00:25:33 [INFO] Starting rke.cattle.io/v1, Kind=ETCDSnapshot controller
2022/08/29 00:25:33 [INFO] Starting management.cattle.io/v3, Kind=Notifier controller
2022/08/29 00:25:33 [INFO] Starting management.cattle.io/v3, Kind=ClusterScan controller
2022/08/29 00:25:33 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlert controller
2022/08/29 00:25:33 [INFO] Reconciling RoleTemplates
2022/08/29 00:25:35 [INFO] Steve auth startup complete
2022/08/29 00:25:35 [INFO] Registering namespaceHandler for adding labels
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=ResourceQuota controller
2022/08/29 00:25:36 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=ConfigMap controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=ServiceAccount controller
2022/08/29 00:25:36 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=Namespace controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=LimitRange controller
2022/08/29 00:25:36 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2022/08/29 00:25:36 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2022/08/29 00:25:36 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=Node controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=Secret controller
2022/08/29 00:25:36 [INFO] Starting cluster agent for local [owner=true]
2022/08/29 00:25:36 [INFO] Starting cluster controllers for local
2022/08/29 00:25:36 [INFO] adding kontainer driver rancherKubernetesEngine
2022/08/29 00:25:36 [INFO] adding kontainer driver googleKubernetesEngine
2022/08/29 00:25:36 [INFO] adding kontainer driver azureKubernetesService
2022/08/29 00:25:36 [INFO] Starting fleet.cattle.io/v1alpha1, Kind=Bundle controller
2022/08/29 00:25:36 [INFO] Starting batch/v1, Kind=Job controller
2022/08/29 00:25:36 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrapTemplate controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=ReplicationController controller
2022/08/29 00:25:36 [INFO] Starting batch/v1beta1, Kind=CronJob controller
2022/08/29 00:25:36 [INFO] Starting rke.cattle.io/v1, Kind=CustomMachine controller
2022/08/29 00:25:36 [INFO] adding kontainer driver amazonElasticContainerService
2022/08/29 00:25:36 [INFO] Starting management.cattle.io/v3, Kind=FleetWorkspace controller
2022/08/29 00:25:36 [INFO] Starting networking.k8s.io/v1, Kind=Ingress controller
2022/08/29 00:25:36 [INFO] Starting rke.cattle.io/v1, Kind=RKECluster controller
2022/08/29 00:25:36 [INFO] Starting fleet.cattle.io/v1alpha1, Kind=Cluster controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=Service controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=Node controller
2022/08/29 00:25:36 [INFO] Starting admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=StatefulSet controller
2022/08/29 00:25:36 [INFO] Starting catalog.cattle.io/v1, Kind=Operation controller
2022/08/29 00:25:36 [INFO] Starting management.cattle.io/v3, Kind=ManagedChart controller
2022/08/29 00:25:36 [INFO] Starting cluster.x-k8s.io/v1beta1, Kind=MachineDeployment controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=DaemonSet controller
2022/08/29 00:25:36 [INFO] Starting rke.cattle.io/v1, Kind=RKEControlPlane controller
2022/08/29 00:25:36 [INFO] Starting cluster.x-k8s.io/v1beta1, Kind=Cluster controller
2022/08/29 00:25:36 [INFO] Starting catalog.cattle.io/v1, Kind=App controller
2022/08/29 00:25:36 [INFO] Starting admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=ReplicaSet controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=Deployment controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=Pod controller
2022/08/29 00:25:36 [INFO] adding kontainer driver baiducloudcontainerengine
2022/08/29 00:25:36 [INFO] adding kontainer driver aliyunkubernetescontainerservice
2022/08/29 00:25:36 [INFO] adding kontainer driver tencentkubernetesengine
2022/08/29 00:25:36 [INFO] adding kontainer driver huaweicontainercloudengine
2022/08/29 00:25:36 [INFO] adding kontainer driver oraclecontainerengine
2022/08/29 00:25:36 [INFO] adding kontainer driver linodekubernetesengine
2022/08/29 00:25:36 [INFO] adding kontainer driver opentelekomcloudcontainerengine
2022/08/29 00:25:36 [INFO] Updating node driver cloudscale
2022/08/29 00:25:36 [INFO] Starting batch/v1, Kind=Job controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=ReplicaSet controller
2022/08/29 00:25:36 [INFO] Starting /v1, Kind=ReplicationController controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=StatefulSet controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=Deployment controller
2022/08/29 00:25:36 [INFO] Starting batch/v1beta1, Kind=CronJob controller
2022/08/29 00:25:36 [INFO] Starting apps/v1, Kind=DaemonSet controller
2022/08/29 00:25:36 [INFO] Starting catalog controller
2022/08/29 00:25:36 [INFO] Starting project-level catalog controller
2022/08/29 00:25:36 [INFO] Starting cluster-level catalog controller
2022/08/29 00:25:37 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=Machine
2022/08/29 00:25:37 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2022/08/29 00:25:37 [INFO] Refreshing driverMetadata in 1440 minutes
2022/08/29 00:25:37 [INFO] update kontainerdriver tencentkubernetesengine
2022/08/29 00:25:37 [INFO] update kontainerdriver rancherkubernetesengine
2022/08/29 00:25:37 [INFO] update kontainerdriver oraclecontainerengine
2022/08/29 00:25:37 [INFO] update kontainerdriver aliyunkubernetescontainerservice
2022/08/29 00:25:37 [INFO] update kontainerdriver opentelekomcloudcontainerengine
2022/08/29 00:25:37 [INFO] update kontainerdriver huaweicontainercloudengine
2022/08/29 00:25:37 [INFO] update kontainerdriver amazonelasticcontainerservice
2022/08/29 00:25:37 [INFO] update kontainerdriver azurekubernetesservice
2022/08/29 00:25:37 [INFO] update kontainerdriver baiducloudcontainerengine
2022/08/29 00:25:37 [INFO] update kontainerdriver linodekubernetesengine
2022/08/29 00:25:37 [INFO] update kontainerdriver googlekubernetesengine
2022/08/29 00:25:37 [INFO] Starting management.cattle.io/v3, Kind=EtcdBackup controller
2022/08/29 00:25:37 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertRule controller
2022/08/29 00:25:37 [INFO] Starting management.cattle.io/v3, Kind=ComposeConfig controller
2022/08/29 00:25:37 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertGroup controller
2022/08/29 00:25:37 [INFO] Starting management.cattle.io/v3, Kind=GlobalDnsProvider controller
2022/08/29 00:25:37 [INFO] getReleaseConfig: reloading config for k3s
2022/08/29 00:25:37 [INFO] getReleaseConfig: reloading config for rke2
2022/08/29 00:25:37 [INFO] Rancher startup complete
2022/08/29 00:25:37 [INFO] checking configmap cattle-system/admincreated to determine if duplicate bindings cleanup needs to run
2022/08/29 00:25:37 [INFO] checking configmap cattle-system/admincreated to determine if orphan bindings cleanup needs to run
2022/08/29 00:25:37 [INFO] [CleanupOrphanBindingsDone] orphan bindings cleanup has already run, skipping
2022/08/29 00:25:37 [INFO] checking configmap cattle-system/admincreated to determine if orphan bindings cleanup needs to run
2022/08/29 00:25:37 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=Machine controller
2022/08/29 00:25:37 [INFO] duplicate bindings cleanup has already run, skipping
2022/08/29 00:25:37 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment
2022/08/29 00:25:37 [INFO] [clean-catalog-orphan-bindings] cleaning up orphaned catalog bindings
2022/08/29 00:25:37 [INFO] [clean-catalog-orphan-bindings] Processing 2 rolebindings
2022/08/29 00:25:37 [INFO] [clean-catalog-orphan-bindings] Deleting orphaned role global-catalog
2022/08/29 00:25:37 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=MachineDeployment controller
2022/08/29 00:25:37 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=Cluster
2022/08/29 00:25:37 [WARNING] [clean-catalog-orphan-bindings] Error when deleting role global-catalog, roles.rbac.authorization.k8s.io "global-catalog" not found
2022/08/29 00:25:37 [WARNING] [CleanupOrphanCatalogBindingsDone] error during orphan binding cleanup: roles.rbac.authorization.k8s.io "global-catalog" not found
2022/08/29 00:25:37 [ERROR] failed to cleanup orphan catalog bindings
2022/08/29 00:25:37 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=Cluster controller
2022/08/29 00:25:37 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck
2022/08/29 00:25:37 [INFO] Watching metadata for cluster.x-k8s.io/v1alpha3, Kind=MachineSet
2022/08/29 00:25:37 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=MachineHealthCheck controller
2022/08/29 00:25:37 [INFO] uploading digitaloceanConfig to nodeconfig schema
2022/08/29 00:25:37 [INFO] Starting cluster.x-k8s.io/v1alpha3, Kind=MachineSet controller
2022/08/29 00:25:37 [INFO] Starting provisioning.cattle.io/v1, Kind=Cluster controller
2022/08/29 00:25:37 [INFO] Starting rke.cattle.io/v1, Kind=ETCDSnapshot controller
2022/08/29 00:25:37 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrap controller
2022/08/29 00:25:37 [INFO] Starting rke.cattle.io/v1, Kind=RKECluster controller
2022/08/29 00:25:37 [INFO] Starting rke.cattle.io/v1, Kind=RKEControlPlane controller
2022/08/29 00:25:37 [INFO] Starting rke.cattle.io/v1, Kind=CustomMachine controller
2022/08/29 00:25:37 [INFO] Starting rke.cattle.io/v1, Kind=RKEBootstrapTemplate controller
2022/08/29 00:25:37 [INFO] uploading digitaloceanConfig to nodetemplateconfig schema
2022/08/29 00:25:38 [INFO] uploading digitaloceancredentialConfig to credentialconfig schema
2022/08/29 00:25:38 [INFO] uploading googlecredentialConfig to credentialconfig schema
2022/08/29 00:25:38 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2022/08/29 00:25:38 [INFO] Starting monitoring.coreos.com/v1, Kind=ServiceMonitor controller
2022/08/29 00:25:38 [INFO] Starting monitoring.coreos.com/v1, Kind=Prometheus controller
2022/08/29 00:25:38 [INFO] Starting management.cattle.io/v3, Kind=SamlToken controller
2022/08/29 00:25:40 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json
2022/08/29 00:25:40 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
2022/08/29 00:25:40 [INFO] Refreshing driverMetadata in 1440 minutes
2022/08/29 00:25:40 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig
2022/08/29 00:25:40 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine
2022/08/29 00:25:40 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate
2022/08/29 00:25:41 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json
2022/08/29 00:25:41 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
2022/08/29 00:25:41 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine
2022/08/29 00:25:41 [ERROR] failed to sync cache for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate
2022/08/29 00:25:41 [INFO] Starting rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig controller
2022/08/29 00:25:41 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine: failed to wait for caches to sync
2022/08/29 00:25:41 [ERROR] failed to start controller for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate: failed to wait for caches to sync
E0829 00:25:41.884438 33 gvks.go:69] failed to sync schemas: failed to sync cache for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine
2022/08/29 00:25:41 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine
2022/08/29 00:25:42 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate
2022/08/29 00:25:42 [INFO] Starting rke-machine.cattle.io/v1, Kind=DigitaloceanMachine controller
2022/08/29 00:25:42 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:38321
2022/08/29 00:25:42 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:33845
2022/08/29 00:25:42 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:43789
2022/08/29 00:25:42 [INFO] Creating token for user user-pksfj
2022/08/29 00:25:42 [INFO] kontainerdriver azurekubernetesservice stopped
2022/08/29 00:25:42 [INFO] dynamic schema for kontainerdriver azurekubernetesservice updating
2022/08/29 00:25:42 [INFO] kontainerdriver googlekubernetesengine stopped
2022/08/29 00:25:42 [INFO] dynamic schema for kontainerdriver googlekubernetesengine updating
2022/08/29 00:25:42 [INFO] kontainerdriver amazonelasticcontainerservice stopped
2022/08/29 00:25:42 [INFO] dynamic schema for kontainerdriver amazonelasticcontainerservice updating
2022/08/29 00:25:42 [INFO] Starting rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate controller
time="2022-08-29 00:25:42" level=info msg="Telemetry Client v0.5.20"
time="2022-08-29 00:25:42" level=info msg="Listening on 0.0.0.0:8114"
time="2022-08-29 00:25:42" level=info msg="Collecting anonymous data from https://localhost:443/v3"
2022/08/29 00:25:43 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json in [0x6ea3898]
2022/08/29 00:25:43 [INFO] getReleaseConfig: reloading config for k3s
2022/08/29 00:25:43 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json in [0x6ea3898]
2022/08/29 00:25:43 [INFO] getReleaseConfig: reloading config for rke2
2022/08/29 00:25:47 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachineTemplate
2022/08/29 00:25:47 [INFO] Watching metadata for rke-machine.cattle.io/v1, Kind=DigitaloceanMachine
2022/08/29 00:25:47 [INFO] Watching metadata for rke-machine-config.cattle.io/v1, Kind=DigitaloceanConfig
2022/08/29 00:25:47 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json in [0x6ea3898]
2022/08/29 00:25:47 [INFO] Loaded configuration from https://releases.rancher.com/kontainer-driver-metadata/release-v2.6/data.json in [0x6ea3898]
And here it is for traefik:
time="2022-08-28T03:07:22Z" level=info msg="Configuration loaded from flags."
time="2022-08-29T00:05:58Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" ingress=rancher namespace=cattle-system serviceName=rancher providerName=kubernetes
time="2022-08-29T00:09:52Z" level=error msg="Skipping service: no endpoints found" ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes
time="2022-08-29T00:11:07Z" level=error msg="Skipping service: no endpoints found" servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher
time="2022-08-29T00:23:23Z" level=error msg="Skipping service: no endpoints found" ingress=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" namespace=cattle-system serviceName=rancher providerName=kubernetes
time="2022-08-29T00:23:42Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=rancher namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2022-08-29T00:24:47Z" level=error msg="Skipping service: no endpoints found" namespace=cattle-system serviceName=rancher servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=rancher
If there’s a flag I can use to retrieve older logs, I can try that. I tried searching online, and the best I could find was -p which it looks like the poster I found was misinformed as this tries to retrieve logs of a previous pod/resource.
1 Like
@bramnet from the above rancher logs it looks OK, the previous log, in my opinion not related to your issue.
Traefik logs looking interesting,
can you please:
change traefik log to debug and post your service definition (ingress and service of rancher)
Okay, I think I set up the log to debug, describe says the environment variable for “log” is set to “DEBUG,” which according to traefik’s documentation is how to do it. I had to look up how to set it with yaml since I’m so used to Rancher’s UI.
I googled around for service definition, I think you mean the yaml for the services. I’m still new (half a year) so common terminology might be lost on me. Here we go.
Traefik:
apiVersion: v1
kind: Service
metadata:
annotations:
field.cattle.io/publicEndpoints: '[{"addresses":["192.168.1.88","192.168.1.90","192.168.1.91"],"port":80,"protocol":"TCP","serviceName":"default:traefik","allNodes":false},{"addresses":["192.168.1.88","192.168.1.90","192.168.1.91"],"port":443,"protocol":"TCP","serviceName":"default:traefik","allNodes":false}]'
meta.helm.sh/release-name: traefik
meta.helm.sh/release-namespace: default
creationTimestamp: "2022-08-28T03:07:06Z"
finalizers:
- svccontroller.k3s.cattle.io/daemonset
labels:
app.kubernetes.io/instance: traefik
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: traefik
helm.sh/chart: traefik-10.24.0
name: traefik
namespace: default
resourceVersion: "37185722"
uid: 87ca4a14-7e32-44c3-8161-d0961571d5b6
spec:
allocateLoadBalancerNodePorts: true
clusterIP: 10.43.249.85
clusterIPs:
- 10.43.249.85
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: web
nodePort: 31454
port: 80
protocol: TCP
targetPort: web
- name: websecure
nodePort: 32579
port: 443
protocol: TCP
targetPort: websecure
selector:
app.kubernetes.io/instance: traefik
app.kubernetes.io/name: traefik
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 192.168.1.88
- ip: 192.168.1.90
- ip: 192.168.1.91
And Rancher:
apiVersion: v1
kind: Service
metadata:
annotations:
meta.helm.sh/release-name: rancher
meta.helm.sh/release-namespace: cattle-system
creationTimestamp: "2022-08-27T22:59:02Z"
labels:
app: rancher
app.kubernetes.io/managed-by: Helm
chart: rancher-2.6.7
heritage: Helm
release: rancher
name: rancher
namespace: cattle-system
resourceVersion: "37065670"
uid: 495f20a3-d8a3-496d-8fa2-9a0d26543f75
spec:
clusterIP: 10.43.234.219
clusterIPs:
- 10.43.234.219
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
- name: https-internal
port: 443
protocol: TCP
targetPort: 444
selector:
app: rancher
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
@bramnet sorry for the late replay ,
Can you post the output of above commands :
curl -v rancher:80kubectl get endpoints rancher -n cattle-systemkubectl get svc -n cattle-system
Not a problem.
Curl
* Trying 23.202.231.169:80...
* Connected to rancher (23.202.231.169) port 80 (#0)
> GET / HTTP/1.1
> Host: rancher
> User-Agent: curl/7.74.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Correlation-Id: aaaaaaaa
< Date: Wed, 31 Aug 2022 05:15:55 GMT
< Content-Length: 0
< Connection: close
<
* Closing connection 0
Kubectl get endpoints:
NAME ENDPOINTS AGE
rancher 10.42.3.243:80,10.42.3.243:444 3d6h
kubectl get svc:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rancher ClusterIP 10.43.234.219 <none> 80/TCP,443/TCP 3d6h
webhook-service ClusterIP 10.43.44.210 <none> 443/TCP 3d6h
rancher-webhook ClusterIP 10.43.106.31 <none> 443/TCP 3d6h
@bramnet did you configure IngressRotue or ingress?
I’ll be honest, I have no idea. Is there a command I could run that would make it evident if I did either? I have a vague recollection of trying to configure ingress, but I’m only 20% sure of it.
After doing some more research, I think I figured I have ingress… I think.
I put in “kubectl get ingress -n cattle-system” and got this returned
NAME CLASS HOSTS ADDRESS PORTS AGE
rancher <none> 192.168.1.88.sslip.io 192.168.1.88,192.168.1.90,192.168.1.91 80, 443 4d2h
Not sure if this helps. I also had it describe it, this was the return.
Name: rancher
Labels: app=rancher
app.kubernetes.io/managed-by=Helm
chart=rancher-2.6.7
heritage=Helm
release=rancher
Namespace: cattle-system
Address: 192.168.1.88,192.168.1.90,192.168.1.91
Ingress Class: <none>
Default backend: <default>
TLS:
tls-rancher-ingress terminates 192.168.1.88.sslip.io
Rules:
Host Path Backends
---- ---- --------
192.168.1.88.sslip.io
rancher:80 (10.42.3.243:80)
Annotations: cert-manager.io/issuer: rancher
cert-manager.io/issuer-kind: Issuer
field.cattle.io/publicEndpoints:
[{"addresses":["192.168.1.88","192.168.1.90","192.168.1.91"],"port":443,"protocol":"HTTPS","serviceName":"cattle-system:rancher","ingressN...
meta.helm.sh/release-name: rancher
meta.helm.sh/release-namespace: cattle-system
nginx.ingress.kubernetes.io/proxy-connect-timeout: 30
nginx.ingress.kubernetes.io/proxy-read-timeout: 1800
nginx.ingress.kubernetes.io/proxy-send-timeout: 1800
Events: <none>
When I reinstalled traefik, I reinstalled using helm. If I now need to add rancher to traefik (somehow) in order for this to work, that might be what I’m missing.