This morning I woke to discover Rancher 2.4.10 on K3 is not available. The K3 cluster seems to be ok, however, all pods in the cattle-system namespace are in a perpetual CrashLoopBackOff cycle:
NAME READY STATUS RESTARTS AGE
cattle-node-agent-crq87 0/1 CrashLoopBackOff 61 4h51m
cattle-node-agent-2mnbp 0/1 CrashLoopBackOff 50 3h52m
rancher-56b887d7d7-zsqc8 0/1 Running 18 75m
rancher-56b887d7d7-dlp6c 0/1 CrashLoopBackOff 18 75m
rancher-56b887d7d7-9rn47 0/1 CrashLoopBackOff 18 75m
cattle-cluster-agent-79467cd4b4-pr69p 0/1 CrashLoopBackOff 11 31m
The cattle node agents and cattle-cluster-agent logs show they are not able to connect to CATTLE_SERVER. This would seem to make sense since none of the rancher pods have a ready status.
The rancher pods themselves have a log that looks like this (and elected leaders continually shifting):
2020/11/30 20:23:03 [INFO] Rancher version v2.4.10 (677ed79a3) is starting 2020/11/30 20:23:03 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:auto Embedded:false HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:} 2020/11/30 20:23:03 [INFO] Listening on /tmp/log.sock I1130 20:23:03.887028 7 http.go:122] HTTP2 has been explicitly disabled 2020/11/30 20:23:04 [INFO] Starting API controllers 2020/11/30 20:23:04 [INFO] Running in clustered mode with ID 10.42.0.123, monitoring endpoint cattle-system/rancher 2020/11/30 20:23:05 [INFO] Starting API controllers I1130 20:23:09.951631 7 http.go:122] HTTP2 has been explicitly disabled I1130 20:23:09.952407 7 http.go:122] HTTP2 has been explicitly disabled 2020/11/30 20:23:09 [INFO] Starting cluster controllers for local 2020/11/30 20:23:10 [INFO] Starting cluster agent for local [owner=false] 2020/11/30 20:23:10 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller 2020/11/30 20:23:10 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller I1130 20:23:19.533368 7 trace.go:116] Trace[1059014376]: "Reflector ListAndWatch" name:github.com/rancher/norman/controller/generic_controller.go:237 (started: 2020-11-30 20:23:05.07245314 +0000 UTC m=+1.476564592) (total time: 14.460864782s): Trace[1059014376]: [14.39649303s] [14.39649303s] Objects listed 2020/11/30 20:23:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller 2020/11/30 20:23:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller I1130 20:23:21.253718 7 leaderelection.go:242] attempting to acquire leader lease kube-system/cattle-controllers... I1130 20:23:21.265434 7 leaderelection.go:252] successfully acquired lease kube-system/cattle-controllers 2020/11/30 20:23:21 [INFO] Starting catalog controller 2020/11/30 20:23:21 [INFO] Starting project-level catalog controller 2020/11/30 20:23:21 [INFO] Starting cluster-level catalog controller 2020/11/30 20:23:21 [INFO] Starting management controllers 2020/11/30 20:23:21 [INFO] Refreshing all schemas 2020/11/30 20:23:21 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller 2020/11/30 20:23:21 [INFO] Starting apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition controller 2020/11/30 20:23:21 [INFO] Refreshing all schemas 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Binding 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ComponentStatus 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ConfigMap 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Endpoints 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Event 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind LimitRange 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Namespace 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Node 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind PersistentVolumeClaim 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind PersistentVolume 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Pod 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind PodTemplate 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ReplicationController 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ResourceQuota 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Secret 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ServiceAccount 2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Service 2020/11/30 20:23:21 [INFO] APIVersion apiregistration.k8s.io/v1 Kind APIService 2020/11/30 20:23:21 [INFO] APIVersion apiregistration.k8s.io/v1beta1 Kind APIService 2020/11/30 20:23:21 [INFO] APIVersion extensions/v1beta1 Kind Ingress 2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind ControllerRevision 2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind DaemonSet 2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind Deployment 2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind ReplicaSet 2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind StatefulSet 2020/11/30 20:23:21 [INFO] APIVersion events.k8s.io/v1beta1 Kind Event 2020/11/30 20:23:21 [INFO] APIVersion authentication.k8s.io/v1 Kind TokenReview 2020/11/30 20:23:21 [INFO] APIVersion authentication.k8s.io/v1beta1 Kind TokenReview 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1 Kind LocalSubjectAccessReview 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectAccessReview 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectRulesReview 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1 Kind SubjectAccessReview 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind LocalSubjectAccessReview 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectAccessReview 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectRulesReview 2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SubjectAccessReview 2020/11/30 20:23:21 [INFO] APIVersion autoscaling/v1 Kind HorizontalPodAutoscaler 2020/11/30 20:23:21 [INFO] APIVersion autoscaling/v2beta1 Kind HorizontalPodAutoscaler 2020/11/30 20:23:21 [INFO] APIVersion autoscaling/v2beta2 Kind HorizontalPodAutoscaler 2020/11/30 20:23:21 [INFO] APIVersion batch/v1 Kind Job 2020/11/30 20:23:21 [INFO] APIVersion batch/v1beta1 Kind CronJob 2020/11/30 20:23:21 [INFO] APIVersion certificates.k8s.io/v1beta1 Kind CertificateSigningRequest 2020/11/30 20:23:21 [INFO] APIVersion networking.k8s.io/v1 Kind NetworkPolicy 2020/11/30 20:23:21 [INFO] APIVersion networking.k8s.io/v1beta1 Kind Ingress 2020/11/30 20:23:21 [INFO] APIVersion policy/v1beta1 Kind PodDisruptionBudget 2020/11/30 20:23:21 [INFO] APIVersion policy/v1beta1 Kind PodSecurityPolicy 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRoleBinding 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRole 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind RoleBinding 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind Role 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRoleBinding 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRole 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind RoleBinding 2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind Role 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1 Kind CSINode 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1 Kind StorageClass 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1 Kind VolumeAttachment 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSIDriver 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSINode 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1beta1 Kind StorageClass 2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1beta1 Kind VolumeAttachment 2020/11/30 20:23:21 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind MutatingWebhookConfiguration 2020/11/30 20:23:21 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind ValidatingWebhookConfiguration 2020/11/30 20:23:21 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind MutatingWebhookConfiguration 2020/11/30 20:23:21 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind ValidatingWebhookConfiguration 2020/11/30 20:23:21 [INFO] APIVersion apiextensions.k8s.io/v1 Kind CustomResourceDefinition 2020/11/30 20:23:21 [INFO] APIVersion apiextensions.k8s.io/v1beta1 Kind CustomResourceDefinition 2020/11/30 20:23:21 [INFO] APIVersion scheduling.k8s.io/v1 Kind PriorityClass 2020/11/30 20:23:21 [INFO] APIVersion scheduling.k8s.io/v1beta1 Kind PriorityClass 2020/11/30 20:23:21 [INFO] APIVersion coordination.k8s.io/v1 Kind Lease 2020/11/30 20:23:21 [INFO] APIVersion coordination.k8s.io/v1beta1 Kind Lease 2020/11/30 20:23:21 [INFO] APIVersion node.k8s.io/v1beta1 Kind RuntimeClass 2020/11/30 20:23:21 [INFO] APIVersion discovery.k8s.io/v1beta1 Kind EndpointSlice 2020/11/30 20:23:21 [INFO] APIVersion helm.cattle.io/v1 Kind HelmChart 2020/11/30 20:23:21 [INFO] APIVersion k3s.cattle.io/v1 Kind Addon 2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind PodMonitor 2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind PrometheusRule 2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind Alertmanager 2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind ServiceMonitor 2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind Prometheus 2020/11/30 20:23:21 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Challenge 2020/11/30 20:23:21 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Order 2020/11/30 20:23:21 [INFO] APIVersion cert-manager.io/v1alpha2 Kind ClusterIssuer 2020/11/30 20:23:21 [INFO] APIVersion cert-manager.io/v1alpha2 Kind CertificateRequest 2020/11/30 20:23:21 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Certificate 2020/11/30 20:23:21 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Issuer 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRole 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterLogging 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind AuthConfig 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplate 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectLogging 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind TemplateContent 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectCatalog 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind NodePool 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind CisConfig 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GroupMember 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterAppRevision 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDnsProvider 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDns 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sServiceOption 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind CisBenchmarkVersion 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplateRevision 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertRule 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlert 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind RoleTemplate 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Token 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind MonitorMetric 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Preference 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Catalog 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertGroup 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind DynamicSchema 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sSystemImage 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind User 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectRoleTemplateBinding 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind RkeAddon 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind TemplateVersion 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind KontainerDriver 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRoleBinding 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplateVersion 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertRule 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectMonitorGraph 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterMonitorGraph 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Feature 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Group 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Template 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Project 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Cluster 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertGroup 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterApp 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Notifier 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRoleTemplateBinding 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind EtcdBackup 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectNetworkPolicy 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind SamlToken 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind NodeTemplate 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplateProjectBinding 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Setting 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplate 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind UserAttribute 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRegistrationToken 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplate 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterScan 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterCatalog 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind NodeDriver 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlert 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Node 2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ComposeConfig 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind App 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential 2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig 2020/11/30 20:23:21 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics 2020/11/30 20:23:21 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=PersistentVolume 2020/11/30 20:23:24 [INFO] Watching metadata for cert-manager.io/v1alpha2, Kind=ClusterIssuer 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=ServiceAccount 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim 2020/11/30 20:23:24 [INFO] Watching metadata for batch/v1, Kind=Job 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver 2020/11/30 20:23:24 [INFO] Watching metadata for autoscaling/v1, Kind=HorizontalPodAutoscaler 2020/11/30 20:23:24 [INFO] Watching metadata for networking.k8s.io/v1beta1, Kind=Ingress 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding 2020/11/30 20:23:24 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=App 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Secret 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Event 2020/11/30 20:23:24 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig 2020/11/30 20:23:24 [INFO] Watching metadata for cert-manager.io/v1alpha2, Kind=Certificate 2020/11/30 20:23:24 [INFO] Watching metadata for cert-manager.io/v1alpha2, Kind=Issuer 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Pod 2020/11/30 20:23:24 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging 2020/11/30 20:23:24 [INFO] Watching metadata for certificates.k8s.io/v1beta1, Kind=CertificateSigningRequest 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=LimitRange 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Service 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=ResourceQuota 2020/11/30 20:23:24 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration 2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PodMonitor 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding 2020/11/30 20:23:24 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy 2020/11/30 20:23:24 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode 2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=DaemonSet 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog 2020/11/30 20:23:24 [INFO] Watching metadata for helm.cattle.io/v1, Kind=HelmChart 2020/11/30 20:23:24 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=PodTemplate 2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion 2020/11/30 20:23:24 [INFO] Watching metadata for policy/v1beta1, Kind=PodDisruptionBudget 2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Namespace 2020/11/30 20:23:24 [INFO] Watching metadata for cert-manager.io/v1alpha2, Kind=CertificateRequest 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole 2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema 2020/11/30 20:23:24 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns 2020/11/30 20:23:24 [INFO] Watching metadata for events.k8s.io/v1beta1, Kind=Event 2020/11/30 20:23:24 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute 2020/11/30 20:23:24 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=ReplicationController 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig 2020/11/30 20:23:24 [INFO] Watching metadata for batch/v1beta1, Kind=CronJob 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Endpoints 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding 2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=StatefulSet 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=User 2020/11/30 20:23:24 [INFO] Watching metadata for discovery.k8s.io/v1beta1, Kind=EndpointSlice 2020/11/30 20:23:24 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=ConfigMap 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup 2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy 2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=Deployment 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp 2020/11/30 20:23:24 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting 2020/11/30 20:23:24 [INFO] Watching metadata for acme.cert-manager.io/v1alpha2, Kind=Order 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature 2020/11/30 20:23:24 [INFO] Watching metadata for acme.cert-manager.io/v1alpha2, Kind=Challenge 2020/11/30 20:23:24 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role 2020/11/30 20:23:24 [INFO] Watching metadata for k3s.cattle.io/v1, Kind=Addon 2020/11/30 20:23:24 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate 2020/11/30 20:23:24 [INFO] Watching metadata for node.k8s.io/v1beta1, Kind=RuntimeClass 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider 2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon 2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager 2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus 2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project 2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Node 2020/11/30 20:23:24 [INFO] Watching metadata for storage.k8s.io/v1beta1, Kind=CSIDriver 2020/11/30 20:23:24 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService I1130 20:23:51.218155 7 trace.go:116] Trace[664750235]: "Reflector ListAndWatch" name:github.com/rancher/norman/controller/generic_controller.go:237 (started: 2020-11-30 20:23:21.341298323 +0000 UTC m=+17.745409776) (total time: 29.876820003s): Trace[664750235]: [29.82704505s] [29.82704505s] Objects listed 2020/11/30 20:23:51 [INFO] update kontainerdriver rancherkubernetesengine 2020/11/30 20:23:51 [INFO] update kontainerdriver baiducloudcontainerengine 2020/11/30 20:23:51 [INFO] update kontainerdriver aliyunkubernetescontainerservice 2020/11/30 20:23:51 [INFO] update kontainerdriver tencentkubernetesengine 2020/11/30 20:23:51 [INFO] update kontainerdriver huaweicontainercloudengine 2020/11/30 20:23:51 [INFO] update kontainerdriver oraclecontainerengine 2020/11/30 20:23:51 [INFO] update kontainerdriver googlekubernetesengine 2020/11/30 20:23:51 [INFO] update kontainerdriver amazonelasticcontainerservice 2020/11/30 20:23:51 [INFO] update kontainerdriver azurekubernetesservice 2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found 2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:51 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized 2020/11/30 20:23:51 [INFO] Refreshing driverMetadata in 1440 minutes 2020/11/30 20:23:51 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized 2020/11/30 20:23:51 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax 2020/11/30 20:23:51 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject github_team://2956598 with role project-member in namespace 2020/11/30 20:23:51 [INFO] Reconciling GlobalRoles 2020/11/30 20:23:51 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller 2020/11/30 20:23:51 [INFO] Reconciling RoleTemplates 2020/11/30 20:23:51 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject github_team://2956598 with role project-member in namespace 2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found 2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:52 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized 2020/11/30 20:23:52 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized 2020/11/30 20:23:52 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.4/data.json 2020/11/30 20:23:52 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json 2020/11/30 20:23:52 [INFO] adding kontainer driver rancherKubernetesEngine 2020/11/30 20:23:52 [INFO] adding kontainer driver googleKubernetesEngine 2020/11/30 20:23:52 [INFO] adding kontainer driver azureKubernetesService 2020/11/30 20:23:52 [INFO] adding kontainer driver amazonElasticContainerService 2020/11/30 20:23:52 [INFO] adding kontainer driver baiducloudcontainerengine 2020/11/30 20:23:52 [INFO] adding kontainer driver aliyunkubernetescontainerservice 2020/11/30 20:23:52 [INFO] adding kontainer driver tencentkubernetesengine 2020/11/30 20:23:52 [INFO] adding kontainer driver huaweicontainercloudengine 2020/11/30 20:23:52 [INFO] adding kontainer driver oraclecontainerengine 2020/11/30 20:23:52 [INFO] Rancher startup complete 2020/11/30 20:23:53 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found 2020/11/30 20:23:53 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found 2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found 2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:53 [INFO] Refreshing driverMetadata in 1440 minutes 2020/11/30 20:23:53 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.4/data.json 2020/11/30 20:23:53 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json 2020/11/30 20:23:55 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found 2020/11/30 20:23:55 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found 2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found 2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:56 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:44949 2020/11/30 20:23:56 [INFO] kontainerdriver googlekubernetesengine stopped 2020/11/30 20:23:56 [INFO] dynamic schema for kontainerdriver googlekubernetesengine updating 2020/11/30 20:23:56 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:44187 2020/11/30 20:23:56 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:44079 2020/11/30 20:23:56 [INFO] kontainerdriver azurekubernetesservice stopped 2020/11/30 20:23:56 [INFO] kontainerdriver amazonelasticcontainerservice stopped 2020/11/30 20:23:56 [INFO] dynamic schema for kontainerdriver amazonelasticcontainerservice updating 2020/11/30 20:23:56 [INFO] dynamic schema for kontainerdriver azurekubernetesservice updating time="2020-11-30 20:23:57" level=info msg="Telemetry Client v0.5.12" time="2020-11-30 20:23:57" level=info msg="Listening on 0.0.0.0:8114" time="2020-11-30 20:23:57" level=info msg="Collecting anonymous data from https://localhost:443/v3" time="2020-11-30 20:23:57" level=error msg="Error collecting data: Get https://localhost:443/v3: dial tcp [::1]:443: connect: connection refused" 2020/11/30 20:23:59 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found 2020/11/30 20:23:59 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found 2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found 2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found I1130 20:23:59.619824 7 trace.go:116] Trace[879221466]: "Reflector ListAndWatch" name:github.com/rancher/steve/pkg/clustercache/controller.go:164 (started: 2020-11-30 20:23:24.341763453 +0000 UTC m=+20.745874908) (total time: 35.278020551s): Trace[879221466]: [35.232208593s] [35.232208593s] Objects listed 2020/11/30 20:24:07 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found 2020/11/30 20:24:07 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found 2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found 2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:17 [ERROR] ClusterController local exit status 1, The connection to the server localhost:443 was refused - did you specify the right host or port? 2020/11/30 20:24:23 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found 2020/11/30 20:24:23 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found 2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found 2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:34 [ERROR] ClusterController c-wh54c [cluster-deploy] failed with : Get https://172.20.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent?timeout=45s: waiting for cluster [c-wh54c] agent to connect 2020/11/30 20:24:35 [ERROR] ClusterController c-cv652 [cluster-deploy] failed with : Get https://10.100.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent?timeout=45s: waiting for cluster [c-cv652] agent to connect 2020/11/30 20:24:42 [ERROR] ClusterController local exit status 1, The connection to the server localhost:443 was refused - did you specify the right host or port? 2020/11/30 20:24:51 [INFO] Stopping cluster agent for c-cv652 2020/11/30 20:24:51 [INFO] Stopping cluster agent for c-wh54c 2020/11/30 20:24:55 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found 2020/11/30 20:24:55 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found 2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found 2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found 2020/11/30 20:25:08 [ERROR] ClusterController local exit status 1, The connection to the server localhost:443 was refused - did you specify the right host or port? 2020/11/30 20:25:21 [ERROR] ClusterController c-cv652 [cluster-deploy] failed with : Get https://10.100.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent?timeout=45s: waiting for cluster [c-cv652] agent to connect 2020/11/30 20:25:21 [ERROR] ClusterController c-wh54c [cluster-deploy] failed with : Get https://172.20.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent?timeout=45s: waiting for cluster [c-wh54c] agent to connect 2020/11/30 20:25:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers 2020/11/30 20:25:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers 2020/11/30 20:25:28 [INFO] Shutting down RoleBindingController controller 2020/11/30 20:25:28 [INFO] Shutting down ClusterRoleBindingController controller 2020/11/30 20:25:28 [INFO] Shutting down EndpointsController controller 2020/11/30 20:25:28 [INFO] Shutting down ServiceController controller 2020/11/30 20:25:28 [INFO] Shutting down NamespaceController controller 2020/11/30 20:25:28 [INFO] Shutting down PodController controller 2020/11/30 20:25:28 [INFO] Shutting down APIServiceController controller 2020/11/30 20:25:28 [INFO] Shutting down apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition workers 2020/11/30 20:25:28 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers 2020/11/30 20:25:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers 2020/11/30 20:25:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers 2020/11/30 20:25:28 [INFO] Shutting down ClusterRoleController controller 2020/11/30 20:25:28 [INFO] Shutting down SourceCodeRepositoryController controller 2020/11/30 20:25:28 [INFO] Shutting down PipelineExecutionController controller 2020/11/30 20:25:28 [INFO] Shutting down SourceCodeCredentialController controller 2020/11/30 20:25:28 [INFO] Shutting down PipelineController controller 2020/11/30 20:25:28 [INFO] Shutting down AppController controller 2020/11/30 20:25:28 [INFO] Shutting down AuthConfigController controller 2020/11/30 20:25:28 [INFO] Shutting down DynamicSchemaController controller 2020/11/30 20:25:28 [INFO] Shutting down GroupController controller 2020/11/30 20:25:28 [INFO] Shutting down GroupMemberController controller 2020/11/30 20:25:28 [INFO] Shutting down RKEAddonController controller 2020/11/30 20:25:28 [INFO] Shutting down RKEK8sServiceOptionController controller 2020/11/30 20:25:28 [INFO] Shutting down RKEK8sSystemImageController controller 2020/11/30 20:25:28 [INFO] Shutting down GlobalDNSController controller 2020/11/30 20:25:28 [INFO] Shutting down MultiClusterAppRevisionController controller 2020/11/30 20:25:28 [INFO] Shutting down MultiClusterAppController controller 2020/11/30 20:25:28 [INFO] Shutting down PodSecurityPolicyTemplateController controller 2020/11/30 20:25:28 [INFO] Shutting down ProjectLoggingController controller 2020/11/30 20:25:28 [INFO] Shutting down NodeDriverController controller 2020/11/30 20:25:28 [INFO] Shutting down SecretController controller 2020/11/30 20:25:28 [INFO] Shutting down NamespaceController controller 2020/11/30 20:25:28 [INFO] Shutting down SecretController controller 2020/11/30 20:25:28 [INFO] Shutting down SecretController controller 2020/11/30 20:25:28 [INFO] Shutting down EndpointsController controller 2020/11/30 20:25:28 [INFO] Shutting down ClusterRoleBindingController controller 2020/11/30 20:25:28 [INFO] Shutting down CatalogTemplateController controller 2020/11/30 20:25:28 [INFO] Shutting down ProjectController controller 2020/11/30 20:25:28 [INFO] Shutting down NodePoolController controller 2020/11/30 20:25:28 [INFO] Shutting down NodeTemplateController controller 2020/11/30 20:25:28 [INFO] Shutting down SettingController controller 2020/11/30 20:25:28 [INFO] Shutting down ProjectCatalogController controller 2020/11/30 20:25:28 [INFO] Shutting down ClusterCatalogController controller 2020/11/30 20:25:28 [INFO] Shutting down CatalogController controller 2020/11/30 20:25:28 [INFO] Shutting down CatalogTemplateVersionController controller 2020/11/30 20:25:28 [INFO] Shutting down FeatureController controller 2020/11/30 20:25:28 [FATAL] leaderelection lost for cattle-controllers
Any idea what might be causing this?