Rancher in CrashLoopBackOff cycle

This morning I woke to discover Rancher 2.4.10 on K3 is not available. The K3 cluster seems to be ok, however, all pods in the cattle-system namespace are in a perpetual CrashLoopBackOff cycle:

NAME READY STATUS RESTARTS AGE
cattle-node-agent-crq87 0/1 CrashLoopBackOff 61 4h51m
cattle-node-agent-2mnbp 0/1 CrashLoopBackOff 50 3h52m
rancher-56b887d7d7-zsqc8 0/1 Running 18 75m
rancher-56b887d7d7-dlp6c 0/1 CrashLoopBackOff 18 75m
rancher-56b887d7d7-9rn47 0/1 CrashLoopBackOff 18 75m
cattle-cluster-agent-79467cd4b4-pr69p 0/1 CrashLoopBackOff 11 31m

The cattle node agents and cattle-cluster-agent logs show they are not able to connect to CATTLE_SERVER. This would seem to make sense since none of the rancher pods have a ready status.

The rancher pods themselves have a log that looks like this (and elected leaders continually shifting):

2020/11/30 20:23:03 [INFO] Rancher version v2.4.10 (677ed79a3) is starting

2020/11/30 20:23:03 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:auto Embedded:false HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:}

2020/11/30 20:23:03 [INFO] Listening on /tmp/log.sock

I1130 20:23:03.887028 7 http.go:122] HTTP2 has been explicitly disabled

2020/11/30 20:23:04 [INFO] Starting API controllers

2020/11/30 20:23:04 [INFO] Running in clustered mode with ID 10.42.0.123, monitoring endpoint cattle-system/rancher

2020/11/30 20:23:05 [INFO] Starting API controllers

I1130 20:23:09.951631 7 http.go:122] HTTP2 has been explicitly disabled

I1130 20:23:09.952407 7 http.go:122] HTTP2 has been explicitly disabled

2020/11/30 20:23:09 [INFO] Starting cluster controllers for local

2020/11/30 20:23:10 [INFO] Starting cluster agent for local [owner=false]

2020/11/30 20:23:10 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller

2020/11/30 20:23:10 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller

I1130 20:23:19.533368 7 trace.go:116] Trace[1059014376]: "Reflector ListAndWatch" name:github.com/rancher/norman/controller/generic_controller.go:237 (started: 2020-11-30 20:23:05.07245314 +0000 UTC m=+1.476564592) (total time: 14.460864782s):

Trace[1059014376]: [14.39649303s] [14.39649303s] Objects listed

2020/11/30 20:23:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller

2020/11/30 20:23:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller

I1130 20:23:21.253718 7 leaderelection.go:242] attempting to acquire leader lease kube-system/cattle-controllers...

I1130 20:23:21.265434 7 leaderelection.go:252] successfully acquired lease kube-system/cattle-controllers

2020/11/30 20:23:21 [INFO] Starting catalog controller

2020/11/30 20:23:21 [INFO] Starting project-level catalog controller

2020/11/30 20:23:21 [INFO] Starting cluster-level catalog controller

2020/11/30 20:23:21 [INFO] Starting management controllers

2020/11/30 20:23:21 [INFO] Refreshing all schemas

2020/11/30 20:23:21 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller

2020/11/30 20:23:21 [INFO] Starting apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition controller

2020/11/30 20:23:21 [INFO] Refreshing all schemas

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Binding

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ComponentStatus

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ConfigMap

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Endpoints

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Event

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind LimitRange

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Namespace

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Node

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind PersistentVolumeClaim

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind PersistentVolume

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Pod

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind PodTemplate

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ReplicationController

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ResourceQuota

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Secret

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind ServiceAccount

2020/11/30 20:23:21 [INFO] APIVersion /v1 Kind Service

2020/11/30 20:23:21 [INFO] APIVersion apiregistration.k8s.io/v1 Kind APIService

2020/11/30 20:23:21 [INFO] APIVersion apiregistration.k8s.io/v1beta1 Kind APIService

2020/11/30 20:23:21 [INFO] APIVersion extensions/v1beta1 Kind Ingress

2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind ControllerRevision

2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind DaemonSet

2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind Deployment

2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind ReplicaSet

2020/11/30 20:23:21 [INFO] APIVersion apps/v1 Kind StatefulSet

2020/11/30 20:23:21 [INFO] APIVersion events.k8s.io/v1beta1 Kind Event

2020/11/30 20:23:21 [INFO] APIVersion authentication.k8s.io/v1 Kind TokenReview

2020/11/30 20:23:21 [INFO] APIVersion authentication.k8s.io/v1beta1 Kind TokenReview

2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1 Kind LocalSubjectAccessReview

2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectAccessReview

2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectRulesReview

2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1 Kind SubjectAccessReview

2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind LocalSubjectAccessReview

2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectAccessReview

2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectRulesReview

2020/11/30 20:23:21 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SubjectAccessReview

2020/11/30 20:23:21 [INFO] APIVersion autoscaling/v1 Kind HorizontalPodAutoscaler

2020/11/30 20:23:21 [INFO] APIVersion autoscaling/v2beta1 Kind HorizontalPodAutoscaler

2020/11/30 20:23:21 [INFO] APIVersion autoscaling/v2beta2 Kind HorizontalPodAutoscaler

2020/11/30 20:23:21 [INFO] APIVersion batch/v1 Kind Job

2020/11/30 20:23:21 [INFO] APIVersion batch/v1beta1 Kind CronJob

2020/11/30 20:23:21 [INFO] APIVersion certificates.k8s.io/v1beta1 Kind CertificateSigningRequest

2020/11/30 20:23:21 [INFO] APIVersion networking.k8s.io/v1 Kind NetworkPolicy

2020/11/30 20:23:21 [INFO] APIVersion networking.k8s.io/v1beta1 Kind Ingress

2020/11/30 20:23:21 [INFO] APIVersion policy/v1beta1 Kind PodDisruptionBudget

2020/11/30 20:23:21 [INFO] APIVersion policy/v1beta1 Kind PodSecurityPolicy

2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRoleBinding

2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRole

2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind RoleBinding

2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind Role

2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRoleBinding

2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRole

2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind RoleBinding

2020/11/30 20:23:21 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind Role

2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1 Kind CSINode

2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1 Kind StorageClass

2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1 Kind VolumeAttachment

2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSIDriver

2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSINode

2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1beta1 Kind StorageClass

2020/11/30 20:23:21 [INFO] APIVersion storage.k8s.io/v1beta1 Kind VolumeAttachment

2020/11/30 20:23:21 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind MutatingWebhookConfiguration

2020/11/30 20:23:21 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind ValidatingWebhookConfiguration

2020/11/30 20:23:21 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind MutatingWebhookConfiguration

2020/11/30 20:23:21 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind ValidatingWebhookConfiguration

2020/11/30 20:23:21 [INFO] APIVersion apiextensions.k8s.io/v1 Kind CustomResourceDefinition

2020/11/30 20:23:21 [INFO] APIVersion apiextensions.k8s.io/v1beta1 Kind CustomResourceDefinition

2020/11/30 20:23:21 [INFO] APIVersion scheduling.k8s.io/v1 Kind PriorityClass

2020/11/30 20:23:21 [INFO] APIVersion scheduling.k8s.io/v1beta1 Kind PriorityClass

2020/11/30 20:23:21 [INFO] APIVersion coordination.k8s.io/v1 Kind Lease

2020/11/30 20:23:21 [INFO] APIVersion coordination.k8s.io/v1beta1 Kind Lease

2020/11/30 20:23:21 [INFO] APIVersion node.k8s.io/v1beta1 Kind RuntimeClass

2020/11/30 20:23:21 [INFO] APIVersion discovery.k8s.io/v1beta1 Kind EndpointSlice

2020/11/30 20:23:21 [INFO] APIVersion helm.cattle.io/v1 Kind HelmChart

2020/11/30 20:23:21 [INFO] APIVersion k3s.cattle.io/v1 Kind Addon

2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind PodMonitor

2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind PrometheusRule

2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind Alertmanager

2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind ServiceMonitor

2020/11/30 20:23:21 [INFO] APIVersion monitoring.coreos.com/v1 Kind Prometheus

2020/11/30 20:23:21 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Challenge

2020/11/30 20:23:21 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Order

2020/11/30 20:23:21 [INFO] APIVersion cert-manager.io/v1alpha2 Kind ClusterIssuer

2020/11/30 20:23:21 [INFO] APIVersion cert-manager.io/v1alpha2 Kind CertificateRequest

2020/11/30 20:23:21 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Certificate

2020/11/30 20:23:21 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Issuer

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRole

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterLogging

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind AuthConfig

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplate

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectLogging

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind TemplateContent

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectCatalog

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind NodePool

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind CisConfig

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GroupMember

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterAppRevision

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDnsProvider

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDns

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sServiceOption

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind CisBenchmarkVersion

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplateRevision

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertRule

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlert

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind RoleTemplate

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Token

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind MonitorMetric

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Preference

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Catalog

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertGroup

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind DynamicSchema

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sSystemImage

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind User

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectRoleTemplateBinding

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind RkeAddon

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind TemplateVersion

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind KontainerDriver

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRoleBinding

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplateVersion

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertRule

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectMonitorGraph

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterMonitorGraph

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Feature

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Group

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Template

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Project

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Cluster

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertGroup

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterApp

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Notifier

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRoleTemplateBinding

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind EtcdBackup

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ProjectNetworkPolicy

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind SamlToken

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind NodeTemplate

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplateProjectBinding

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Setting

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplate

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind UserAttribute

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRegistrationToken

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplate

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterScan

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterCatalog

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind NodeDriver

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlert

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind Node

2020/11/30 20:23:21 [INFO] APIVersion management.cattle.io/v3 Kind ComposeConfig

2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution

2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting

2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline

2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository

2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision

2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind App

2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential

2020/11/30 20:23:21 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig

2020/11/30 20:23:21 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics

2020/11/30 20:23:21 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=PersistentVolume

2020/11/30 20:23:24 [INFO] Watching metadata for cert-manager.io/v1alpha2, Kind=ClusterIssuer

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=ServiceAccount

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim

2020/11/30 20:23:24 [INFO] Watching metadata for batch/v1, Kind=Job

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver

2020/11/30 20:23:24 [INFO] Watching metadata for autoscaling/v1, Kind=HorizontalPodAutoscaler

2020/11/30 20:23:24 [INFO] Watching metadata for networking.k8s.io/v1beta1, Kind=Ingress

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding

2020/11/30 20:23:24 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding

2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting

2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=App

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Secret

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Event

2020/11/30 20:23:24 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig

2020/11/30 20:23:24 [INFO] Watching metadata for cert-manager.io/v1alpha2, Kind=Certificate

2020/11/30 20:23:24 [INFO] Watching metadata for cert-manager.io/v1alpha2, Kind=Issuer

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Pod

2020/11/30 20:23:24 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging

2020/11/30 20:23:24 [INFO] Watching metadata for certificates.k8s.io/v1beta1, Kind=CertificateSigningRequest

2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=LimitRange

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Service

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=ResourceQuota

2020/11/30 20:23:24 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration

2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PodMonitor

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding

2020/11/30 20:23:24 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy

2020/11/30 20:23:24 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode

2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=DaemonSet

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog

2020/11/30 20:23:24 [INFO] Watching metadata for helm.cattle.io/v1, Kind=HelmChart

2020/11/30 20:23:24 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=PodTemplate

2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor

2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion

2020/11/30 20:23:24 [INFO] Watching metadata for policy/v1beta1, Kind=PodDisruptionBudget

2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Namespace

2020/11/30 20:23:24 [INFO] Watching metadata for cert-manager.io/v1alpha2, Kind=CertificateRequest

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole

2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema

2020/11/30 20:23:24 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns

2020/11/30 20:23:24 [INFO] Watching metadata for events.k8s.io/v1beta1, Kind=Event

2020/11/30 20:23:24 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute

2020/11/30 20:23:24 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=ReplicationController

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig

2020/11/30 20:23:24 [INFO] Watching metadata for batch/v1beta1, Kind=CronJob

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier

2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Endpoints

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding

2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=StatefulSet

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=User

2020/11/30 20:23:24 [INFO] Watching metadata for discovery.k8s.io/v1beta1, Kind=EndpointSlice

2020/11/30 20:23:24 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment

2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=ConfigMap

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup

2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy

2020/11/30 20:23:24 [INFO] Watching metadata for apps/v1, Kind=Deployment

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp

2020/11/30 20:23:24 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting

2020/11/30 20:23:24 [INFO] Watching metadata for acme.cert-manager.io/v1alpha2, Kind=Order

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature

2020/11/30 20:23:24 [INFO] Watching metadata for acme.cert-manager.io/v1alpha2, Kind=Challenge

2020/11/30 20:23:24 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role

2020/11/30 20:23:24 [INFO] Watching metadata for k3s.cattle.io/v1, Kind=Addon

2020/11/30 20:23:24 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate

2020/11/30 20:23:24 [INFO] Watching metadata for node.k8s.io/v1beta1, Kind=RuntimeClass

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion

2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider

2020/11/30 20:23:24 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon

2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager

2020/11/30 20:23:24 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus

2020/11/30 20:23:24 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project

2020/11/30 20:23:24 [INFO] Watching metadata for /v1, Kind=Node

2020/11/30 20:23:24 [INFO] Watching metadata for storage.k8s.io/v1beta1, Kind=CSIDriver

2020/11/30 20:23:24 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService

I1130 20:23:51.218155 7 trace.go:116] Trace[664750235]: "Reflector ListAndWatch" name:github.com/rancher/norman/controller/generic_controller.go:237 (started: 2020-11-30 20:23:21.341298323 +0000 UTC m=+17.745409776) (total time: 29.876820003s):

Trace[664750235]: [29.82704505s] [29.82704505s] Objects listed

2020/11/30 20:23:51 [INFO] update kontainerdriver rancherkubernetesengine

2020/11/30 20:23:51 [INFO] update kontainerdriver baiducloudcontainerengine

2020/11/30 20:23:51 [INFO] update kontainerdriver aliyunkubernetescontainerservice

2020/11/30 20:23:51 [INFO] update kontainerdriver tencentkubernetesengine

2020/11/30 20:23:51 [INFO] update kontainerdriver huaweicontainercloudengine

2020/11/30 20:23:51 [INFO] update kontainerdriver oraclecontainerengine

2020/11/30 20:23:51 [INFO] update kontainerdriver googlekubernetesengine

2020/11/30 20:23:51 [INFO] update kontainerdriver amazonelasticcontainerservice

2020/11/30 20:23:51 [INFO] update kontainerdriver azurekubernetesservice

2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found

2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:51 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:51 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized

2020/11/30 20:23:51 [INFO] Refreshing driverMetadata in 1440 minutes

2020/11/30 20:23:51 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized

2020/11/30 20:23:51 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax

2020/11/30 20:23:51 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject github_team://2956598 with role project-member in namespace

2020/11/30 20:23:51 [INFO] Reconciling GlobalRoles

2020/11/30 20:23:51 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller

2020/11/30 20:23:51 [INFO] Reconciling RoleTemplates

2020/11/30 20:23:51 [INFO] [mgmt-auth-prtb-controller] Creating roleBinding for subject github_team://2956598 with role project-member in namespace

2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found

2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:52 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:52 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized

2020/11/30 20:23:52 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : refresh daemon not yet initialized

2020/11/30 20:23:52 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.4/data.json

2020/11/30 20:23:52 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json

2020/11/30 20:23:52 [INFO] adding kontainer driver rancherKubernetesEngine

2020/11/30 20:23:52 [INFO] adding kontainer driver googleKubernetesEngine

2020/11/30 20:23:52 [INFO] adding kontainer driver azureKubernetesService

2020/11/30 20:23:52 [INFO] adding kontainer driver amazonElasticContainerService

2020/11/30 20:23:52 [INFO] adding kontainer driver baiducloudcontainerengine

2020/11/30 20:23:52 [INFO] adding kontainer driver aliyunkubernetescontainerservice

2020/11/30 20:23:52 [INFO] adding kontainer driver tencentkubernetesengine

2020/11/30 20:23:52 [INFO] adding kontainer driver huaweicontainercloudengine

2020/11/30 20:23:52 [INFO] adding kontainer driver oraclecontainerengine

2020/11/30 20:23:52 [INFO] Rancher startup complete

2020/11/30 20:23:53 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found

2020/11/30 20:23:53 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found

2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found

2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:53 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:53 [INFO] Refreshing driverMetadata in 1440 minutes

2020/11/30 20:23:53 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.4/data.json

2020/11/30 20:23:53 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json

2020/11/30 20:23:55 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found

2020/11/30 20:23:55 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found

2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found

2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:56 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:44949

2020/11/30 20:23:56 [INFO] kontainerdriver googlekubernetesengine stopped

2020/11/30 20:23:56 [INFO] dynamic schema for kontainerdriver googlekubernetesengine updating

2020/11/30 20:23:56 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:44187

2020/11/30 20:23:56 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:44079

2020/11/30 20:23:56 [INFO] kontainerdriver azurekubernetesservice stopped

2020/11/30 20:23:56 [INFO] kontainerdriver amazonelasticcontainerservice stopped

2020/11/30 20:23:56 [INFO] dynamic schema for kontainerdriver amazonelasticcontainerservice updating

2020/11/30 20:23:56 [INFO] dynamic schema for kontainerdriver azurekubernetesservice updating

time="2020-11-30 20:23:57" level=info msg="Telemetry Client v0.5.12"

time="2020-11-30 20:23:57" level=info msg="Listening on 0.0.0.0:8114"

time="2020-11-30 20:23:57" level=info msg="Collecting anonymous data from https://localhost:443/v3"

time="2020-11-30 20:23:57" level=error msg="Error collecting data: Get https://localhost:443/v3: dial tcp [::1]:443: connect: connection refused"

2020/11/30 20:23:59 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found

2020/11/30 20:23:59 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found

2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found

2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:23:59 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

I1130 20:23:59.619824 7 trace.go:116] Trace[879221466]: "Reflector ListAndWatch" name:github.com/rancher/steve/pkg/clustercache/controller.go:164 (started: 2020-11-30 20:23:24.341763453 +0000 UTC m=+20.745874908) (total time: 35.278020551s):

Trace[879221466]: [35.232208593s] [35.232208593s] Objects listed

2020/11/30 20:24:07 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found

2020/11/30 20:24:07 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found

2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found

2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:07 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:17 [ERROR] ClusterController local exit status 1, The connection to the server localhost:443 was refused - did you specify the right host or port?

2020/11/30 20:24:23 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found

2020/11/30 20:24:23 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found

2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found

2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:23 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:34 [ERROR] ClusterController c-wh54c [cluster-deploy] failed with : Get https://172.20.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent?timeout=45s: waiting for cluster [c-wh54c] agent to connect

2020/11/30 20:24:35 [ERROR] ClusterController c-cv652 [cluster-deploy] failed with : Get https://10.100.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent?timeout=45s: waiting for cluster [c-cv652] agent to connect

2020/11/30 20:24:42 [ERROR] ClusterController local exit status 1, The connection to the server localhost:443 was refused - did you specify the right host or port?

2020/11/30 20:24:51 [INFO] Stopping cluster agent for c-cv652

2020/11/30 20:24:51 [INFO] Stopping cluster agent for c-wh54c

2020/11/30 20:24:55 [ERROR] UserAttributeController u-ixoevjmsq5 [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ixoevjmsq5-secret" not found

2020/11/30 20:24:55 [ERROR] UserAttributeController u-ygauthgjnd [mgmt-auth-userattributes-controller] failed with : secret "cattle-system/u-ygauthgjnd-secret" not found

2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-nz6xz [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/kafka-test" not found

2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-r9nwj [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-glm86 [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-xrbjr [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-t86rm [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:24:55 [ERROR] MultiClusterAppRevisionController cattle-global-data/mcapprevision-f98ql [management-multiclusterapp-revisions-rbac] failed with : multiClusterApp.management.cattle.io "cattle-global-data/postgres-test" not found

2020/11/30 20:25:08 [ERROR] ClusterController local exit status 1, The connection to the server localhost:443 was refused - did you specify the right host or port?

2020/11/30 20:25:21 [ERROR] ClusterController c-cv652 [cluster-deploy] failed with : Get https://10.100.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent?timeout=45s: waiting for cluster [c-cv652] agent to connect

2020/11/30 20:25:21 [ERROR] ClusterController c-wh54c [cluster-deploy] failed with : Get https://172.20.0.1:443/apis/apps/v1/namespaces/cattle-system/daemonsets/cattle-node-agent?timeout=45s: waiting for cluster [c-wh54c] agent to connect

2020/11/30 20:25:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers

2020/11/30 20:25:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers

2020/11/30 20:25:28 [INFO] Shutting down RoleBindingController controller

2020/11/30 20:25:28 [INFO] Shutting down ClusterRoleBindingController controller

2020/11/30 20:25:28 [INFO] Shutting down EndpointsController controller

2020/11/30 20:25:28 [INFO] Shutting down ServiceController controller

2020/11/30 20:25:28 [INFO] Shutting down NamespaceController controller

2020/11/30 20:25:28 [INFO] Shutting down PodController controller

2020/11/30 20:25:28 [INFO] Shutting down APIServiceController controller

2020/11/30 20:25:28 [INFO] Shutting down apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition workers

2020/11/30 20:25:28 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers

2020/11/30 20:25:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers

2020/11/30 20:25:28 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers

2020/11/30 20:25:28 [INFO] Shutting down ClusterRoleController controller

2020/11/30 20:25:28 [INFO] Shutting down SourceCodeRepositoryController controller

2020/11/30 20:25:28 [INFO] Shutting down PipelineExecutionController controller

2020/11/30 20:25:28 [INFO] Shutting down SourceCodeCredentialController controller

2020/11/30 20:25:28 [INFO] Shutting down PipelineController controller

2020/11/30 20:25:28 [INFO] Shutting down AppController controller

2020/11/30 20:25:28 [INFO] Shutting down AuthConfigController controller

2020/11/30 20:25:28 [INFO] Shutting down DynamicSchemaController controller

2020/11/30 20:25:28 [INFO] Shutting down GroupController controller

2020/11/30 20:25:28 [INFO] Shutting down GroupMemberController controller

2020/11/30 20:25:28 [INFO] Shutting down RKEAddonController controller

2020/11/30 20:25:28 [INFO] Shutting down RKEK8sServiceOptionController controller

2020/11/30 20:25:28 [INFO] Shutting down RKEK8sSystemImageController controller

2020/11/30 20:25:28 [INFO] Shutting down GlobalDNSController controller

2020/11/30 20:25:28 [INFO] Shutting down MultiClusterAppRevisionController controller

2020/11/30 20:25:28 [INFO] Shutting down MultiClusterAppController controller

2020/11/30 20:25:28 [INFO] Shutting down PodSecurityPolicyTemplateController controller

2020/11/30 20:25:28 [INFO] Shutting down ProjectLoggingController controller

2020/11/30 20:25:28 [INFO] Shutting down NodeDriverController controller

2020/11/30 20:25:28 [INFO] Shutting down SecretController controller

2020/11/30 20:25:28 [INFO] Shutting down NamespaceController controller

2020/11/30 20:25:28 [INFO] Shutting down SecretController controller

2020/11/30 20:25:28 [INFO] Shutting down SecretController controller

2020/11/30 20:25:28 [INFO] Shutting down EndpointsController controller

2020/11/30 20:25:28 [INFO] Shutting down ClusterRoleBindingController controller

2020/11/30 20:25:28 [INFO] Shutting down CatalogTemplateController controller

2020/11/30 20:25:28 [INFO] Shutting down ProjectController controller

2020/11/30 20:25:28 [INFO] Shutting down NodePoolController controller

2020/11/30 20:25:28 [INFO] Shutting down NodeTemplateController controller

2020/11/30 20:25:28 [INFO] Shutting down SettingController controller

2020/11/30 20:25:28 [INFO] Shutting down ProjectCatalogController controller

2020/11/30 20:25:28 [INFO] Shutting down ClusterCatalogController controller

2020/11/30 20:25:28 [INFO] Shutting down CatalogController controller

2020/11/30 20:25:28 [INFO] Shutting down CatalogTemplateVersionController controller

2020/11/30 20:25:28 [INFO] Shutting down FeatureController controller

2020/11/30 20:25:28 [FATAL] leaderelection lost for cattle-controllers

Any idea what might be causing this?