Hi we are facing problems with rancher 2.4.5.
We had the rancher application stuck with crashloopbackoff after a few hours running and get back healthy after hours of downtime.
This behaviour starts 3 weeks ago without any previus occurence for almost 400days of cluster operation.
I appreciate any tips or a way to fix that.
Thanks all.
We’ve got this log of rancher (cattle-system).
2021/01/05 17:45:19 [INFO] Rancher version v2.4.5 (c642f1caa) is starting
2021/01/05 17:45:19 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:auto Embedded:false HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:}
2021/01/05 17:45:19 [INFO] Listening on /tmp/log.sock
I0105 17:45:19.704113 8 http.go:116] HTTP2 has been explicitly disabled
2021/01/05 17:45:19 [INFO] Starting API controllers
2021/01/05 17:45:20 [INFO] Running in clustered mode with ID 10.42.2.99, monitoring endpoint cattle-system/rancher
2021/01/05 17:45:20 [INFO] Starting API controllers
I0105 17:45:21.214921 8 http.go:116] HTTP2 has been explicitly disabled
I0105 17:45:21.215542 8 http.go:116] HTTP2 has been explicitly disabled
I0105 17:45:21.216121 8 http.go:116] HTTP2 has been explicitly disabled
I0105 17:45:21.218586 8 http.go:116] HTTP2 has been explicitly disabled
2021/01/05 17:45:21 [INFO] Starting cluster controllers for local
2021/01/05 17:45:21 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/01/05 17:45:21 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/01/05 17:45:21 [INFO] Starting cluster agent for local [owner=false]
2021/01/05 17:45:21 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/01/05 17:45:21 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
I0105 17:45:22.509559 8 leaderelection.go:242] attempting to acquire leader lease kube-system/cattle-controllers...
2021/01/05 17:45:22 [INFO] Refreshing all schemas
2021/01/05 17:45:22 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/01/05 17:45:22 [INFO] Starting apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition controller
2021/01/05 17:45:22 [INFO] Refreshing all schemas
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind Binding
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind ComponentStatus
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind ConfigMap
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind Endpoints
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind Event
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind LimitRange
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind Namespace
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind Node
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind PersistentVolumeClaim
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind PersistentVolume
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind Pod
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind PodTemplate
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind ReplicationController
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind ResourceQuota
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind Secret
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind ServiceAccount
2021/01/05 17:45:22 [INFO] APIVersion /v1 Kind Service
2021/01/05 17:45:22 [INFO] APIVersion apiregistration.k8s.io/v1 Kind APIService
2021/01/05 17:45:22 [INFO] APIVersion apiregistration.k8s.io/v1beta1 Kind APIService
2021/01/05 17:45:22 [INFO] APIVersion extensions/v1beta1 Kind DaemonSet
2021/01/05 17:45:22 [INFO] APIVersion extensions/v1beta1 Kind Deployment
2021/01/05 17:45:22 [INFO] APIVersion extensions/v1beta1 Kind Ingress
2021/01/05 17:45:22 [INFO] APIVersion extensions/v1beta1 Kind NetworkPolicy
2021/01/05 17:45:22 [INFO] APIVersion extensions/v1beta1 Kind PodSecurityPolicy
2021/01/05 17:45:22 [INFO] APIVersion extensions/v1beta1 Kind ReplicaSet
2021/01/05 17:45:22 [INFO] APIVersion extensions/v1beta1 Kind ReplicationControllerDummy
2021/01/05 17:45:22 [INFO] APIVersion apps/v1 Kind ControllerRevision
2021/01/05 17:45:22 [INFO] APIVersion apps/v1 Kind DaemonSet
2021/01/05 17:45:22 [INFO] APIVersion apps/v1 Kind Deployment
2021/01/05 17:45:22 [INFO] APIVersion apps/v1 Kind ReplicaSet
2021/01/05 17:45:22 [INFO] APIVersion apps/v1 Kind StatefulSet
2021/01/05 17:45:22 [INFO] APIVersion apps/v1beta2 Kind ControllerRevision
2021/01/05 17:45:22 [INFO] APIVersion apps/v1beta2 Kind DaemonSet
2021/01/05 17:45:22 [INFO] APIVersion apps/v1beta2 Kind Deployment
2021/01/05 17:45:22 [INFO] APIVersion apps/v1beta2 Kind ReplicaSet
2021/01/05 17:45:22 [INFO] APIVersion apps/v1beta2 Kind StatefulSet
2021/01/05 17:45:22 [INFO] APIVersion apps/v1beta1 Kind ControllerRevision
2021/01/05 17:45:22 [INFO] APIVersion apps/v1beta1 Kind Deployment
2021/01/05 17:45:22 [INFO] APIVersion apps/v1beta1 Kind StatefulSet
2021/01/05 17:45:22 [INFO] APIVersion events.k8s.io/v1beta1 Kind Event
2021/01/05 17:45:22 [INFO] APIVersion authentication.k8s.io/v1 Kind TokenReview
2021/01/05 17:45:22 [INFO] APIVersion authentication.k8s.io/v1beta1 Kind TokenReview
2021/01/05 17:45:22 [INFO] APIVersion authorization.k8s.io/v1 Kind LocalSubjectAccessReview
2021/01/05 17:45:22 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectAccessReview
2021/01/05 17:45:22 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectRulesReview
2021/01/05 17:45:22 [INFO] APIVersion authorization.k8s.io/v1 Kind SubjectAccessReview
2021/01/05 17:45:22 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind LocalSubjectAccessReview
2021/01/05 17:45:22 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectAccessReview
2021/01/05 17:45:22 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectRulesReview
2021/01/05 17:45:22 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SubjectAccessReview
2021/01/05 17:45:22 [INFO] APIVersion autoscaling/v1 Kind HorizontalPodAutoscaler
2021/01/05 17:45:22 [INFO] APIVersion autoscaling/v2beta1 Kind HorizontalPodAutoscaler
2021/01/05 17:45:22 [INFO] APIVersion autoscaling/v2beta2 Kind HorizontalPodAutoscaler
2021/01/05 17:45:22 [INFO] APIVersion batch/v1 Kind Job
2021/01/05 17:45:22 [INFO] APIVersion batch/v1beta1 Kind CronJob
2021/01/05 17:45:22 [INFO] APIVersion certificates.k8s.io/v1beta1 Kind CertificateSigningRequest
2021/01/05 17:45:22 [INFO] APIVersion networking.k8s.io/v1 Kind NetworkPolicy
2021/01/05 17:45:22 [INFO] APIVersion networking.k8s.io/v1beta1 Kind Ingress
2021/01/05 17:45:22 [INFO] APIVersion policy/v1beta1 Kind PodDisruptionBudget
2021/01/05 17:45:22 [INFO] APIVersion policy/v1beta1 Kind PodSecurityPolicy
2021/01/05 17:45:22 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRoleBinding
2021/01/05 17:45:22 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRole
2021/01/05 17:45:22 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind RoleBinding
2021/01/05 17:45:22 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind Role
2021/01/05 17:45:22 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRoleBinding
2021/01/05 17:45:22 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRole
2021/01/05 17:45:22 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind RoleBinding
2021/01/05 17:45:22 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind Role
2021/01/05 17:45:22 [INFO] APIVersion storage.k8s.io/v1 Kind StorageClass
2021/01/05 17:45:22 [INFO] APIVersion storage.k8s.io/v1 Kind VolumeAttachment
2021/01/05 17:45:22 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSIDriver
2021/01/05 17:45:22 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSINode
2021/01/05 17:45:22 [INFO] APIVersion storage.k8s.io/v1beta1 Kind StorageClass
2021/01/05 17:45:22 [INFO] APIVersion storage.k8s.io/v1beta1 Kind VolumeAttachment
2021/01/05 17:45:22 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind MutatingWebhookConfiguration
2021/01/05 17:45:22 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind ValidatingWebhookConfiguration
2021/01/05 17:45:22 [INFO] APIVersion apiextensions.k8s.io/v1beta1 Kind CustomResourceDefinition
2021/01/05 17:45:22 [INFO] APIVersion scheduling.k8s.io/v1 Kind PriorityClass
2021/01/05 17:45:22 [INFO] APIVersion scheduling.k8s.io/v1beta1 Kind PriorityClass
2021/01/05 17:45:22 [INFO] APIVersion coordination.k8s.io/v1 Kind Lease
2021/01/05 17:45:22 [INFO] APIVersion coordination.k8s.io/v1beta1 Kind Lease
2021/01/05 17:45:22 [INFO] APIVersion node.k8s.io/v1beta1 Kind RuntimeClass
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind FelixConfiguration
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind BlockAffinity
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkPolicy
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMHandle
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkSet
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind HostEndpoint
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPPool
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkSet
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMConfig
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMBlock
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPConfiguration
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkPolicy
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind ClusterInformation
2021/01/05 17:45:22 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPPeer
2021/01/05 17:45:22 [INFO] APIVersion monitoring.coreos.com/v1 Kind PrometheusRule
2021/01/05 17:45:22 [INFO] APIVersion monitoring.coreos.com/v1 Kind PodMonitor
2021/01/05 17:45:22 [INFO] APIVersion monitoring.coreos.com/v1 Kind ThanosRuler
2021/01/05 17:45:22 [INFO] APIVersion monitoring.coreos.com/v1 Kind Alertmanager
2021/01/05 17:45:22 [INFO] APIVersion monitoring.coreos.com/v1 Kind ServiceMonitor
2021/01/05 17:45:22 [INFO] APIVersion monitoring.coreos.com/v1 Kind Prometheus
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind AuthConfig
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind Notifier
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ClusterMonitorGraph
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDnsProvider
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind Catalog
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ProjectLogging
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind CisConfig
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind NodeDriver
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind Setting
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ClusterCatalog
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind DynamicSchema
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind Group
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ProjectMonitorGraph
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind Preference
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind NodePool
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterAppRevision
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind KontainerDriver
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ProjectNetworkPolicy
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind TemplateVersion
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind CisBenchmarkVersion
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind MonitorMetric
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind NodeTemplate
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDns
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind RoleTemplate
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRegistrationToken
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplate
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlert
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRole
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind EtcdBackup
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlert
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind Project
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplateProjectBinding
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind Cluster
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplateVersion
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind Template
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ComposeConfig
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ProjectRoleTemplateBinding
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplate
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplate
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterApp
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind GroupMember
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ClusterScan
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind Feature
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertRule
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sServiceOption
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ClusterLogging
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertGroup
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind User
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRoleBinding
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind TemplateContent
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ListenConfig
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind Node
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplateRevision
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sSystemImage
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind RkeAddon
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRoleTemplateBinding
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind Token
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertGroup
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ProjectCatalog
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind UserAttribute
2021/01/05 17:45:22 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertRule
2021/01/05 17:45:22 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution
2021/01/05 17:45:22 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision
2021/01/05 17:45:22 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/01/05 17:45:22 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting
2021/01/05 17:45:22 [INFO] APIVersion project.cattle.io/v3 Kind App
2021/01/05 17:45:22 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline
2021/01/05 17:45:22 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/01/05 17:45:22 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/01/05 17:45:22 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/01/05 17:45:22 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics
2021/01/05 17:45:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline
2021/01/05 17:45:23 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor
2021/01/05 17:45:23 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPConfiguration
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert
2021/01/05 17:45:23 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=Event
2021/01/05 17:45:23 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMConfig
2021/01/05 17:45:23 [INFO] Watching metadata for storage.k8s.io/v1beta1, Kind=CSIDriver
2021/01/05 17:45:23 [INFO] Watching metadata for policy/v1beta1, Kind=PodDisruptionBudget
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=ResourceQuota
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=LimitRange
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns
2021/01/05 17:45:23 [INFO] Watching metadata for extensions/v1beta1, Kind=DaemonSet
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=ReplicationController
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken
2021/01/05 17:45:23 [INFO] Watching metadata for extensions/v1beta1, Kind=PodSecurityPolicy
2021/01/05 17:45:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool
2021/01/05 17:45:23 [INFO] Watching metadata for admissionregistration.k8s.io/v1beta1, Kind=MutatingWebhookConfiguration
2021/01/05 17:45:23 [INFO] Watching metadata for batch/v1, Kind=Job
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=PersistentVolume
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMBlock
2021/01/05 17:45:23 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding
2021/01/05 17:45:23 [INFO] Watching metadata for apps/v1, Kind=Deployment
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=ServiceAccount
2021/01/05 17:45:23 [INFO] Watching metadata for autoscaling/v1, Kind=HorizontalPodAutoscaler
2021/01/05 17:45:23 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass
2021/01/05 17:45:23 [INFO] Watching metadata for admissionregistration.k8s.io/v1beta1, Kind=ValidatingWebhookConfiguration
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup
2021/01/05 17:45:23 [INFO] Watching metadata for extensions/v1beta1, Kind=NetworkPolicy
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
2021/01/05 17:45:23 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus
2021/01/05 17:45:23 [INFO] Watching metadata for events.k8s.io/v1beta1, Kind=Event
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric
2021/01/05 17:45:23 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkSet
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=ConfigMap
2021/01/05 17:45:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=Service
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ListenConfig
2021/01/05 17:45:23 [INFO] Watching metadata for apps/v1, Kind=StatefulSet
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate
2021/01/05 17:45:23 [INFO] Watching metadata for networking.k8s.io/v1beta1, Kind=Ingress
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=Node
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent
2021/01/05 17:45:23 [INFO] Watching metadata for storage.k8s.io/v1beta1, Kind=CSINode
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding
2021/01/05 17:45:23 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver
2021/01/05 17:45:23 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule
2021/01/05 17:45:23 [INFO] Watching metadata for node.k8s.io/v1beta1, Kind=RuntimeClass
2021/01/05 17:45:23 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment
2021/01/05 17:45:23 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkSet
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=PodTemplate
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BlockAffinity
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
2021/01/05 17:45:23 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy
2021/01/05 17:45:23 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PodMonitor
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=FelixConfiguration
2021/01/05 17:45:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPPeer
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog
2021/01/05 17:45:23 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ThanosRuler
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon
2021/01/05 17:45:23 [INFO] Watching metadata for extensions/v1beta1, Kind=ReplicaSet
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=Secret
2021/01/05 17:45:23 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=Endpoints
2021/01/05 17:45:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=ClusterInformation
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkPolicy
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=HostEndpoint
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup
2021/01/05 17:45:23 [INFO] Watching metadata for apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token
2021/01/05 17:45:23 [INFO] Watching metadata for certificates.k8s.io/v1beta1, Kind=CertificateSigningRequest
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=User
2021/01/05 17:45:23 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMHandle
2021/01/05 17:45:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=App
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPPool
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig
2021/01/05 17:45:23 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=Pod
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert
2021/01/05 17:45:23 [INFO] Watching metadata for batch/v1beta1, Kind=CronJob
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph
2021/01/05 17:45:23 [INFO] Watching metadata for apps/v1, Kind=DaemonSet
2021/01/05 17:45:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkPolicy
2021/01/05 17:45:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision
2021/01/05 17:45:23 [INFO] Watching metadata for /v1, Kind=Namespace
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion
2021/01/05 17:45:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding
2021/01/05 17:45:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential
2021/01/05 17:45:23 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager
2021/01/05 17:46:54 [INFO] Stopping cluster agent for c-kclzj
2021/01/05 17:46:56 [INFO] Stopping cluster agent for c-bks4b
2021/01/05 17:46:58 [INFO] Stopping cluster agent for c-b6pgw
2021/01/05 17:46:59 [INFO] Stopping cluster agent for c-slvlr
2021/01/05 17:47:40 [INFO] Shutting down CatalogTemplateVersionController controller
2021/01/05 17:47:40 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
2021/01/05 17:47:40 [INFO] Shutting down UserAttributeController controller
2021/01/05 17:47:40 [INFO] Shutting down NodeTemplateController controller
2021/01/05 17:47:40 [INFO] Shutting down GroupMemberController controller
2021/01/05 17:47:40 [INFO] Shutting down SettingController controller
2021/01/05 17:47:40 [INFO] Shutting down SecretController controller
2021/01/05 17:47:40 [INFO] Shutting down RKEK8sSystemImageController controller
2021/01/05 17:47:40 [INFO] Shutting down SecretController controller
2021/01/05 17:47:40 [INFO] Shutting down MultiClusterAppRevisionController controller
2021/01/05 17:47:40 [INFO] Shutting down ClusterTemplateRevisionController controller
2021/01/05 17:47:40 [INFO] Shutting down NodeDriverController controller
2021/01/05 17:47:40 [INFO] Shutting down GlobalRoleBindingController controller
2021/01/05 17:47:40 [INFO] Shutting down UserController controller
2021/01/05 17:47:40 [INFO] Shutting down ProjectCatalogController controller
2021/01/05 17:47:40 [INFO] Shutting down KontainerDriverController controller
2021/01/05 17:47:40 [INFO] Shutting down ClusterCatalogController controller
2021/01/05 17:47:40 [INFO] Shutting down TokenController controller
2021/01/05 17:47:40 [INFO] Shutting down ClusterController controller
2021/01/05 17:47:40 [INFO] Shutting down MultiClusterAppController controller
I0105 17:47:40.354779 8 trace.go:116] Trace[446554467]: "Reflector ListAndWatch" name:github.com/rancher/steve/pkg/clustercache/controller.go:164 (started: 2021-01-05 17:45:23.339854626 +0000 UTC m=+3.808563428) (total time: 2m17.014851119s):
Trace[446554467]: [2m17.014851119s] [2m17.014851119s] END
2021/01/05 17:47:40 [INFO] Shutting down EndpointsController controller
2021/01/05 17:47:40 [INFO] Shutting down RKEAddonController controller
2021/01/05 17:47:40 [INFO] Shutting down CatalogController controller
2021/01/05 17:47:40 [INFO] Shutting down GroupController controller
2021/01/05 17:47:40 [INFO] Shutting down CisConfigController controller
2021/01/05 17:47:40 [INFO] Shutting down PodSecurityPolicyTemplateController controller
2021/01/05 17:47:40 [INFO] Shutting down ProjectRoleTemplateBindingController controller
2021/01/05 17:47:40 [INFO] Shutting down ProjectLoggingController controller
2021/01/05 17:47:40 [INFO] Shutting down CatalogTemplateController controller
2021/01/05 17:47:40 [INFO] Shutting down GlobalDNSController controller
2021/01/05 17:47:40 [INFO] Shutting down ClusterRegistrationTokenController controller
2021/01/05 17:47:40 [INFO] Shutting down RKEK8sServiceOptionController controller
2021/01/05 17:47:40 [INFO] Shutting down CisConfigController controller
2021/01/05 17:47:40 [INFO] Shutting down CisBenchmarkVersionController controller
2021/01/05 17:47:40 [INFO] Shutting down PipelineController controller
2021/01/05 17:47:40 [INFO] Shutting down CisBenchmarkVersionController controller
2021/01/05 17:47:40 [INFO] Shutting down AppController controller
2021/01/05 17:47:40 [INFO] Shutting down SourceCodeCredentialController controller
2021/01/05 17:47:40 [INFO] Shutting down RoleTemplateController controller
2021/01/05 17:47:40 [INFO] Shutting down NodeController controller
2021/01/05 17:47:40 [INFO] Shutting down SecretController controller
2021/01/05 17:47:40 [INFO] Shutting down AuthConfigController controller
2021/01/05 17:47:40 [INFO] Shutting down NamespaceController controller
2021/01/05 17:47:40 [INFO] Shutting down DynamicSchemaController controller
2021/01/05 17:47:40 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2021/01/05 17:47:40 [INFO] Shutting down ClusterRoleBindingController controller
2021/01/05 17:47:40 [INFO] Shutting down PipelineExecutionController controller
2021/01/05 17:47:40 [INFO] Shutting down ClusterRoleTemplateBindingController controller
2021/01/05 17:47:40 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2021/01/05 17:47:40 [INFO] Shutting down ClusterRoleController controller
2021/01/05 17:47:40 [INFO] Shutting down apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition workers
2021/01/05 17:47:40 [FATAL] leaderelection lost for cattle-controllers
2021/01/05 17:47:40 [INFO] Shutting down NodePoolController controller
2021/01/05 17:47:40 [INFO] Shutting down ProjectController controller
2021/01/05 17:47:40 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers