Hello,
I recently notice that I get an Error from a Cronjob related to my Backups running for the Volumes, saying:
Job longhorn-system/pvc-xxx failed to complete.
I think that the Job is making the real Backup of the Volume from the Taken Snapshot, because if I set the Setting to only make Snapshots there is no Job (and so no Error). I don’t have an Idee how I can start debugging here.
Hope someone can help 
greetings
Could you please provide which Longhorn version you’re using?
Also, could you please check the job’s log and longhorn-manager log to see if there is any clue on this.
I am using Version 1.1.2
I could’nt check the Jobs Log, because ther ist no Pod associate with the Job and the Alert Runbook Tells me to use “kubectl logs ”.
The longhorn-manager has Entries about refused TCP Connections to a Cluster IP
Blockquote
E0907 07:13:55.982088 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.ShareManager: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/sharemanagers?allowWatchBookmarks=true&resourceVersion=58937168&timeout=9m8s&timeoutSeconds=548&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:55 am E0907 07:13:55.983195 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Setting: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/settings?allowWatchBookmarks=true&resourceVersion=77705601&timeout=9m9s&timeoutSeconds=549&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:55 am E0907 07:13:55.984209 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CSIDriver: Get “https://10.43.0.1:443/apis/storage.k8s.io/v1beta1/csidrivers?allowWatchBookmarks=true&resourceVersion=76360156&timeout=6m46s&timeoutSeconds=406&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:55 am E0907 07:13:55.985515 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: Get “https://10.43.0.1:443/api/v1/secrets?allowWatchBookmarks=true&resourceVersion=77728863&timeout=5m23s&timeoutSeconds=323&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:55 am E0907 07:13:55.986481 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.DaemonSet: Get “https://10.43.0.1:443/apis/apps/v1/daemonsets?allowWatchBookmarks=true&resourceVersion=76361061&timeout=7m26s&timeoutSeconds=446&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:55 am E0907 07:13:55.987572 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Engine: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/engines?allowWatchBookmarks=true&resourceVersion=78137149&timeout=7m35s&timeoutSeconds=455&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:55 am E0907 07:13:55.989782 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PriorityClass: Get “https://10.43.0.1:443/apis/scheduling.k8s.io/v1/priorityclasses?allowWatchBookmarks=true&resourceVersion=58937040&timeout=6m6s&timeoutSeconds=366&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:55 am E0907 07:13:55.989796 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: Get “https://10.43.0.1:443/api/v1/persistentvolumes?allowWatchBookmarks=true&resourceVersion=58937040&timeout=7m58s&timeoutSeconds=478&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:55 am E0907 07:13:55.999155 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CronJob: Get “https://10.43.0.1:443/apis/batch/v1beta1/cronjobs?allowWatchBookmarks=true&resourceVersion=78002429&timeout=7m48s&timeoutSeconds=468&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.000362 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.BackingImage: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/backingimages?allowWatchBookmarks=true&resourceVersion=76359545&timeout=8m12s&timeoutSeconds=492&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.001831 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.InstanceManager: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/instancemanagers?allowWatchBookmarks=true&resourceVersion=76361168&timeout=6m34s&timeoutSeconds=394&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.003014 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Volume: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/volumes?allowWatchBookmarks=true&resourceVersion=78137150&timeout=5m52s&timeoutSeconds=352&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.005185 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Node: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/nodes?allowWatchBookmarks=true&resourceVersion=78137192&timeout=9m57s&timeoutSeconds=597&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.005410 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.BackingImageManager: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/backingimagemanagers?allowWatchBookmarks=true&resourceVersion=76359545&timeout=6m33s&timeoutSeconds=393&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.006884 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: Get “https://10.43.0.1:443/apis/policy/v1beta1/poddisruptionbudgets?allowWatchBookmarks=true&resourceVersion=76360462&timeout=8m35s&timeoutSeconds=515&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.007638 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Deployment: Get “https://10.43.0.1:443/apis/apps/v1/deployments?allowWatchBookmarks=true&resourceVersion=78132209&timeout=5m5s&timeoutSeconds=305&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.970445 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: Get “https://10.43.0.1:443/apis/storage.k8s.io/v1/storageclasses?allowWatchBookmarks=true&resourceVersion=58937040&timeout=9m45s&timeoutSeconds=585&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.972854 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Replica: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/replicas?allowWatchBookmarks=true&resourceVersion=64543185&timeout=9m5s&timeoutSeconds=545&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.973928 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: Get “https://10.43.0.1:443/api/v1/persistentvolumeclaims?allowWatchBookmarks=true&resourceVersion=58937040&timeout=7m59s&timeoutSeconds=479&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.974993 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: Get “https://10.43.0.1:443/api/v1/services?allowWatchBookmarks=true&resourceVersion=76360723&timeout=8m14s&timeoutSeconds=494&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.976684 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: Get “https://10.43.0.1:443/api/v1/pods?allowWatchBookmarks=true&resourceVersion=78132205&timeout=9m12s&timeoutSeconds=552&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.979028 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: Get “https://10.43.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=78137100&timeout=9m58s&timeoutSeconds=598&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.980287 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.EngineImage: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/engineimages?allowWatchBookmarks=true&resourceVersion=76360333&timeout=5m24s&timeoutSeconds=324&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.981579 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ConfigMap: Get “https://10.43.0.1:443/api/v1/configmaps?allowWatchBookmarks=true&resourceVersion=78137194&timeout=7m8s&timeoutSeconds=428&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.982646 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.ShareManager: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/sharemanagers?allowWatchBookmarks=true&resourceVersion=58937168&timeout=7m53s&timeoutSeconds=473&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.983812 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Setting: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/settings?allowWatchBookmarks=true&resourceVersion=77705601&timeout=5m4s&timeoutSeconds=304&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.985096 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CSIDriver: Get “https://10.43.0.1:443/apis/storage.k8s.io/v1beta1/csidrivers?allowWatchBookmarks=true&resourceVersion=76360156&timeout=7m43s&timeoutSeconds=463&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.986110 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: Get “https://10.43.0.1:443/api/v1/secrets?allowWatchBookmarks=true&resourceVersion=77728863&timeout=8m37s&timeoutSeconds=517&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.987082 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.DaemonSet: Get “https://10.43.0.1:443/apis/apps/v1/daemonsets?allowWatchBookmarks=true&resourceVersion=76361061&timeout=9m47s&timeoutSeconds=587&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.988836 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Engine: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/engines?allowWatchBookmarks=true&resourceVersion=78137149&timeout=8m34s&timeoutSeconds=514&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.990965 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PriorityClass: Get “https://10.43.0.1:443/apis/scheduling.k8s.io/v1/priorityclasses?allowWatchBookmarks=true&resourceVersion=58937040&timeout=8m0s&timeoutSeconds=480&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.991508 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: Get “https://10.43.0.1:443/api/v1/persistentvolumes?allowWatchBookmarks=true&resourceVersion=58937040&timeout=5m35s&timeoutSeconds=335&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:56 am E0907 07:13:56.999780 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CronJob: Get “https://10.43.0.1:443/apis/batch/v1beta1/cronjobs?allowWatchBookmarks=true&resourceVersion=78002429&timeout=5m8s&timeoutSeconds=308&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.002346 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.BackingImage: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/backingimages?allowWatchBookmarks=true&resourceVersion=76359545&timeout=9m24s&timeoutSeconds=564&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.002406 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.InstanceManager: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/instancemanagers?allowWatchBookmarks=true&resourceVersion=76361168&timeout=7m26s&timeoutSeconds=446&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.003551 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Volume: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/volumes?allowWatchBookmarks=true&resourceVersion=78137150&timeout=8m37s&timeoutSeconds=517&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.005725 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Node: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/nodes?allowWatchBookmarks=true&resourceVersion=78137192&timeout=9m14s&timeoutSeconds=554&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.006912 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.BackingImageManager: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/backingimagemanagers?allowWatchBookmarks=true&resourceVersion=76359545&timeout=5m38s&timeoutSeconds=338&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.007946 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: Get “https://10.43.0.1:443/apis/policy/v1beta1/poddisruptionbudgets?allowWatchBookmarks=true&resourceVersion=76360462&timeout=6m16s&timeoutSeconds=376&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.008962 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Deployment: Get “https://10.43.0.1:443/apis/apps/v1/deployments?allowWatchBookmarks=true&resourceVersion=78132209&timeout=9m49s&timeoutSeconds=589&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.971276 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: Get “https://10.43.0.1:443/apis/storage.k8s.io/v1/storageclasses?allowWatchBookmarks=true&resourceVersion=58937040&timeout=9m58s&timeoutSeconds=598&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.973667 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Replica: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/replicas?allowWatchBookmarks=true&resourceVersion=64543185&timeout=9m10s&timeoutSeconds=550&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.974969 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: Get “https://10.43.0.1:443/api/v1/persistentvolumeclaims?allowWatchBookmarks=true&resourceVersion=58937040&timeout=6m50s&timeoutSeconds=410&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.975833 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: Get “https://10.43.0.1:443/api/v1/services?allowWatchBookmarks=true&resourceVersion=76360723&timeout=7m16s&timeoutSeconds=436&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.977374 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: Get “https://10.43.0.1:443/api/v1/pods?allowWatchBookmarks=true&resourceVersion=78132205&timeout=5m23s&timeoutSeconds=323&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.979607 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: Get “https://10.43.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=78137100&timeout=7m2s&timeoutSeconds=422&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.981056 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.EngineImage: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/engineimages?allowWatchBookmarks=true&resourceVersion=76360333&timeout=5m6s&timeoutSeconds=306&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.982140 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ConfigMap: Get “https://10.43.0.1:443/api/v1/configmaps?allowWatchBookmarks=true&resourceVersion=78137194&timeout=5m51s&timeoutSeconds=351&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.983263 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.ShareManager: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/sharemanagers?allowWatchBookmarks=true&resourceVersion=58937168&timeout=8m49s&timeoutSeconds=529&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.984355 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Setting: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/settings?allowWatchBookmarks=true&resourceVersion=77705601&timeout=8m42s&timeoutSeconds=522&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.986506 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CSIDriver: Get “https://10.43.0.1:443/apis/storage.k8s.io/v1beta1/csidrivers?allowWatchBookmarks=true&resourceVersion=76360156&timeout=9m0s&timeoutSeconds=540&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.987619 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: Get “https://10.43.0.1:443/api/v1/secrets?allowWatchBookmarks=true&resourceVersion=77728863&timeout=5m34s&timeoutSeconds=334&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.988567 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.DaemonSet: Get “https://10.43.0.1:443/apis/apps/v1/daemonsets?allowWatchBookmarks=true&resourceVersion=76361061&timeout=8m25s&timeoutSeconds=505&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.990442 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Engine: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/engines?allowWatchBookmarks=true&resourceVersion=78137149&timeout=5m3s&timeoutSeconds=303&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.991765 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PriorityClass: Get “https://10.43.0.1:443/apis/scheduling.k8s.io/v1/priorityclasses?allowWatchBookmarks=true&resourceVersion=58937040&timeout=6m55s&timeoutSeconds=415&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:57 am E0907 07:13:57.992669 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: Get “https://10.43.0.1:443/api/v1/persistentvolumes?allowWatchBookmarks=true&resourceVersion=58937040&timeout=7m24s&timeoutSeconds=444&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.000978 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CronJob: Get “https://10.43.0.1:443/apis/batch/v1beta1/cronjobs?allowWatchBookmarks=true&resourceVersion=78002429&timeout=5m54s&timeoutSeconds=354&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.002929 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.BackingImage: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/backingimages?allowWatchBookmarks=true&resourceVersion=76359545&timeout=6m19s&timeoutSeconds=379&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.003963 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.InstanceManager: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/instancemanagers?allowWatchBookmarks=true&resourceVersion=76361168&timeout=5m15s&timeoutSeconds=315&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.005106 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Volume: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/volumes?allowWatchBookmarks=true&resourceVersion=78137150&timeout=5m10s&timeoutSeconds=310&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.006290 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Node: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/nodes?allowWatchBookmarks=true&resourceVersion=78137192&timeout=6m44s&timeoutSeconds=404&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.007477 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.BackingImageManager: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/backingimagemanagers?allowWatchBookmarks=true&resourceVersion=76359545&timeout=7m59s&timeoutSeconds=479&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.008688 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: Get “https://10.43.0.1:443/apis/policy/v1beta1/poddisruptionbudgets?allowWatchBookmarks=true&resourceVersion=76360462&timeout=8m32s&timeoutSeconds=512&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.009479 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Deployment: Get “https://10.43.0.1:443/apis/apps/v1/deployments?allowWatchBookmarks=true&resourceVersion=78132209&timeout=7m5s&timeoutSeconds=425&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.973961 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.StorageClass: Get “https://10.43.0.1:443/apis/storage.k8s.io/v1/storageclasses?allowWatchBookmarks=true&resourceVersion=58937040&timeout=6m41s&timeoutSeconds=401&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.974416 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Replica: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/replicas?allowWatchBookmarks=true&resourceVersion=64543185&timeout=6m43s&timeoutSeconds=403&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.975628 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolumeClaim: Get “https://10.43.0.1:443/api/v1/persistentvolumeclaims?allowWatchBookmarks=true&resourceVersion=58937040&timeout=8m0s&timeoutSeconds=480&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.977372 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: Get “https://10.43.0.1:443/api/v1/services?allowWatchBookmarks=true&resourceVersion=76360723&timeout=7m43s&timeoutSeconds=463&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.978584 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Pod: Get “https://10.43.0.1:443/api/v1/pods?allowWatchBookmarks=true&resourceVersion=78132205&timeout=8m27s&timeoutSeconds=507&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.980202 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: Get “https://10.43.0.1:443/api/v1/nodes?allowWatchBookmarks=true&resourceVersion=78137100&timeout=7m22s&timeoutSeconds=442&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.981813 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.EngineImage: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/engineimages?allowWatchBookmarks=true&resourceVersion=76360333&timeout=6m39s&timeoutSeconds=399&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.982801 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.ConfigMap: Get “https://10.43.0.1:443/api/v1/configmaps?allowWatchBookmarks=true&resourceVersion=78137194&timeout=5m56s&timeoutSeconds=356&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.984190 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.ShareManager: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/sharemanagers?allowWatchBookmarks=true&resourceVersion=58937168&timeout=6m36s&timeoutSeconds=396&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.984965 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Setting: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/settings?allowWatchBookmarks=true&resourceVersion=77705601&timeout=6m24s&timeoutSeconds=384&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.987055 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CSIDriver: Get “https://10.43.0.1:443/apis/storage.k8s.io/v1beta1/csidrivers?allowWatchBookmarks=true&resourceVersion=76360156&timeout=6m27s&timeoutSeconds=387&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.989318 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Secret: Get “https://10.43.0.1:443/api/v1/secrets?allowWatchBookmarks=true&resourceVersion=77728863&timeout=5m38s&timeoutSeconds=338&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.989322 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.DaemonSet: Get “https://10.43.0.1:443/apis/apps/v1/daemonsets?allowWatchBookmarks=true&resourceVersion=76361061&timeout=6m20s&timeoutSeconds=380&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.990955 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Engine: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/engines?allowWatchBookmarks=true&resourceVersion=78137149&timeout=5m17s&timeoutSeconds=317&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.992402 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PriorityClass: Get “https://10.43.0.1:443/apis/scheduling.k8s.io/v1/priorityclasses?allowWatchBookmarks=true&resourceVersion=58937040&timeout=6m41s&timeoutSeconds=401&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:58 am E0907 07:13:58.993507 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.PersistentVolume: Get “https://10.43.0.1:443/api/v1/persistentvolumes?allowWatchBookmarks=true&resourceVersion=58937040&timeout=9m41s&timeoutSeconds=581&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:59 am E0907 07:13:59.001700 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.CronJob: Get “https://10.43.0.1:443/apis/batch/v1beta1/cronjobs?allowWatchBookmarks=true&resourceVersion=78002429&timeout=5m37s&timeoutSeconds=337&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:59 am E0907 07:13:59.003810 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.BackingImage: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/backingimages?allowWatchBookmarks=true&resourceVersion=76359545&timeout=7m27s&timeoutSeconds=447&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:59 am E0907 07:13:59.004607 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.InstanceManager: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/instancemanagers?allowWatchBookmarks=true&resourceVersion=76361168&timeout=9m48s&timeoutSeconds=588&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:59 am E0907 07:13:59.005767 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Volume: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/volumes?allowWatchBookmarks=true&resourceVersion=78137150&timeout=6m33s&timeoutSeconds=393&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:59 am E0907 07:13:59.006909 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.Node: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/nodes?allowWatchBookmarks=true&resourceVersion=78137192&timeout=5m2s&timeoutSeconds=302&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:59 am E0907 07:13:59.008129 1 reflector.go:282] github.com/longhorn/longhorn-manager/k8s/pkg/client/informers/externalversions/factory.go:117: Failed to watch *v1beta1.BackingImageManager: Get “https://10.43.0.1:443/apis/longhorn.io/v1beta1/backingimagemanagers?allowWatchBookmarks=true&resourceVersion=76359545&timeout=5m33s&timeoutSeconds=333&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:59 am E0907 07:13:59.009279 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.PodDisruptionBudget: Get “https://10.43.0.1:443/apis/policy/v1beta1/poddisruptionbudgets?allowWatchBookmarks=true&resourceVersion=76360462&timeout=7m59s&timeoutSeconds=479&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Tue, Sep 7 2021 9:13:59 am E0907 07:13:59.010591 1 reflector.go:282] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Deployment: Get “https://10.43.0.1:443/apis/apps/v1/deployments?allowWatchBookmarks=true&resourceVersion=78132209&timeout=5m52s&timeoutSeconds=352&watch=true”: dial tcp 10.43.0.1:443: connect: connection refused
Addition: Just checked. This 10.43.0.1 IP dont exist in the Cluster
The 10.43.0.1 is the kube-apiserver IP address.
The problem seems like the Longhorn is unable to access the kube-apiserver.
Could you please check your cluster networking running correctly?
All Network-Pods are up and running.
I don’t experience any other Problems according to Network.
Is there a specific thing I should check?
Nope. Perhaps you could download the support bundle first, and then try to restart the Longhorn manager DaemonSet.
I’m not 100% sure you hit this issue [BUG] High CPU utilization for Longhorn manager sometimes due to exhaustion of all the available sockets because of socket leaking log stream · Issue #2778 · longhorn/longhorn · GitHub or not.
But from the Longhorn manager log you provided, it’s like the Longhorn manager server can’t reach the kube-apiserver.