Observed a panic: "close of closed channel"

Rancher 2.x
1

Hi,

My cluster of rancher instances are down and the following message appears:

E0603 07:35:39.313167       8 runtime.go:78] Observed a panic: "close of closed channel" (close of closed channel)
goroutine 25573 [running]:
github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime.logPanic(0x3683a20, 0x45ab540)
	/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x3683a20, 0x45ab540)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop(0xc006480700)
	/go/src/github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache/shared_informer.go:632 +0x2e2
github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc001180d80, 0xc00ccf3e50)
	/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x59
created by github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62
panic: close of closed channel [recovered]
	panic: close of closed channel

goroutine 25573 [running]:
github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x105
panic(0x3683a20, 0x45ab540)
	/usr/local/go/src/runtime/panic.go:679 +0x1b2
github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache.(*processorListener).pop(0xc006480700)
	/go/src/github.com/rancher/rancher/vendor/k8s.io/client-go/tools/cache/shared_informer.go:632 +0x2e2
github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1(0xc001180d80, 0xc00ccf3e50)
	/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x59
created by github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/src/github.com/rancher/rancher/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:69 +0x62

Any idea ?

I use rancher v2.4.2.

2

Can you please give more context on the issue? When did this start to happen? Where are you running Rancher? What versions are you using (Rancher/RKE/Kubernetes)?