Rancher failed on subscribe several objects(namespace, persistentVolume, storageClass, etc) time out

Rancher
1

Hi,

I am facing a problem with my rancher 2.5.9 installation. When I try to visualize my “custom cluster” I get the screen below:

img-01
img-011417×324 5.5 KB

When I try to kubectl to the cluster, I get the error:

img-02
img-021581×61 4.57 KB

When I try to kubectl the cluster using the “control-plane” nodes directly it works fine.
When I search the pod with the IP on the error I discover it is a rancher server pod.

img-03
img-031654×716 17 KB

Looking at the problematic pod’s logs I got:

2021/08/18 19:48:29 [ERROR] failed on subscribe persistentVolume: Get "https://10.100.192.63:6443/api/v1/persistentvolumes?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:29 [ERROR] failed on subscribe apiService: Get "https://10.100.192.63:6443/apis/apiregistration.k8s.io/v1/apiservices?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:29 [ERROR] failed on subscribe storageClass: Get "https://10.100.192.63:6443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:29 [ERROR] failed on subscribe namespace: Get "https://10.100.192.63:6443/api/v1/namespaces?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:30 [ERROR] failed on subscribe apiService: Get "https://10.100.192.63:6443/apis/apiregistration.k8s.io/v1/apiservices?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:30 [ERROR] failed on subscribe namespace: Get "https://10.100.192.63:6443/api/v1/namespaces?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:30 [ERROR] failed on subscribe persistentVolume: Get "https://10.100.192.63:6443/api/v1/persistentvolumes?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:30 [ERROR] failed on subscribe storageClass: Get "https://10.100.192.63:6443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:32 [ERROR] Unknown error: Get "https://10.100.192.63:6443/api/v1/persistentvolumes?timeout=45s": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:37 [ERROR] Unknown error: Get "https://10.100.192.63:6443/api/v1/namespaces?timeout=45s": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:50 [ERROR] failed on subscribe storageClass: Get "https://10.100.192.63:6443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:50 [ERROR] failed on subscribe namespace: Get "https://10.100.192.63:6443/api/v1/namespaces?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:50 [ERROR] failed on subscribe apiService: Get "https://10.100.192.63:6443/apis/apiregistration.k8s.io/v1/apiservices?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:50 [ERROR] failed on subscribe persistentVolume: Get "https://10.100.192.63:6443/api/v1/persistentvolumes?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:51 [ERROR] failed on subscribe storageClass: Get "https://10.100.192.63:6443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:51 [ERROR] failed on subscribe apiService: Get "https://10.100.192.63:6443/apis/apiregistration.k8s.io/v1/apiservices?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:51 [ERROR] failed on subscribe persistentVolume: Get "https://10.100.192.63:6443/api/v1/persistentvolumes?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:48:51 [ERROR] failed on subscribe namespace: Get "https://10.100.192.63:6443/api/v1/namespaces?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:11 [ERROR] failed on subscribe storageClass: Get "https://10.100.192.63:6443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:11 [ERROR] failed on subscribe apiService: Get "https://10.100.192.63:6443/apis/apiregistration.k8s.io/v1/apiservices?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:11 [ERROR] failed on subscribe persistentVolume: Get "https://10.100.192.63:6443/api/v1/persistentvolumes?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:11 [ERROR] failed on subscribe namespace: Get "https://10.100.192.63:6443/api/v1/namespaces?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:12 [ERROR] failed on subscribe namespace: Get "https://10.100.192.63:6443/api/v1/namespaces?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:12 [ERROR] failed on subscribe apiService: Get "https://10.100.192.63:6443/apis/apiregistration.k8s.io/v1/apiservices?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:12 [ERROR] failed on subscribe persistentVolume: Get "https://10.100.192.63:6443/api/v1/persistentvolumes?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:12 [ERROR] failed on subscribe storageClass: Get "https://10.100.192.63:6443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:33 [ERROR] failed on subscribe storageClass: Get "https://10.100.192.63:6443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:33 [ERROR] failed on subscribe apiService: Get "https://10.100.192.63:6443/apis/apiregistration.k8s.io/v1/apiservices?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:33 [ERROR] failed on subscribe namespace: Get "https://10.100.192.63:6443/api/v1/namespaces?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:33 [ERROR] failed on subscribe persistentVolume: Get "https://10.100.192.63:6443/api/v1/persistentvolumes?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:34 [ERROR] failed on subscribe storageClass: Get "https://10.100.192.63:6443/apis/storage.k8s.io/v1/storageclasses?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:34 [ERROR] failed on subscribe persistentVolume: Get "https://10.100.192.63:6443/api/v1/persistentvolumes?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:34 [ERROR] failed on subscribe namespace: Get "https://10.100.192.63:6443/api/v1/namespaces?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:34 [ERROR] failed on subscribe apiService: Get "https://10.100.192.63:6443/apis/apiregistration.k8s.io/v1/apiservices?resourceVersion=0&timeout=30m0s&timeoutSeconds=1800&watch=true": write tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout
2021/08/18 19:49:40 [INFO] error in remotedialer server [400]: read tcp 10.42.0.50:80->10.42.2.59:60240: i/o timeout

When I delete the problematic pod, everything starts to work properly.

I work with Rancher 2.5.9.
The rancher server cluster has 3 servers running etcd, control plane and rancher server. The “workload” cluster is a “custom cluster” running on bare metal. I use traefik ingress controller and metalLb as “loadBalancer”. Looking at the nodes I could not find anything different on CPU or memory usage.

Can anyone help me with this?

Best Regards,

Paulo Leal