Liveness probe failed connection refused

kamal_boumahdi · August 1, 2021, 4:33pm

Hello everyone, I hope you’re having a great time.
here’s the issue I’m having :

~$ kubectl -n cattle-system get po
NAME                               READY   STATUS      RESTARTS   AGE
helm-operation-8plqf               0/2     Completed   0          46m
helm-operation-95r5c               0/2     Completed   0          44m
helm-operation-bjkrm               0/2     Completed   0          43m
helm-operation-cszqz               0/2     Completed   0          45m
helm-operation-mvh84               0/2     Completed   0          44m
rancher-df6c8dcd6-2fhmf            0/1     Evicted     0          33m
rancher-df6c8dcd6-2hdsn            0/1     Evicted     0          3m8s
rancher-df6c8dcd6-2p8gt            0/1     Evicted     0          27m
...
rancher-df6c8dcd6-xnljn            1/1     Running     0          16m
rancher-df6c8dcd6-z8s5j            0/1     Evicted     0          16m
rancher-df6c8dcd6-zf82l            0/1     Evicted     0          3m10s
rancher-df6c8dcd6-zqg4k            0/1     Evicted     0          53m
rancher-df6c8dcd6-zxc26            0/1     Evicted     0          27m
rancher-df6c8dcd6-zxwl5            0/1     Evicted     0          3m2s
rancher-webhook-69b7997768-5r59d   1/1     Running     0          45m

so I it seems that these pods keep getting created then evicted, I try to describe one of them and here is what I get:

~$ kubectl describe po rancher-df6c8dcd6-2fhmf -n cattle-system
Name:           rancher-df6c8dcd6-2fhmf
Namespace:      cattle-system
Priority:       0
Node:           joxstar/
Start Time:     Sun, 01 Aug 2021 15:36:43 +0000
Labels:         app=rancher
                pod-template-hash=df6c8dcd6
                release=rancher
Annotations:    cni.projectcalico.org/podIP: 10.42.0.148/32
                cni.projectcalico.org/podIPs: 10.42.0.148/32
Status:         Failed
Reason:         Evicted
Message:        The node was low on resource: ephemeral-storage. Container rancher was using 416152Ki, which exceeds its request of 0.
IP:
IPs:            <none>
Controlled By:  ReplicaSet/rancher-df6c8dcd6
Containers:
  rancher:
    Image:      rancher/rancher:v2.5.9
    Port:       80/TCP
    Host Port:  0/TCP
    Args:
      --http-listen-port=80
      --https-listen-port=443
      --add-local=true
    Liveness:   http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness:  http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      CATTLE_NAMESPACE:     cattle-system
      CATTLE_PEER_SERVICE:  rancher
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from rancher-token-8dc7l (ro)
Volumes:
  rancher-token-8dc7l:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rancher-token-8dc7l
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  42m (x6 over 47m)  default-scheduler  0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate.
  Normal   Scheduled         41m                default-scheduler  Successfully assigned cattle-system/rancher-df6c8dcd6-2fhmf to joxstar
  Normal   Pulled            41m                kubelet            Container image "rancher/rancher:v2.5.9" already present on machine
  Normal   Created           41m                kubelet            Created container rancher
  Normal   Started           41m                kubelet            Started container rancher
  Warning  Unhealthy         41m                kubelet            Readiness probe failed: Get http://10.42.0.148:80/healthz: dial tcp 10.42.0.148:80: connect: connection refused
  Warning  Evicted           29m                kubelet            The node was low on resource: ephemeral-storage. Container rancher was using 416152Ki, which exceeds its request of 0.
  Normal   Killing           29m                kubelet            Stopping container rancher

Now I understand that these pods are consuming resources, but why are they not deleted?
also what’s causing these failures?

 Liveness:   http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
 Readiness:  http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3

I know I must have messed up somewhere (being such a noob), but how do I go about fixing it?
any help is very much appreciated.
possibly helpful information:

it’s an RKE sigle node cluster.
using ubuntu 20.04 with VMware workstation.
I installed nginx as a layer 4 LB.

superseb · August 2, 2021, 11:51am

The default threshold for Kubernetes to clean up terminated pods is 12500, we changed it to 1000 so unless you have a 1000 it won’t automatically cleanup. You can do it manually though.

In general, you are running out of resources, did you check the storage on the machines and the logs of the kubelet? The failures probably come from the resource problem as well, but you can check the logs of the Rancher pods (it is described on Rancher Docs: Rancher HA)

kamal_boumahdi · August 3, 2021, 4:58pm

Thanks a lot, I really appreciate your input.
So if I understand you correctly, these failures might be related to storage.

 Liveness:   http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
 Readiness:  http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3

Storage on the machines:
this is from the node,

~$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               2.8G     0  2.8G   0% /dev
tmpfs                              570M  3.2M  567M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv   19G   15G  3.0G  84% /
tmpfs                              2.8G     0  2.8G   0% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                              2.8G     0  2.8G   0% /sys/fs/cgroup
/dev/loop1                          56M   56M     0 100% /snap/core18/1944
/dev/loop2                          68M   68M     0 100% /snap/lxd/20326
/dev/loop0                          56M   56M     0 100% /snap/core18/2074
/dev/loop5                          33M   33M     0 100% /snap/snapd/12398
/dev/loop3                          71M   71M     0 100% /snap/lxd/21029
/dev/sda2                          976M  200M  710M  22% /boot
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/d392c473ddafd39f835ef5af5dac0ee49c3f8f264fcf55035f5cf4215487bd9c/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/861d417b513fd84f101af1b0471dd7c0d3960739438e3203b3324c2c7d98ea78/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/2f857e84ddde915a39d9097c514957d02a5f9e46ac37cf45a5cf6db7eb979501/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/0bf82cfd0cce5200da8a180620e52043ebbe335c766a676b6a60e40511368d64/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/3cb14e053278c7f6b716f45ab2a77a0262ad458bfaac076339ebfe765a66d075/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/b7345745d75470f836823eba82d1f7928584adda15ee8c2967696364b802efc2/mounts/shm
shm                                 64M     0   64M   0% /var/lib/docker/containers/3c0ea6d5bcddb59168195cd411a3bb21e399d5a2eb9332f4918908bd44fd54c8/mounts/shm
shm                                 64M     0   64M   0% /var/lib/docker/containers/01b97faf4b9296a108ce3b59eefba859d5d7f6e094d5b8dbf719ab5864bff8ad/mounts/shm
shm                                 64M     0   64M   0% /var/lib/docker/containers/a0656042661e238f94bde8a5c34fc33fb01f0b76fb2672e6a9f5ea2f703c5a41/mounts/shm
shm                                 64M     0   64M   0% /var/lib/docker/containers/ae22fabae178bcd1bccac89f5fc2659b17a2ec16549d6f37da2913bdb1a6ffdf/mounts/shm
/dev/loop6                          33M   33M     0 100% /snap/snapd/12704
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/bec4ed7ecc3edd7de4e1c48c6ea563cad598003825eebc0136051194295a3e04/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/3cd89bb3a7866f9dad3bfd55128d0a486a6ca122c8d121165fdbb9e7f809ca05/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/07dbbb8d8d6805e0a131536ce725ea1dfa8f2191814b314f81b7e6f125ec4191/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/462461b3357720d2e78ce39f4ad920fe2ee590e7bbb4f86469e002ed776a06c0/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/a6eb2912e31ffdf3d9fa7dbd1a5da1ca89b532aa179e64a39f10c4a3ad00f3e9/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/47c91123ff2d97608d2203847cd492c8e96902f7065caadbab786d412aeb70b7/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/dc9dbbe39c796c14bb383a1a07ae881a2a0af0221a4d60e2e8c51a7c1795706b/mounts/shm
shm                                 64M     0   64M   0% /var/lib/docker/containers/f58bd8ef88866c56f2e9577e05e06f3a265076ec3bdbf79c38086f6c13a371ca/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/5f282847f0321d51132c1c24ddb6618d372cec4b062998896e3863e3b912da4e/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/38020b77c7da34ef82455b0f2f434985e5dd6b3fa0852501a7b3b8f7db815a6e/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/799e81a76e4e70cba1415f5810fea86cd12cb40017691124981a4c9b61acb84d/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/4748a0449675b165e9e1d56cccb80598bdc759e9a348fb4f5048886bfe3f9d85/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/5e2e128834df14909a5503cb3e08842448fde687e1b339926f5c6ca186d15921/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/9ebc701293ef0121f46b092936341cbe02b0a73f30ec70d7852212dff025fffd/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/9a36ff4639bf20f48d9c8d432c7487332206d7e2c5e1ad12e2b9c1e3c640dc60/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/ce91f0972465fdb63c226d1d886daa5b31492ed9e3e50636144943a5b8d7ef7b/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/9d4752e9e5d123ad8fd257dccd167a91867b36ec009a214dd490fc843dc54bc0/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/76a3489d96bf610c02965402d4833377432111d4f616715ad820b6a29c136585/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/7e6f4f1aa0241d5a464eeb5b0fb286f3a29ff05135e9c4d794a233dd7fb55c4e/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/0472c91a8ce4dc05b079070b2be730cee3ee7d4969871eb8f1e98a859ca21fdd/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/435c66b1affa2b2eeb8af1c0f9d3f5006d1081831946f965e39245e28a9f9d26/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/3e6666543a7c0e8d178154f653e29a23accf02b3326c35b2cad5e2b515ebd660/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/87e44c2973d0d32c5a941c0fda2754e23b9550ea0b202071f1ec62d51993a88e/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/51033b72d0230e98b624865d8ce2fd81c6b48b288de99aa2c76f5d66ae489491/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/000d7f046735d3c061ca73472044590c692542f0a2eb16035493306d86e2fe59/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/38fcaac2c17558abce2c6ae9b6c2c73d4025aed01fda89f1d345950254e3afed/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/14822a707b31d2b85390838c89ef32cd3b1850be6337a04319726c518bb89786/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/028581af7ee7a14f5dec181aeaaf374eafd295701f32bc733d23a8ace7f23961/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/6d9522db823484a9819ddd2dd60a6c23b78e43d24c8ec6226becb8fac52ff088/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/d795b1fe639fb44f78f9cdf50f1122f256a4a9679fa07d5f0a5284ef0c937c32/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/a87741123c91134bb9a36eb795df4c24c53b575f9e27b536bc75c279d73c6e3e/mounts/shm
shm                                 64M     0   64M   0% /var/lib/docker/containers/5f533ee3b152670f220fde48a647e56d71ac8fffad1ebc84ec099bae6d1c04f3/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/2b113c3ca500ce89d29527d1ed704403af204c49ce0d58d686987aed728a9cb9/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/a3150886ca02e1462fee695d971373606cf0acb8b3bed38e3c499a8572264c06/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/c5d2caf6bb454b32710bae0d4f2937149a99927583f32d20420ec7be69b0b3e4/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/0f6efd41812028fbfa413ad21a994f526f414de4fb414aa2524af41353c42d86/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/76ac325e2b7b8d845b3fd194163e8a6c498520963a6758a49514cd629b6928f0/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/b1fb21283c8ca6268666f5b9e0fbf47a53bebe08db90a639910f7069d06f3408/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/53ee260a8864066fe4e887a6a83a4b501308c5ff57c216895cfc7a60287a1789/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/95ede95613e9da50f134a094c740e78dca4958a720c88d797ee59f8004d589c2/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/60259319bb18d9db884217194d4c52d20ee38a988a777b061cb553037befc862/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/208c748fdf715b726b9d55383e3bf53be4b6c042c0a534cfec31f46ac9df6934/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/ce59b119fc5bb3f5907bb900d368fa8b868adb70b901ac26aaad62b35d9f184c/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/03d8f07c57c95449a13145c90798c78da56df28b0cf6cdadce5ad2a26338a0b3/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/847ffa689ca3f860d61c774988631458fd003668f701098726377895d8ee154e/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/8072e39ef7fe6981ad7f1e57a092fe7e75a08c0af73d255f328c1b676e171efd/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/efa5c71cafddf20d7a16c66f5adab45765398b3040bccf8c9e9aba5343d3f1b5/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/16e1e2dd57c9a2fc629f9e82589f016304f37e5bc5b5f0fe780ae556a36c3f40/merged
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/d2ebe14847c2a6ab6cced64fe885e164252a9ba61bfd7e1fc854b56fc3956848/merged
shm                                 64M     0   64M   0% /var/lib/docker/containers/9321c1d5fc825f882bbace743b09c000d2021ea599d4e95b225e0e74ac21bf74/mounts/shm
overlay                             19G   15G  3.0G  84% /var/lib/docker/overlay2/128d261f32f41934ceb0689ea854b20aa9f6ee9cbb0b54dd22c665e958b87697/merged
tmpfs                              570M     0  570M   0% /run/user/1000

and if I’m reading this correctly, at 84% it does seem almost saturated.

But according to the Rancher logs, this Readiness probe failure took place well before the storage issue ( a little over 10 mins before).

Which makes me think that something else is causing these pods to fail (or be considered unhealthy), and since for some reason (which I don’t know) they’re just hanging there instead of being completely irradiated.

While more pods are created to satisfy the desired state (to ultimately meet the same fate). We eventually end up with the storage problem.

Not sure though, but if that’s right then I need to find out what’s causing this Readiness probe failure in the first place.

what’s your take on that? thanks again for your help, in the mean time I’ll spin up another VM with exaggerated storage capacity and see if I get the same issue just to be sure.

superseb · August 3, 2021, 8:58pm

The logs of the Rancher pods/containers will most likely give the best indication why the checks are failing, see previous reply.

kamal_boumahdi · August 6, 2021, 6:16pm

I got rid of the storage problem by switching to other VMs, yet it seems like that wasn’t the problem.
kubectl -n cattle-system get pods returns that everything is okay.

~$ kubectl -n cattle-system get pods -l app=rancher -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
rancher-cbbdf9fd5-n4cgr   1/1     Running   0          23m   10.42.0.10   fuji1   <none>           <none>
rancher-cbbdf9fd5-nj99v   1/1     Running   0          23m   10.42.0.11   fuji1   <none>           <none>
rancher-cbbdf9fd5-pmz5b   1/1     Running   0          23m   10.42.0.9    fuji1   <none>           <none>

but describe shows the same issue as before, for all three pods

~$ kubectl -n cattle-system describe pods -l app=rancher
Name:         rancher-cbbdf9fd5-n4cgr
Namespace:    cattle-system
Priority:     0
Node:         fuji1/192.168.157.141
Start Time:   Fri, 06 Aug 2021 17:32:54 +0000
Labels:       app=rancher
              pod-template-hash=cbbdf9fd5
              release=rancher
Annotations:  cni.projectcalico.org/podIP: 10.42.0.10/32
              cni.projectcalico.org/podIPs: 10.42.0.10/32
Status:       Running
IP:           10.42.0.10
IPs:
  IP:           10.42.0.10
Controlled By:  ReplicaSet/rancher-cbbdf9fd5
Containers:
  rancher:
    Container ID:  docker://318db6fdff3225e4066ffc56999370103e776a9a175c9358f637334d46b89b0a
    Image:         rancher/rancher:v2.5.9
    Image ID:      docker-pullable://rancher/rancher@sha256:4cfd6f4ecf64a8bb117738cc30c57c63617eaf4b3bdbb49dff024f36e3e589bb
    Port:          80/TCP
    Host Port:     0/TCP
    Args:
      --no-cacerts
      --http-listen-port=80
      --https-listen-port=443
      --add-local=true
    State:          Running
      Started:      Fri, 06 Aug 2021 17:38:47 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness:      http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      CATTLE_NAMESPACE:     cattle-system
      CATTLE_PEER_SERVICE:  rancher
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from rancher-token-bqsls (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  rancher-token-bqsls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rancher-token-bqsls
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  23m                default-scheduler  Successfully assigned cattle-system/rancher-cbbdf9fd5-n4cgr to fuji1
  Normal   Pulling    23m                kubelet            Pulling image "rancher/rancher:v2.5.9"
  Normal   Pulled     18m                kubelet            Successfully pulled image "rancher/rancher:v2.5.9"
  Normal   Created    18m                kubelet            Created container rancher
  Normal   Started    17m                kubelet            Started container rancher
  Warning  Unhealthy  16m (x3 over 17m)  kubelet            Readiness probe failed: Get http://10.42.0.10:80/healthz: dial tcp 10.42.0.10:80: connect: connection refused
  Warning  Unhealthy  16m                kubelet            Liveness probe failed: Get http://10.42.0.10:80/healthz: dial tcp 10.42.0.10:80: connect: connection refused


Name:         rancher-cbbdf9fd5-nj99v
Namespace:    cattle-system
Priority:     0
Node:         fuji1/192.168.157.141
Start Time:   Fri, 06 Aug 2021 17:32:54 +0000
Labels:       app=rancher
              pod-template-hash=cbbdf9fd5
              release=rancher
Annotations:  cni.projectcalico.org/podIP: 10.42.0.11/32
              cni.projectcalico.org/podIPs: 10.42.0.11/32
Status:       Running
IP:           10.42.0.11
IPs:
  IP:           10.42.0.11
Controlled By:  ReplicaSet/rancher-cbbdf9fd5
Containers:
  rancher:
    Container ID:  docker://2ba8702efdaa5503d27804bc1a751ea661618049081dbfef530d04d0d6651546
    Image:         rancher/rancher:v2.5.9
    Image ID:      docker-pullable://rancher/rancher@sha256:4cfd6f4ecf64a8bb117738cc30c57c63617eaf4b3bdbb49dff024f36e3e589bb
    Port:          80/TCP
    Host Port:     0/TCP
    Args:
      --no-cacerts
      --http-listen-port=80
      --https-listen-port=443
      --add-local=true
    State:          Running
      Started:      Fri, 06 Aug 2021 17:38:47 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness:      http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      CATTLE_NAMESPACE:     cattle-system
      CATTLE_PEER_SERVICE:  rancher
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from rancher-token-bqsls (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  rancher-token-bqsls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rancher-token-bqsls
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  23m                default-scheduler  Successfully assigned cattle-system/rancher-cbbdf9fd5-nj99v to fuji1
  Normal   Pulling    23m                kubelet            Pulling image "rancher/rancher:v2.5.9"
  Normal   Pulled     18m                kubelet            Successfully pulled image "rancher/rancher:v2.5.9"
  Normal   Created    18m                kubelet            Created container rancher
  Normal   Started    17m                kubelet            Started container rancher
  Warning  Unhealthy  16m (x3 over 17m)  kubelet            Readiness probe failed: Get http://10.42.0.11:80/healthz: dial tcp 10.42.0.11:80: connect: connection refused
  Warning  Unhealthy  16m                kubelet            Liveness probe failed: Get http://10.42.0.11:80/healthz: dial tcp 10.42.0.11:80: connect: connection refused


Name:         rancher-cbbdf9fd5-pmz5b
Namespace:    cattle-system
Priority:     0
Node:         fuji1/192.168.157.141
Start Time:   Fri, 06 Aug 2021 17:32:54 +0000
Labels:       app=rancher
              pod-template-hash=cbbdf9fd5
              release=rancher
Annotations:  cni.projectcalico.org/podIP: 10.42.0.9/32
              cni.projectcalico.org/podIPs: 10.42.0.9/32
Status:       Running
IP:           10.42.0.9
IPs:
  IP:           10.42.0.9
Controlled By:  ReplicaSet/rancher-cbbdf9fd5
Containers:
  rancher:
    Container ID:  docker://93cd1749b5cc11a64bb9c6d88fb4e00f0e8c965c29ff8e7cf0ff61db70b12497
    Image:         rancher/rancher:v2.5.9
    Image ID:      docker-pullable://rancher/rancher@sha256:4cfd6f4ecf64a8bb117738cc30c57c63617eaf4b3bdbb49dff024f36e3e589bb
    Port:          80/TCP
    Host Port:     0/TCP
    Args:
      --no-cacerts
      --http-listen-port=80
      --https-listen-port=443
      --add-local=true
    State:          Running
      Started:      Fri, 06 Aug 2021 17:38:47 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:80/healthz delay=60s timeout=1s period=30s #success=1 #failure=3
    Readiness:      http-get http://:80/healthz delay=5s timeout=1s period=30s #success=1 #failure=3
    Environment:
      CATTLE_NAMESPACE:     cattle-system
      CATTLE_PEER_SERVICE:  rancher
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from rancher-token-bqsls (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  rancher-token-bqsls:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  rancher-token-bqsls
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  23m                default-scheduler  Successfully assigned cattle-system/rancher-cbbdf9fd5-pmz5b to fuji1
  Normal   Pulling    23m                kubelet            Pulling image "rancher/rancher:v2.5.9"
  Normal   Pulled     18m                kubelet            Successfully pulled image "rancher/rancher:v2.5.9"
  Normal   Created    18m                kubelet            Created container rancher
  Normal   Started    17m                kubelet            Started container rancher
  Warning  Unhealthy  16m                kubelet            Liveness probe failed: Get http://10.42.0.9:80/healthz: dial tcp 10.42.0.9:80: connect: connection refused
  Warning  Unhealthy  16m (x4 over 17m)  kubelet            Readiness probe failed: Get http://10.42.0.9:80/healthz: dial tcp 10.42.0.9:80: connect: connection refused

and here are the logs. “fleet-local” has been modified ? I didn’t do that, does it have to do with RKEs version and Ranchers perhaps?

~$ kubectl -n cattle-system logs -l app=rancher
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind App
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/08/06 17:44:42 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/08/06 17:44:42 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/08/06 17:44:42 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/08/06 17:44:42 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/08/06 17:44:42 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics
2021/08/06 17:44:48 [INFO] Watching metadata for gitjob.cattle.io/v1, Kind=GitJob
2021/08/06 17:44:48 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterGroup
2021/08/06 17:44:48 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
2021/08/06 17:45:17 [WARNING] error updating ns fleet-local status: Operation cannot be fulfilled on namespaces "fleet-local": the object has been modified; please apply your changes to the latest version and try again
2021/08/06 17:45:17 [WARNING] error updating ns cluster-fleet-local-local-1a3d67d0a899 status: Operation cannot be fulfilled on namespaces "cluster-fleet-local-local-1a3d67d0a899": the object has been modified; please apply your changes to the latest version and try again
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind App
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/08/06 17:44:41 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/08/06 17:44:41 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/08/06 17:44:41 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics

the warning about the unhealthiness of the pods described earlier

~$ kubectl -n cattle-system get events
LAST SEEN   TYPE      REASON              OBJECT                                                     MESSAGE
21m         Normal    CREATE              ingress/cm-acme-http-solver-fllqn                          Ingress cattle-system/cm-acme-http-solver-fllqn
13m         Normal    UPDATE              ingress/cm-acme-http-solver-fllqn                          Ingress cattle-system/cm-acme-http-solver-fllqn
<unknown>   Normal    Scheduled           pod/cm-acme-http-solver-nnffq                              Successfully assigned cattle-system/cm-acme-http-solver-nnffq to fuji1
21m         Normal    Pulling             pod/cm-acme-http-solver-nnffq                              Pulling image "quay.io/jetstack/cert-manager-acmesolver:v1.0.4"
15m         Normal    Pulled              pod/cm-acme-http-solver-nnffq                              Successfully pulled image "quay.io/jetstack/cert-manager-acmesolver:v1.0.4"
15m         Normal    Created             pod/cm-acme-http-solver-nnffq                              Created container acmesolver
15m         Normal    Started             pod/cm-acme-http-solver-nnffq                              Started container acmesolver
<unknown>   Normal    Scheduled           pod/helm-operation-9hhdw                                   Successfully assigned cattle-system/helm-operation-9hhdw to fuji1
13m         Normal    Pulling             pod/helm-operation-9hhdw                                   Pulling image "rancher/shell:v0.1.6"
12m         Normal    Pulled              pod/helm-operation-9hhdw                                   Successfully pulled image "rancher/shell:v0.1.6"
12m         Normal    Created             pod/helm-operation-9hhdw                                   Created container helm
12m         Normal    Started             pod/helm-operation-9hhdw                                   Started container helm
12m         Normal    Pulled              pod/helm-operation-9hhdw                                   Container image "rancher/shell:v0.1.6" already present on machine
12m         Normal    Created             pod/helm-operation-9hhdw                                   Created container proxy
12m         Normal    Started             pod/helm-operation-9hhdw                                   Started container proxy
<unknown>   Normal    Scheduled           pod/helm-operation-glggv                                   Successfully assigned cattle-system/helm-operation-glggv to fuji1
11m         Normal    Pulled              pod/helm-operation-glggv                                   Container image "rancher/shell:v0.1.6" already present on machine
11m         Normal    Created             pod/helm-operation-glggv                                   Created container helm
11m         Normal    Started             pod/helm-operation-glggv                                   Started container helm
11m         Normal    Pulled              pod/helm-operation-glggv                                   Container image "rancher/shell:v0.1.6" already present on machine
11m         Normal    Created             pod/helm-operation-glggv                                   Created container proxy
11m         Normal    Started             pod/helm-operation-glggv                                   Started container proxy
<unknown>   Normal    Scheduled           pod/helm-operation-j45bz                                   Successfully assigned cattle-system/helm-operation-j45bz to fuji1
12m         Normal    Pulling             pod/helm-operation-j45bz                                   Pulling image "rancher/shell:v0.1.6"
12m         Normal    Pulled              pod/helm-operation-j45bz                                   Successfully pulled image "rancher/shell:v0.1.6"
12m         Normal    Created             pod/helm-operation-j45bz                                   Created container helm
12m         Normal    Started             pod/helm-operation-j45bz                                   Started container helm
12m         Normal    Pulled              pod/helm-operation-j45bz                                   Container image "rancher/shell:v0.1.6" already present on machine
12m         Normal    Created             pod/helm-operation-j45bz                                   Created container proxy
12m         Normal    Started             pod/helm-operation-j45bz                                   Started container proxy
<unknown>   Normal    Scheduled           pod/helm-operation-n8w9w                                   Successfully assigned cattle-system/helm-operation-n8w9w to fuji1
10m         Normal    Pulled              pod/helm-operation-n8w9w                                   Container image "rancher/shell:v0.1.6" already present on machine
10m         Normal    Created             pod/helm-operation-n8w9w                                   Created container helm
10m         Normal    Started             pod/helm-operation-n8w9w                                   Started container helm
10m         Normal    Pulled              pod/helm-operation-n8w9w                                   Container image "rancher/shell:v0.1.6" already present on machine
10m         Normal    Created             pod/helm-operation-n8w9w                                   Created container proxy
10m         Normal    Started             pod/helm-operation-n8w9w                                   Started container proxy
<unknown>   Normal    Scheduled           pod/helm-operation-wsljx                                   Successfully assigned cattle-system/helm-operation-wsljx to fuji1
11m         Normal    Pulled              pod/helm-operation-wsljx                                   Container image "rancher/shell:v0.1.6" already present on machine
11m         Normal    Created             pod/helm-operation-wsljx                                   Created container helm
11m         Normal    Started             pod/helm-operation-wsljx                                   Started container helm
11m         Normal    Pulled              pod/helm-operation-wsljx                                   Container image "rancher/shell:v0.1.6" already present on machine
11m         Normal    Created             pod/helm-operation-wsljx                                   Created container proxy
11m         Normal    Started             pod/helm-operation-wsljx                                   Started container proxy
<unknown>   Normal    Scheduled           pod/rancher-cbbdf9fd5-n4cgr                                Successfully assigned cattle-system/rancher-cbbdf9fd5-n4cgr to fuji1
21m         Normal    Pulling             pod/rancher-cbbdf9fd5-n4cgr                                Pulling image "rancher/rancher:v2.5.9"
15m         Normal    Pulled              pod/rancher-cbbdf9fd5-n4cgr                                Successfully pulled image "rancher/rancher:v2.5.9"
15m         Normal    Created             pod/rancher-cbbdf9fd5-n4cgr                                Created container rancher
15m         Normal    Started             pod/rancher-cbbdf9fd5-n4cgr                                Started container rancher
14m         Warning   Unhealthy           pod/rancher-cbbdf9fd5-n4cgr                                Readiness probe failed: Get http://10.42.0.10:80/healthz: dial tcp 10.42.0.10:80: connect: connection refused
14m         Warning   Unhealthy           pod/rancher-cbbdf9fd5-n4cgr                                Liveness probe failed: Get http://10.42.0.10:80/healthz: dial tcp 10.42.0.10:80: connect: connection refused
<unknown>   Normal    Scheduled           pod/rancher-cbbdf9fd5-nj99v                                Successfully assigned cattle-system/rancher-cbbdf9fd5-nj99v to fuji1
21m         Normal    Pulling             pod/rancher-cbbdf9fd5-nj99v                                Pulling image "rancher/rancher:v2.5.9"
15m         Normal    Pulled              pod/rancher-cbbdf9fd5-nj99v                                Successfully pulled image "rancher/rancher:v2.5.9"
15m         Normal    Created             pod/rancher-cbbdf9fd5-nj99v                                Created container rancher
15m         Normal    Started             pod/rancher-cbbdf9fd5-nj99v                                Started container rancher
14m         Warning   Unhealthy           pod/rancher-cbbdf9fd5-nj99v                                Readiness probe failed: Get http://10.42.0.11:80/healthz: dial tcp 10.42.0.11:80: connect: connection refused
14m         Warning   Unhealthy           pod/rancher-cbbdf9fd5-nj99v                                Liveness probe failed: Get http://10.42.0.11:80/healthz: dial tcp 10.42.0.11:80: connect: connection refused
<unknown>   Normal    Scheduled           pod/rancher-cbbdf9fd5-pmz5b                                Successfully assigned cattle-system/rancher-cbbdf9fd5-pmz5b to fuji1
21m         Normal    Pulling             pod/rancher-cbbdf9fd5-pmz5b                                Pulling image "rancher/rancher:v2.5.9"
16m         Normal    Pulled              pod/rancher-cbbdf9fd5-pmz5b                                Successfully pulled image "rancher/rancher:v2.5.9"
15m         Normal    Created             pod/rancher-cbbdf9fd5-pmz5b                                Created container rancher
15m         Normal    Started             pod/rancher-cbbdf9fd5-pmz5b                                Started container rancher
14m         Warning   Unhealthy           pod/rancher-cbbdf9fd5-pmz5b                                Readiness probe failed: Get http://10.42.0.9:80/healthz: dial tcp 10.42.0.9:80: connect: connection refused
14m         Warning   Unhealthy           pod/rancher-cbbdf9fd5-pmz5b                                Liveness probe failed: Get http://10.42.0.9:80/healthz: dial tcp 10.42.0.9:80: connect: connection refused
21m         Normal    SuccessfulCreate    replicaset/rancher-cbbdf9fd5                               Created pod: rancher-cbbdf9fd5-n4cgr
21m         Normal    SuccessfulCreate    replicaset/rancher-cbbdf9fd5                               Created pod: rancher-cbbdf9fd5-nj99v
21m         Normal    SuccessfulCreate    replicaset/rancher-cbbdf9fd5                               Created pod: rancher-cbbdf9fd5-pmz5b
<unknown>   Normal    Scheduled           pod/rancher-webhook-69b7997768-ptdph                       Successfully assigned cattle-system/rancher-webhook-69b7997768-ptdph to fuji1
12m         Normal    Pulling             pod/rancher-webhook-69b7997768-ptdph                       Pulling image "rancher/rancher-webhook:v0.1.1"
11m         Normal    Pulled              pod/rancher-webhook-69b7997768-ptdph                       Successfully pulled image "rancher/rancher-webhook:v0.1.1"
11m         Normal    Created             pod/rancher-webhook-69b7997768-ptdph                       Created container rancher-webhook
11m         Normal    Started             pod/rancher-webhook-69b7997768-ptdph                       Started container rancher-webhook
12m         Normal    SuccessfulCreate    replicaset/rancher-webhook-69b7997768                      Created pod: rancher-webhook-69b7997768-ptdph
12m         Normal    ScalingReplicaSet   deployment/rancher-webhook                                 Scaled up replica set rancher-webhook-69b7997768 to 1
21m         Normal    ScalingReplicaSet   deployment/rancher                                         Scaled up replica set rancher-cbbdf9fd5 to 3
21m         Normal    CREATE              ingress/rancher                                            Ingress cattle-system/rancher
13m         Normal    UPDATE              ingress/rancher                                            Ingress cattle-system/rancher
21m         Normal    CreateCertificate   ingress/rancher                                            Successfully created Certificate "tls-rancher-ingress"
21m         Normal    Started             challenge/tls-rancher-ingress-c55rq-193660333-2832573432   Challenge scheduled for processing
21m         Normal    Presented           challenge/tls-rancher-ingress-c55rq-193660333-2832573432   Presented challenge using HTTP-01 challenge mechanism
21m         Normal    Created             order/tls-rancher-ingress-c55rq-193660333                  Created Challenge resource "tls-rancher-ingress-c55rq-193660333-2832573432" for domain "rancher.my.org"
21m         Normal    IssuerNotReady      certificaterequest/tls-rancher-ingress-c55rq               Referenced issuer does not have a Ready status condition
21m         Normal    Issuing             certificate/tls-rancher-ingress                            Issuing certificate as Secret does not exist
21m         Normal    Generated           certificate/tls-rancher-ingress                            Stored new private key in temporary Secret resource "tls-rancher-ingress-n5f5l"
21m         Normal    Requested           certificate/tls-rancher-ingress                            Created new CertificateRequest resource "tls-rancher-ingress-c55rq"

so what do you think? can you point me in the right direction ?

superseb · August 6, 2021, 9:43pm

Not sure if I am following, the output shows Running and 0 restarts. Are you saying that changes and they restart randomly? Can you get logs for each pod (or for the pod that is restarting) and post them in full?

kamal_boumahdi · August 8, 2021, 1:14pm

earlier this morning I checked, and this what I found:

$ kubectl -n cattle-system get pods -l app=rancher -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
rancher-cbbdf9fd5-n4cgr   1/1     Running   2          43h   10.42.0.34   fuji1   <none>           <none>
rancher-cbbdf9fd5-nj99v   1/1     Running   2          43h   10.42.0.39   fuji1   <none>           <none>
rancher-cbbdf9fd5-pmz5b   1/1     Running   2          43h   10.42.0.33   fuji1   <none>           <none>

and the logs go like this:
for rancher-cbbdf9fd5-n4cgr

$ kubectl -n cattle-system logs rancher-cbbdf9fd5-n4cgr
2021/08/08 12:48:47 [INFO] Rancher version v2.5.9 (3c5418944) is starting
2021/08/08 12:48:47 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Agent:false Features: ClusterRegistry:}
2021/08/08 12:48:47 [INFO] Listening on /tmp/log.sock
2021/08/08 12:48:54 [INFO] Starting API controllers
2021/08/08 12:48:55 [INFO] Starting management.cattle.io/v3, Kind=User controller
2021/08/08 12:48:55 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2021/08/08 12:48:55 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2021/08/08 12:48:55 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2021/08/08 12:48:55 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:48:55 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2021/08/08 12:48:58 [INFO] Starting apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition controller
2021/08/08 12:48:58 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:48:58 [INFO] Starting management.cattle.io/v3, Kind=Feature controller
2021/08/08 12:48:58 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:48:58 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:48:58 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:48:58 [INFO] Starting management.cattle.io/v3, Kind=Preference controller
2021/08/08 12:48:58 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:48:58 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2021/08/08 12:48:58 [INFO] Starting catalog.cattle.io/v1, Kind=ClusterRepo controller
2021/08/08 12:48:58 [INFO] Starting management.cattle.io/v3, Kind=Setting controller
2021/08/08 12:48:58 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:48:58 [INFO] Starting /v1, Kind=ConfigMap controller
I0808 12:48:58.578644       7 leaderelection.go:243] attempting to acquire leader lease kube-system/cattle-controllers...
2021/08/08 12:48:59 [INFO] Refreshing all schemas
2021/08/08 12:49:01 [INFO] Refreshing all schemas
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Binding
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind ComponentStatus
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind ConfigMap
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Endpoints
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Event
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind LimitRange
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Namespace
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Node
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind PersistentVolumeClaim
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind PersistentVolume
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Pod
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind PodTemplate
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind ReplicationController
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind ResourceQuota
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Secret
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind ServiceAccount
2021/08/08 12:49:01 [INFO] APIVersion /v1 Kind Service
2021/08/08 12:49:01 [INFO] APIVersion apiregistration.k8s.io/v1 Kind APIService
2021/08/08 12:49:01 [INFO] APIVersion apiregistration.k8s.io/v1beta1 Kind APIService
2021/08/08 12:49:01 [INFO] APIVersion extensions/v1beta1 Kind Ingress
2021/08/08 12:49:01 [INFO] APIVersion apps/v1 Kind ControllerRevision
2021/08/08 12:49:01 [INFO] APIVersion apps/v1 Kind DaemonSet
2021/08/08 12:49:01 [INFO] APIVersion apps/v1 Kind Deployment
2021/08/08 12:49:01 [INFO] APIVersion apps/v1 Kind ReplicaSet
2021/08/08 12:49:01 [INFO] APIVersion apps/v1 Kind StatefulSet
2021/08/08 12:49:01 [INFO] APIVersion events.k8s.io/v1beta1 Kind Event
2021/08/08 12:49:01 [INFO] APIVersion authentication.k8s.io/v1 Kind TokenReview
2021/08/08 12:49:01 [INFO] APIVersion authentication.k8s.io/v1beta1 Kind TokenReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1 Kind LocalSubjectAccessReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectAccessReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectRulesReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1 Kind SubjectAccessReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind LocalSubjectAccessReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectAccessReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectRulesReview
2021/08/08 12:49:01 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SubjectAccessReview
2021/08/08 12:49:01 [INFO] APIVersion autoscaling/v1 Kind HorizontalPodAutoscaler
2021/08/08 12:49:01 [INFO] APIVersion autoscaling/v2beta1 Kind HorizontalPodAutoscaler
2021/08/08 12:49:01 [INFO] APIVersion autoscaling/v2beta2 Kind HorizontalPodAutoscaler
2021/08/08 12:49:01 [INFO] APIVersion batch/v1 Kind Job
2021/08/08 12:49:01 [INFO] APIVersion batch/v1beta1 Kind CronJob
2021/08/08 12:49:01 [INFO] APIVersion certificates.k8s.io/v1beta1 Kind CertificateSigningRequest
2021/08/08 12:49:01 [INFO] APIVersion networking.k8s.io/v1 Kind NetworkPolicy
2021/08/08 12:49:01 [INFO] APIVersion networking.k8s.io/v1beta1 Kind Ingress
2021/08/08 12:49:01 [INFO] APIVersion policy/v1beta1 Kind PodDisruptionBudget
2021/08/08 12:49:01 [INFO] APIVersion policy/v1beta1 Kind PodSecurityPolicy
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRoleBinding
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRole
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind RoleBinding
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind Role
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRoleBinding
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRole
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind RoleBinding
2021/08/08 12:49:01 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind Role
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1 Kind CSINode
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1 Kind StorageClass
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1 Kind VolumeAttachment
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSIDriver
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSINode
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1beta1 Kind StorageClass
2021/08/08 12:49:01 [INFO] APIVersion storage.k8s.io/v1beta1 Kind VolumeAttachment
2021/08/08 12:49:01 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind MutatingWebhookConfiguration
2021/08/08 12:49:01 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind ValidatingWebhookConfiguration
2021/08/08 12:49:01 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind MutatingWebhookConfiguration
2021/08/08 12:49:01 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind ValidatingWebhookConfiguration
2021/08/08 12:49:01 [INFO] APIVersion apiextensions.k8s.io/v1 Kind CustomResourceDefinition
2021/08/08 12:49:01 [INFO] APIVersion apiextensions.k8s.io/v1beta1 Kind CustomResourceDefinition
2021/08/08 12:49:01 [INFO] APIVersion scheduling.k8s.io/v1 Kind PriorityClass
2021/08/08 12:49:01 [INFO] APIVersion scheduling.k8s.io/v1beta1 Kind PriorityClass
2021/08/08 12:49:01 [INFO] APIVersion coordination.k8s.io/v1 Kind Lease
2021/08/08 12:49:01 [INFO] APIVersion coordination.k8s.io/v1beta1 Kind Lease
2021/08/08 12:49:01 [INFO] APIVersion node.k8s.io/v1beta1 Kind RuntimeClass
2021/08/08 12:49:01 [INFO] APIVersion discovery.k8s.io/v1beta1 Kind EndpointSlice
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1 Kind Challenge
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1 Kind Order
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1beta1 Kind Order
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1beta1 Kind Challenge
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1alpha3 Kind Order
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1alpha3 Kind Challenge
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Challenge
2021/08/08 12:49:01 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Order
2021/08/08 12:49:01 [INFO] APIVersion catalog.cattle.io/v1 Kind ClusterRepo
2021/08/08 12:49:01 [INFO] APIVersion catalog.cattle.io/v1 Kind App
2021/08/08 12:49:01 [INFO] APIVersion catalog.cattle.io/v1 Kind Operation
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1 Kind ClusterIssuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1 Kind CertificateRequest
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1 Kind Issuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1 Kind Certificate
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1beta1 Kind CertificateRequest
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1beta1 Kind Issuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1beta1 Kind Certificate
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1beta1 Kind ClusterIssuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha3 Kind Issuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha3 Kind Certificate
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha3 Kind ClusterIssuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha3 Kind CertificateRequest
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Certificate
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha2 Kind ClusterIssuer
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha2 Kind CertificateRequest
2021/08/08 12:49:01 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Issuer
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkSet
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMHandle
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPPeer
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkSet
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkPolicy
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMBlock
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkPolicy
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind FelixConfiguration
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind HostEndpoint
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPPool
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMConfig
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind ClusterInformation
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind BlockAffinity
2021/08/08 12:49:01 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPConfiguration
2021/08/08 12:49:01 [INFO] APIVersion gitjob.cattle.io/v1 Kind GitJob
2021/08/08 12:49:01 [INFO] APIVersion monitoring.coreos.com/v1 Kind PrometheusRule
2021/08/08 12:49:01 [INFO] APIVersion monitoring.coreos.com/v1 Kind Prometheus
2021/08/08 12:49:01 [INFO] APIVersion monitoring.coreos.com/v1 Kind Alertmanager
2021/08/08 12:49:01 [INFO] APIVersion monitoring.coreos.com/v1 Kind ServiceMonitor
2021/08/08 12:49:01 [INFO] APIVersion rancher.cattle.io/v1 Kind Project
2021/08/08 12:49:01 [INFO] APIVersion rancher.cattle.io/v1 Kind RoleTemplate
2021/08/08 12:49:01 [INFO] APIVersion rancher.cattle.io/v1 Kind RoleTemplateBinding
2021/08/08 12:49:01 [INFO] APIVersion rancher.cattle.io/v1 Kind Cluster
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind BundleDeployment
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind BundleNamespaceMapping
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Bundle
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Cluster
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Content
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterRegistration
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind GitRepoRestriction
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterRegistrationToken
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind GitRepo
2021/08/08 12:49:01 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterGroup
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterAppRevision
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sSystemImage
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind EtcdBackup
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRoleBinding
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind RoleTemplate
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Cluster
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Catalog
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterApp
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind DynamicSchema
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind MonitorMetric
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRoleTemplateBinding
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertRule
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Feature
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Token
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlert
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind NodePool
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Project
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind TemplateVersion
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Preference
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplateRevision
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlert
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind RkeAddon
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterMonitorGraph
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Group
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplate
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplate
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterScan
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sServiceOption
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Setting
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectNetworkPolicy
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind SamlToken
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectLogging
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplate
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind KontainerDriver
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRole
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind CisBenchmarkVersion
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertGroup
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterCatalog
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectCatalog
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind TemplateContent
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind GroupMember
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind User
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind UserAttribute
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind FleetWorkspace
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplateProjectBinding
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterLogging
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertGroup
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Template
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ComposeConfig
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind CisConfig
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind NodeTemplate
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRegistrationToken
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Notifier
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind AuthConfig
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDnsProvider
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectMonitorGraph
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertRule
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind ProjectRoleTemplateBinding
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind NodeDriver
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind Node
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDns
2021/08/08 12:49:01 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplateVersion
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind App
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/08/08 12:49:01 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/08/08 12:49:01 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/08/08 12:49:01 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics
2021/08/08 12:49:03 [INFO] Running in clustered mode with ID 10.42.0.34, monitoring endpoint cattle-system/rancher
2021/08/08 12:49:17 [INFO] Starting cluster controllers for local
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplateVersion controller
2021/08/08 12:49:21 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeRepository controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=GlobalDns controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=GlobalRole controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller
2021/08/08 12:49:21 [INFO] Starting project.cattle.io/v3, Kind=Pipeline controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterApp controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=CisConfig controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplate controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sSystemImage controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=Catalog controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=CisBenchmarkVersion controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=Project controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
2021/08/08 12:49:21 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeCredential controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterAppRevision controller
2021/08/08 12:49:21 [INFO] Starting project.cattle.io/v3, Kind=App controller
2021/08/08 12:49:21 [INFO] Starting project.cattle.io/v3, Kind=PipelineExecution controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=DynamicSchema controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=NodeTemplate controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=NodeDriver controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=AuthConfig controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ClusterCatalog controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=User controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=RkeAddon controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplate controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
2021/08/08 12:49:21 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:49:21 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=KontainerDriver controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ProjectLogging controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ClusterRegistrationToken controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sServiceOption controller
2021/08/08 12:49:21 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplate controller
2021/08/08 12:49:23 [INFO] Starting cluster agent for local [owner=false]
2021/08/08 12:49:23 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:49:23 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:49:23 [INFO] Starting /v1, Kind=Service controller
2021/08/08 12:49:23 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:49:23 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:49:23 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:49:23 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:49:23 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:49:23 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion
2021/08/08 12:49:28 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
2021/08/08 12:49:28 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=ResourceQuota
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=PodTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease
2021/08/08 12:49:28 [INFO] Watching metadata for storage.k8s.io/v1beta1, Kind=CSIDriver
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=PersistentVolume
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver
2021/08/08 12:49:28 [INFO] Watching metadata for batch/v1, Kind=Job
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting
2021/08/08 12:49:28 [INFO] Watching metadata for autoscaling/v2beta2, Kind=HorizontalPodAutoscaler
2021/08/08 12:49:28 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template
2021/08/08 12:49:28 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=ClusterRepo
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Cluster
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkSet
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=FelixConfiguration
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Namespace
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule
2021/08/08 12:49:28 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=RoleTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for cert-manager.io/v1, Kind=Issuer
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Pod
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkPolicy
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision
2021/08/08 12:49:28 [INFO] Watching metadata for events.k8s.io/v1beta1, Kind=Event
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=User
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistrationToken
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group
2021/08/08 12:49:28 [INFO] Watching metadata for discovery.k8s.io/v1beta1, Kind=EndpointSlice
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Service
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=ReplicationController
2021/08/08 12:49:28 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding
2021/08/08 12:49:28 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Challenge
2021/08/08 12:49:28 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=Operation
2021/08/08 12:49:28 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=ClusterInformation
2021/08/08 12:49:28 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
2021/08/08 12:49:28 [INFO] Watching metadata for apps/v1, Kind=Deployment
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkPolicy
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleDeployment
2021/08/08 12:49:28 [INFO] Watching metadata for networking.k8s.io/v1beta1, Kind=Ingress
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Endpoints
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=LimitRange
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPPool
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Content
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
2021/08/08 12:49:28 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=Cluster
2021/08/08 12:49:28 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Order
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Event
2021/08/08 12:49:28 [INFO] Watching metadata for apps/v1, Kind=StatefulSet
2021/08/08 12:49:28 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Node
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleNamespaceMapping
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph
2021/08/08 12:49:28 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMBlock
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for node.k8s.io/v1beta1, Kind=RuntimeClass
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
2021/08/08 12:49:28 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule
2021/08/08 12:49:28 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Bundle
2021/08/08 12:49:28 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=RoleTemplateBinding
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig
2021/08/08 12:49:28 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole
2021/08/08 12:49:28 [INFO] Watching metadata for certificates.k8s.io/v1beta1, Kind=CertificateSigningRequest
2021/08/08 12:49:28 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token
2021/08/08 12:49:28 [INFO] Watching metadata for cert-manager.io/v1, Kind=CertificateRequest
2021/08/08 12:49:28 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=Secret
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BlockAffinity
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=HostEndpoint
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for batch/v1beta1, Kind=CronJob
2021/08/08 12:49:28 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=App
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=FleetWorkspace
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterGroup
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=ConfigMap
2021/08/08 12:49:28 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding
2021/08/08 12:49:28 [INFO] Watching metadata for policy/v1beta1, Kind=PodDisruptionBudget
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting
2021/08/08 12:49:28 [INFO] Watching metadata for cert-manager.io/v1, Kind=ClusterIssuer
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline
2021/08/08 12:49:28 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition
2021/08/08 12:49:28 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkSet
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMConfig
2021/08/08 12:49:28 [INFO] Watching metadata for apps/v1, Kind=DaemonSet
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepo
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema
2021/08/08 12:49:28 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=App
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPConfiguration
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog
2021/08/08 12:49:28 [INFO] Watching metadata for cert-manager.io/v1, Kind=Certificate
2021/08/08 12:49:28 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=ServiceAccount
2021/08/08 12:49:28 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=Project
2021/08/08 12:49:28 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepoRestriction
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption
2021/08/08 12:49:28 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate
2021/08/08 12:49:28 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPPeer
2021/08/08 12:49:28 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMHandle
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging
2021/08/08 12:49:28 [INFO] Watching metadata for gitjob.cattle.io/v1, Kind=GitJob
2021/08/08 12:49:28 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup
2021/08/08 12:49:28 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential
2021/08/08 12:49:28 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision
I0808 12:49:38.802997       7 trace.go:205] Trace[1819992954]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168 (08-Aug-2021 12:49:28.725) (total time: 10077ms):
Trace[1819992954]: ---"Objects listed" 10077ms (12:49:00.802)
Trace[1819992954]: [10.077395935s] [10.077395935s] END
2021/08/08 12:50:04 [INFO] Listening on :443
2021/08/08 12:50:04 [INFO] Listening on :80
2021/08/08 12:50:04 [INFO] Listening on :444
2021/08/08 12:50:04 [INFO] Updating TLS secret for serving-cert (count: 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:f3a4b6f81a3e624a595617f5e18f34a1a0186c063d72b1262fd09343af54a325]
2021/08/08 12:50:05 [INFO] Active TLS secret serving-cert (ver=36375) (count 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:05 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:50:05 [INFO] Active TLS secret serving-cert (ver=36375) (count 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:07 [INFO] Updating TLS secret for serving-cert (count: 9): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:cbb8784b086c09c2d80b5a2c202cd7e584e513ad03f6a216463a80a2e76d0558]
2021/08/08 12:50:07 [INFO] Active TLS secret serving-cert (ver=36388) (count 9): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:cbb8784b086c09c2d80b5a2c202cd7e584e513ad03f6a216463a80a2e76d0558]
2021/08/08 12:50:07 [INFO] Active TLS secret serving-cert (ver=36388) (count 9): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:cbb8784b086c09c2d80b5a2c202cd7e584e513ad03f6a216463a80a2e76d0558]
2021/08/08 12:50:18 [INFO] Adding peer wss://10.42.0.33/v3/connect, 10.42.0.33
2021/08/08 12:50:19 [ERROR] Failed to connect to peer wss://10.42.0.33/v3/connect [local ID=10.42.0.34]: websocket: bad handshake
2021/08/08 12:50:24 [ERROR] Failed to connect to peer wss://10.42.0.33/v3/connect [local ID=10.42.0.34]: websocket: bad handshake
2021/08/08 12:50:24 [INFO] Handling backend connection request [10.42.0.33]
2021/08/08 12:50:30 [INFO] Active TLS secret serving-cert (ver=36529) (count 10): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.39:10.42.0.39 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:886488085d31e3ebc49f83d909f43c692a45719f4293c566b41e72dd78dcf92c]
2021/08/08 12:50:31 [INFO] Adding peer wss://10.42.0.39/v3/connect, 10.42.0.39
2021/08/08 12:50:31 [INFO] Stopping cluster agent for local
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
W0808 12:50:31.013667       7 reflector.go:437] pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168: watch of *v1.ClusterRole ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0808 12:50:31.013760       7 reflector.go:437] pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168: watch of *v1.APIService ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Pod workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Endpoints workers
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding workers
2021/08/08 12:50:31 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Service workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Namespace workers
2021/08/08 12:50:31 [INFO] Registering project network policy
2021/08/08 12:50:31 [INFO] Not registering CIS controller for non RKE cluster: local
2021/08/08 12:50:31 [INFO] registering podsecuritypolicy cluster handler for cluster local
2021/08/08 12:50:31 [INFO] registering podsecuritypolicy project handler for cluster local
2021/08/08 12:50:31 [INFO] registering podsecuritypolicy namespace handler for cluster local
2021/08/08 12:50:31 [INFO] registering podsecuritypolicy serviceaccount handler for cluster local
2021/08/08 12:50:31 [INFO] registering podsecuritypolicy template handler for cluster local
2021/08/08 12:50:31 [INFO] Registering monitoring for cluster "local"
2021/08/08 12:50:31 [INFO] Registering istio for cluster "local"
2021/08/08 12:50:32 [INFO] Registering namespaceHandler for adding labels
2021/08/08 12:50:32 [INFO] Starting cluster controllers for local
2021/08/08 12:50:32 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeProviderConfig controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ProjectAlertGroup controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ClusterScan controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ClusterLogging controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=Notifier controller
2021/08/08 12:50:32 [INFO] Starting project.cattle.io/v3, Kind=PipelineSetting controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertGroup controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ProjectAlert controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ProjectAlertRule controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ProjectNetworkPolicy controller
2021/08/08 12:50:32 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertRule controller
2021/08/08 12:50:33 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:50:33 [INFO] Starting batch/v1, Kind=Job controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=LimitRange controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Service controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:50:33 [INFO] Starting monitoring.coreos.com/v1, Kind=Prometheus controller
2021/08/08 12:50:33 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:50:33 [INFO] Starting cluster agent for local [owner=true]
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=ReplicationController controller
2021/08/08 12:50:33 [INFO] Starting apps/v1, Kind=DaemonSet controller
2021/08/08 12:50:33 [INFO] Starting extensions/v1beta1, Kind=Ingress controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=ConfigMap controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=ResourceQuota controller
2021/08/08 12:50:33 [INFO] Starting apps/v1, Kind=Deployment controller
2021/08/08 12:50:33 [INFO] Starting apps/v1, Kind=ReplicaSet controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Node controller
2021/08/08 12:50:33 [INFO] Starting policy/v1beta1, Kind=PodSecurityPolicy controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Event controller
2021/08/08 12:50:33 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:50:33 [INFO] Starting apps/v1, Kind=StatefulSet controller
2021/08/08 12:50:33 [INFO] Starting batch/v1beta1, Kind=CronJob controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:50:33 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=ServiceAccount controller
2021/08/08 12:50:33 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:50:33 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:50:33 [INFO] Starting networking.k8s.io/v1, Kind=NetworkPolicy controller
2021/08/08 12:50:33 [INFO] Starting monitoring.coreos.com/v1, Kind=ServiceMonitor controller
2021/08/08 12:50:34 [INFO] Handling backend connection request [10.42.0.39]

for rancher-cbbdf9fd5-nj99v

$ kubectl -n cattle-system logs rancher-cbbdf9fd5-nj99v
2021/08/08 12:48:45 [INFO] Rancher version v2.5.9 (3c5418944) is starting
2021/08/08 12:48:45 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Agent:false Features: ClusterRegistry:}
2021/08/08 12:48:45 [INFO] Listening on /tmp/log.sock
2021/08/08 12:48:52 [INFO] Starting API controllers
2021/08/08 12:48:52 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2021/08/08 12:48:52 [INFO] Starting management.cattle.io/v3, Kind=User controller
2021/08/08 12:48:52 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2021/08/08 12:48:52 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2021/08/08 12:48:52 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:48:52 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
I0808 12:48:54.061007       7 leaderelection.go:243] attempting to acquire leader lease kube-system/cattle-controllers...
2021/08/08 12:48:54 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:48:54 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:48:54 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:48:54 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
2021/08/08 12:48:54 [INFO] Starting apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition controller
2021/08/08 12:48:54 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:48:54 [INFO] Starting catalog.cattle.io/v1, Kind=ClusterRepo controller
2021/08/08 12:48:54 [INFO] Starting management.cattle.io/v3, Kind=Setting controller
2021/08/08 12:48:54 [INFO] Starting /v1, Kind=ConfigMap controller
2021/08/08 12:48:54 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:48:54 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:48:54 [INFO] Starting management.cattle.io/v3, Kind=Feature controller
2021/08/08 12:48:54 [INFO] Starting management.cattle.io/v3, Kind=Preference controller
2021/08/08 12:48:54 [INFO] Refreshing all schemas
2021/08/08 12:48:58 [INFO] Refreshing all schemas
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Binding
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind ComponentStatus
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind ConfigMap
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Endpoints
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Event
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind LimitRange
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Namespace
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Node
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind PersistentVolumeClaim
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind PersistentVolume
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Pod
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind PodTemplate
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind ReplicationController
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind ResourceQuota
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Secret
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind ServiceAccount
2021/08/08 12:48:59 [INFO] APIVersion /v1 Kind Service
2021/08/08 12:48:59 [INFO] APIVersion apiregistration.k8s.io/v1 Kind APIService
2021/08/08 12:48:59 [INFO] APIVersion apiregistration.k8s.io/v1beta1 Kind APIService
2021/08/08 12:48:59 [INFO] APIVersion extensions/v1beta1 Kind Ingress
2021/08/08 12:48:59 [INFO] APIVersion apps/v1 Kind ControllerRevision
2021/08/08 12:48:59 [INFO] APIVersion apps/v1 Kind DaemonSet
2021/08/08 12:48:59 [INFO] APIVersion apps/v1 Kind Deployment
2021/08/08 12:48:59 [INFO] APIVersion apps/v1 Kind ReplicaSet
2021/08/08 12:48:59 [INFO] APIVersion apps/v1 Kind StatefulSet
2021/08/08 12:48:59 [INFO] APIVersion events.k8s.io/v1beta1 Kind Event
2021/08/08 12:48:59 [INFO] APIVersion authentication.k8s.io/v1 Kind TokenReview
2021/08/08 12:48:59 [INFO] APIVersion authentication.k8s.io/v1beta1 Kind TokenReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1 Kind LocalSubjectAccessReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectAccessReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectRulesReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1 Kind SubjectAccessReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind LocalSubjectAccessReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectAccessReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectRulesReview
2021/08/08 12:48:59 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SubjectAccessReview
2021/08/08 12:48:59 [INFO] APIVersion autoscaling/v1 Kind HorizontalPodAutoscaler
2021/08/08 12:48:59 [INFO] APIVersion autoscaling/v2beta1 Kind HorizontalPodAutoscaler
2021/08/08 12:48:59 [INFO] APIVersion autoscaling/v2beta2 Kind HorizontalPodAutoscaler
2021/08/08 12:48:59 [INFO] APIVersion batch/v1 Kind Job
2021/08/08 12:48:59 [INFO] APIVersion batch/v1beta1 Kind CronJob
2021/08/08 12:48:59 [INFO] APIVersion certificates.k8s.io/v1beta1 Kind CertificateSigningRequest
2021/08/08 12:48:59 [INFO] APIVersion networking.k8s.io/v1 Kind NetworkPolicy
2021/08/08 12:48:59 [INFO] APIVersion networking.k8s.io/v1beta1 Kind Ingress
2021/08/08 12:48:59 [INFO] APIVersion policy/v1beta1 Kind PodDisruptionBudget
2021/08/08 12:48:59 [INFO] APIVersion policy/v1beta1 Kind PodSecurityPolicy
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRoleBinding
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRole
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind RoleBinding
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind Role
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRoleBinding
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRole
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind RoleBinding
2021/08/08 12:48:59 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind Role
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1 Kind CSINode
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1 Kind StorageClass
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1 Kind VolumeAttachment
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSIDriver
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSINode
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1beta1 Kind StorageClass
2021/08/08 12:48:59 [INFO] APIVersion storage.k8s.io/v1beta1 Kind VolumeAttachment
2021/08/08 12:48:59 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind MutatingWebhookConfiguration
2021/08/08 12:48:59 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind ValidatingWebhookConfiguration
2021/08/08 12:48:59 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind MutatingWebhookConfiguration
2021/08/08 12:48:59 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind ValidatingWebhookConfiguration
2021/08/08 12:48:59 [INFO] APIVersion apiextensions.k8s.io/v1 Kind CustomResourceDefinition
2021/08/08 12:48:59 [INFO] APIVersion apiextensions.k8s.io/v1beta1 Kind CustomResourceDefinition
2021/08/08 12:48:59 [INFO] APIVersion scheduling.k8s.io/v1 Kind PriorityClass
2021/08/08 12:48:59 [INFO] APIVersion scheduling.k8s.io/v1beta1 Kind PriorityClass
2021/08/08 12:48:59 [INFO] APIVersion coordination.k8s.io/v1 Kind Lease
2021/08/08 12:48:59 [INFO] APIVersion coordination.k8s.io/v1beta1 Kind Lease
2021/08/08 12:48:59 [INFO] APIVersion node.k8s.io/v1beta1 Kind RuntimeClass
2021/08/08 12:48:59 [INFO] APIVersion discovery.k8s.io/v1beta1 Kind EndpointSlice
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1 Kind Challenge
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1 Kind Order
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1beta1 Kind Order
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1beta1 Kind Challenge
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1alpha3 Kind Order
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1alpha3 Kind Challenge
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Challenge
2021/08/08 12:48:59 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Order
2021/08/08 12:48:59 [INFO] APIVersion catalog.cattle.io/v1 Kind ClusterRepo
2021/08/08 12:48:59 [INFO] APIVersion catalog.cattle.io/v1 Kind App
2021/08/08 12:48:59 [INFO] APIVersion catalog.cattle.io/v1 Kind Operation
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1 Kind ClusterIssuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1 Kind CertificateRequest
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1 Kind Issuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1 Kind Certificate
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1beta1 Kind CertificateRequest
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1beta1 Kind Issuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1beta1 Kind Certificate
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1beta1 Kind ClusterIssuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha3 Kind Issuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha3 Kind Certificate
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha3 Kind ClusterIssuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha3 Kind CertificateRequest
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Certificate
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha2 Kind ClusterIssuer
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha2 Kind CertificateRequest
2021/08/08 12:48:59 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Issuer
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkSet
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMHandle
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPPeer
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkSet
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkPolicy
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMBlock
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkPolicy
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind FelixConfiguration
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind HostEndpoint
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPPool
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMConfig
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind ClusterInformation
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind BlockAffinity
2021/08/08 12:48:59 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPConfiguration
2021/08/08 12:48:59 [INFO] APIVersion gitjob.cattle.io/v1 Kind GitJob
2021/08/08 12:48:59 [INFO] APIVersion monitoring.coreos.com/v1 Kind PrometheusRule
2021/08/08 12:48:59 [INFO] APIVersion monitoring.coreos.com/v1 Kind Prometheus
2021/08/08 12:48:59 [INFO] APIVersion monitoring.coreos.com/v1 Kind Alertmanager
2021/08/08 12:48:59 [INFO] APIVersion monitoring.coreos.com/v1 Kind ServiceMonitor
2021/08/08 12:48:59 [INFO] APIVersion rancher.cattle.io/v1 Kind Project
2021/08/08 12:48:59 [INFO] APIVersion rancher.cattle.io/v1 Kind RoleTemplate
2021/08/08 12:48:59 [INFO] APIVersion rancher.cattle.io/v1 Kind RoleTemplateBinding
2021/08/08 12:48:59 [INFO] APIVersion rancher.cattle.io/v1 Kind Cluster
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind BundleDeployment
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind BundleNamespaceMapping
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Bundle
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Cluster
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Content
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterRegistration
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind GitRepoRestriction
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterRegistrationToken
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind GitRepo
2021/08/08 12:48:59 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterGroup
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterAppRevision
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sSystemImage
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind EtcdBackup
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRoleBinding
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind RoleTemplate
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Cluster
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Catalog
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterApp
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind DynamicSchema
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind MonitorMetric
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRoleTemplateBinding
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertRule
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Feature
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Token
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlert
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind NodePool
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Project
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind TemplateVersion
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Preference
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplateRevision
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlert
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind RkeAddon
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterMonitorGraph
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Group
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplate
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplate
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterScan
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sServiceOption
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Setting
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectNetworkPolicy
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind SamlToken
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectLogging
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplate
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind KontainerDriver
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRole
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind CisBenchmarkVersion
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertGroup
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterCatalog
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectCatalog
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind TemplateContent
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind GroupMember
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind User
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind UserAttribute
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind FleetWorkspace
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplateProjectBinding
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterLogging
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertGroup
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Template
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ComposeConfig
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind CisConfig
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind NodeTemplate
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRegistrationToken
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Notifier
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind AuthConfig
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDnsProvider
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectMonitorGraph
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertRule
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind ProjectRoleTemplateBinding
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind NodeDriver
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind Node
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDns
2021/08/08 12:48:59 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplateVersion
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind App
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/08/08 12:48:59 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/08/08 12:48:59 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/08/08 12:48:59 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics
2021/08/08 12:49:01 [INFO] Running in clustered mode with ID 10.42.0.39, monitoring endpoint cattle-system/rancher
2021/08/08 12:49:11 [INFO] Starting cluster controllers for local
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplate controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2021/08/08 12:49:15 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeRepository controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=GlobalDns controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ClusterRegistrationToken controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sServiceOption controller
2021/08/08 12:49:15 [INFO] Starting project.cattle.io/v3, Kind=App controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplate controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=AuthConfig controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=KontainerDriver controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2021/08/08 12:49:15 [INFO] Starting project.cattle.io/v3, Kind=PipelineExecution controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplateVersion controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterAppRevision controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=RkeAddon controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=GlobalRole controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=CisBenchmarkVersion controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=User controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sSystemImage controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=CisConfig controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ClusterCatalog controller
2021/08/08 12:49:15 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=NodeTemplate controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=Project controller
2021/08/08 12:49:15 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=DynamicSchema controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ProjectLogging controller
2021/08/08 12:49:15 [INFO] Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller
2021/08/08 12:49:16 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeCredential controller
2021/08/08 12:49:16 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterApp controller
2021/08/08 12:49:16 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2021/08/08 12:49:16 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding controller
2021/08/08 12:49:16 [INFO] Starting management.cattle.io/v3, Kind=NodeDriver controller
2021/08/08 12:49:16 [INFO] Starting management.cattle.io/v3, Kind=Catalog controller
2021/08/08 12:49:16 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplate controller
2021/08/08 12:49:16 [INFO] Starting project.cattle.io/v3, Kind=Pipeline controller
2021/08/08 12:49:20 [INFO] Starting cluster agent for local [owner=false]
2021/08/08 12:49:20 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:49:20 [INFO] Starting /v1, Kind=Service controller
2021/08/08 12:49:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:49:20 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:49:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:49:20 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:49:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:49:20 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:49:20 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate
2021/08/08 12:49:22 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=RoleTemplate
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember
2021/08/08 12:49:22 [INFO] Watching metadata for apps/v1, Kind=Deployment
2021/08/08 12:49:22 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Order
2021/08/08 12:49:22 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BlockAffinity
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema
2021/08/08 12:49:22 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterGroup
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert
2021/08/08 12:49:22 [INFO] Watching metadata for /v1, Kind=ResourceQuota
2021/08/08 12:49:22 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting
2021/08/08 12:49:22 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster
2021/08/08 12:49:22 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepo
2021/08/08 12:49:22 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus
2021/08/08 12:49:22 [INFO] Watching metadata for cert-manager.io/v1, Kind=CertificateRequest
2021/08/08 12:49:22 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding
2021/08/08 12:49:22 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference
2021/08/08 12:49:22 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
2021/08/08 12:49:22 [INFO] Watching metadata for /v1, Kind=LimitRange
2021/08/08 12:49:22 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepoRestriction
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting
2021/08/08 12:49:22 [INFO] Watching metadata for networking.k8s.io/v1beta1, Kind=Ingress
2021/08/08 12:49:22 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=ClusterRepo
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature
2021/08/08 12:49:22 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=FelixConfiguration
2021/08/08 12:49:22 [INFO] Watching metadata for batch/v1, Kind=Job
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=FleetWorkspace
2021/08/08 12:49:22 [INFO] Watching metadata for /v1, Kind=ConfigMap
2021/08/08 12:49:22 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPConfiguration
2021/08/08 12:49:22 [INFO] Watching metadata for /v1, Kind=Pod
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
2021/08/08 12:49:22 [INFO] Watching metadata for storage.k8s.io/v1beta1, Kind=CSIDriver
2021/08/08 12:49:22 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass
2021/08/08 12:49:22 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
2021/08/08 12:49:22 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPPool
2021/08/08 12:49:22 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkSet
2021/08/08 12:49:22 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Content
2021/08/08 12:49:22 [INFO] Watching metadata for /v1, Kind=Secret
2021/08/08 12:49:22 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition
2021/08/08 12:49:22 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule
2021/08/08 12:49:22 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric
2021/08/08 12:49:23 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass
2021/08/08 12:49:23 [INFO] Watching metadata for apps/v1, Kind=StatefulSet
2021/08/08 12:49:23 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=Project
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=Node
2021/08/08 12:49:23 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=RoleTemplateBinding
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole
2021/08/08 12:49:23 [INFO] Watching metadata for cert-manager.io/v1, Kind=ClusterIssuer
2021/08/08 12:49:23 [INFO] Watching metadata for batch/v1beta1, Kind=CronJob
2021/08/08 12:49:23 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
2021/08/08 12:49:23 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role
2021/08/08 12:49:23 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy
2021/08/08 12:49:23 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=PersistentVolume
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=HostEndpoint
2021/08/08 12:49:23 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Challenge
2021/08/08 12:49:23 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistrationToken
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node
2021/08/08 12:49:23 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Cluster
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier
2021/08/08 12:49:23 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkPolicy
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup
2021/08/08 12:49:23 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=Cluster
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate
2021/08/08 12:49:23 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
2021/08/08 12:49:23 [INFO] Watching metadata for cert-manager.io/v1, Kind=Issuer
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkPolicy
2021/08/08 12:49:23 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
2021/08/08 12:49:23 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleDeployment
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=Service
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup
2021/08/08 12:49:23 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkSet
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig
2021/08/08 12:49:23 [INFO] Watching metadata for node.k8s.io/v1beta1, Kind=RuntimeClass
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=PodTemplate
2021/08/08 12:49:23 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=ReplicationController
2021/08/08 12:49:23 [INFO] Watching metadata for discovery.k8s.io/v1beta1, Kind=EndpointSlice
2021/08/08 12:49:23 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleNamespaceMapping
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group
2021/08/08 12:49:23 [INFO] Watching metadata for certificates.k8s.io/v1beta1, Kind=CertificateSigningRequest
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=Namespace
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=ClusterInformation
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=Event
2021/08/08 12:49:23 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=App
2021/08/08 12:49:23 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider
2021/08/08 12:49:23 [INFO] Watching metadata for policy/v1beta1, Kind=PodDisruptionBudget
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMConfig
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=Endpoints
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMHandle
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool
2021/08/08 12:49:23 [INFO] Watching metadata for apps/v1, Kind=DaemonSet
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMBlock
2021/08/08 12:49:23 [INFO] Watching metadata for events.k8s.io/v1beta1, Kind=Event
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=User
2021/08/08 12:49:23 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=App
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog
2021/08/08 12:49:23 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding
2021/08/08 12:49:23 [INFO] Watching metadata for /v1, Kind=ServiceAccount
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging
2021/08/08 12:49:23 [INFO] Watching metadata for autoscaling/v2beta2, Kind=HorizontalPodAutoscaler
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig
2021/08/08 12:49:23 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Bundle
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule
2021/08/08 12:49:23 [INFO] Watching metadata for gitjob.cattle.io/v1, Kind=GitJob
2021/08/08 12:49:23 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=Operation
2021/08/08 12:49:23 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline
2021/08/08 12:49:23 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPPeer
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert
2021/08/08 12:49:23 [INFO] Watching metadata for cert-manager.io/v1, Kind=Certificate
2021/08/08 12:49:23 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy
2021/08/08 12:50:04 [INFO] Listening on :443
2021/08/08 12:50:04 [INFO] Listening on :80
2021/08/08 12:50:04 [INFO] Listening on :444
2021/08/08 12:50:04 [INFO] Updating TLS secret for serving-cert (count: 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.39:10.42.0.39 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:977d70d9405a56ac2287891f4372a5623e689d54cc67c12a4413af9182c58173]
2021/08/08 12:50:05 [INFO] Active TLS secret serving-cert (ver=36375) (count 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:05 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:50:05 [INFO] Active TLS secret serving-cert (ver=36375) (count 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:07 [INFO] Active TLS secret serving-cert (ver=36388) (count 9): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:cbb8784b086c09c2d80b5a2c202cd7e584e513ad03f6a216463a80a2e76d0558]
2021/08/08 12:50:18 [INFO] Adding peer wss://10.42.0.33/v3/connect, 10.42.0.33
2021/08/08 12:50:19 [ERROR] Failed to connect to peer wss://10.42.0.33/v3/connect [local ID=10.42.0.39]: websocket: bad handshake
2021/08/08 12:50:24 [ERROR] Failed to connect to peer wss://10.42.0.33/v3/connect [local ID=10.42.0.39]: websocket: bad handshake
2021/08/08 12:50:24 [INFO] Adding peer wss://10.42.0.34/v3/connect, 10.42.0.34
2021/08/08 12:50:24 [ERROR] Failed to connect to peer wss://10.42.0.34/v3/connect [local ID=10.42.0.39]: websocket: bad handshake
2021/08/08 12:50:29 [ERROR] Failed to connect to peer wss://10.42.0.33/v3/connect [local ID=10.42.0.39]: websocket: bad handshake
2021/08/08 12:50:29 [ERROR] Failed to connect to peer wss://10.42.0.34/v3/connect [local ID=10.42.0.39]: websocket: bad handshake
2021/08/08 12:50:30 [INFO] Updating TLS secret for serving-cert (count: 10): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.39:10.42.0.39 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:886488085d31e3ebc49f83d909f43c692a45719f4293c566b41e72dd78dcf92c]
2021/08/08 12:50:30 [INFO] Active TLS secret serving-cert (ver=36529) (count 10): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.39:10.42.0.39 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:886488085d31e3ebc49f83d909f43c692a45719f4293c566b41e72dd78dcf92c]
2021/08/08 12:50:30 [INFO] Active TLS secret serving-cert (ver=36529) (count 10): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.39:10.42.0.39 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:886488085d31e3ebc49f83d909f43c692a45719f4293c566b41e72dd78dcf92c]
2021/08/08 12:50:31 [INFO] Handling backend connection request [10.42.0.34]
2021/08/08 12:50:31 [INFO] Handling backend connection request [10.42.0.33]

and for rancher-cbbdf9fd5-pmz5b

$ kubectl -n cattle-system logs rancher-cbbdf9fd5-pmz5b
2021/08/08 12:48:48 [INFO] Rancher version v2.5.9 (3c5418944) is starting
2021/08/08 12:48:48 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:true AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Agent:false Features: ClusterRegistry:}
2021/08/08 12:48:48 [INFO] Listening on /tmp/log.sock
2021/08/08 12:49:00 [INFO] Starting API controllers
2021/08/08 12:49:01 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2021/08/08 12:49:01 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2021/08/08 12:49:01 [INFO] Starting management.cattle.io/v3, Kind=User controller
2021/08/08 12:49:01 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2021/08/08 12:49:01 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2021/08/08 12:49:01 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:49:03 [INFO] Starting management.cattle.io/v3, Kind=Preference controller
2021/08/08 12:49:03 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:49:03 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:49:03 [INFO] Starting management.cattle.io/v3, Kind=Cluster controller
I0808 12:49:03.891804       7 leaderelection.go:243] attempting to acquire leader lease kube-system/cattle-controllers...
2021/08/08 12:49:03 [INFO] Starting apiextensions.k8s.io/v1beta1, Kind=CustomResourceDefinition controller
2021/08/08 12:49:03 [INFO] Starting /v1, Kind=ConfigMap controller
2021/08/08 12:49:03 [INFO] Starting catalog.cattle.io/v1, Kind=ClusterRepo controller
2021/08/08 12:49:03 [INFO] Starting management.cattle.io/v3, Kind=Feature controller
2021/08/08 12:49:03 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:49:03 [INFO] Starting management.cattle.io/v3, Kind=Setting controller
2021/08/08 12:49:03 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:49:03 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:49:03 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
I0808 12:49:04.048828       7 leaderelection.go:253] successfully acquired lease kube-system/cattle-controllers
2021/08/08 12:49:04 [INFO] Refreshing all schemas
2021/08/08 12:49:06 [INFO] Steve auth startup complete
2021/08/08 12:49:09 [INFO] Refreshing all schemas
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Binding
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind ComponentStatus
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind ConfigMap
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Endpoints
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Event
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind LimitRange
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Namespace
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Node
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind PersistentVolumeClaim
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind PersistentVolume
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Pod
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind PodTemplate
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind ReplicationController
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind ResourceQuota
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Secret
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind ServiceAccount
2021/08/08 12:49:09 [INFO] APIVersion /v1 Kind Service
2021/08/08 12:49:09 [INFO] APIVersion apiregistration.k8s.io/v1 Kind APIService
2021/08/08 12:49:09 [INFO] APIVersion apiregistration.k8s.io/v1beta1 Kind APIService
2021/08/08 12:49:09 [INFO] APIVersion extensions/v1beta1 Kind Ingress
2021/08/08 12:49:09 [INFO] APIVersion apps/v1 Kind ControllerRevision
2021/08/08 12:49:09 [INFO] APIVersion apps/v1 Kind DaemonSet
2021/08/08 12:49:09 [INFO] APIVersion apps/v1 Kind Deployment
2021/08/08 12:49:09 [INFO] APIVersion apps/v1 Kind ReplicaSet
2021/08/08 12:49:09 [INFO] APIVersion apps/v1 Kind StatefulSet
2021/08/08 12:49:09 [INFO] APIVersion events.k8s.io/v1beta1 Kind Event
2021/08/08 12:49:09 [INFO] APIVersion authentication.k8s.io/v1 Kind TokenReview
2021/08/08 12:49:09 [INFO] APIVersion authentication.k8s.io/v1beta1 Kind TokenReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1 Kind LocalSubjectAccessReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectAccessReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1 Kind SelfSubjectRulesReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1 Kind SubjectAccessReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind LocalSubjectAccessReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectAccessReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SelfSubjectRulesReview
2021/08/08 12:49:09 [INFO] APIVersion authorization.k8s.io/v1beta1 Kind SubjectAccessReview
2021/08/08 12:49:09 [INFO] APIVersion autoscaling/v1 Kind HorizontalPodAutoscaler
2021/08/08 12:49:09 [INFO] APIVersion autoscaling/v2beta1 Kind HorizontalPodAutoscaler
2021/08/08 12:49:09 [INFO] APIVersion autoscaling/v2beta2 Kind HorizontalPodAutoscaler
2021/08/08 12:49:09 [INFO] APIVersion batch/v1 Kind Job
2021/08/08 12:49:09 [INFO] APIVersion batch/v1beta1 Kind CronJob
2021/08/08 12:49:09 [INFO] APIVersion certificates.k8s.io/v1beta1 Kind CertificateSigningRequest
2021/08/08 12:49:09 [INFO] APIVersion networking.k8s.io/v1 Kind NetworkPolicy
2021/08/08 12:49:09 [INFO] APIVersion networking.k8s.io/v1beta1 Kind Ingress
2021/08/08 12:49:09 [INFO] APIVersion policy/v1beta1 Kind PodDisruptionBudget
2021/08/08 12:49:09 [INFO] APIVersion policy/v1beta1 Kind PodSecurityPolicy
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRoleBinding
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind ClusterRole
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind RoleBinding
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1 Kind Role
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRoleBinding
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind ClusterRole
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind RoleBinding
2021/08/08 12:49:09 [INFO] APIVersion rbac.authorization.k8s.io/v1beta1 Kind Role
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1 Kind CSINode
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1 Kind StorageClass
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1 Kind VolumeAttachment
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSIDriver
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1beta1 Kind CSINode
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1beta1 Kind StorageClass
2021/08/08 12:49:09 [INFO] APIVersion storage.k8s.io/v1beta1 Kind VolumeAttachment
2021/08/08 12:49:09 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind MutatingWebhookConfiguration
2021/08/08 12:49:09 [INFO] APIVersion admissionregistration.k8s.io/v1 Kind ValidatingWebhookConfiguration
2021/08/08 12:49:09 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind MutatingWebhookConfiguration
2021/08/08 12:49:09 [INFO] APIVersion admissionregistration.k8s.io/v1beta1 Kind ValidatingWebhookConfiguration
2021/08/08 12:49:09 [INFO] APIVersion apiextensions.k8s.io/v1 Kind CustomResourceDefinition
2021/08/08 12:49:09 [INFO] APIVersion apiextensions.k8s.io/v1beta1 Kind CustomResourceDefinition
2021/08/08 12:49:09 [INFO] APIVersion scheduling.k8s.io/v1 Kind PriorityClass
2021/08/08 12:49:09 [INFO] APIVersion scheduling.k8s.io/v1beta1 Kind PriorityClass
2021/08/08 12:49:09 [INFO] APIVersion coordination.k8s.io/v1 Kind Lease
2021/08/08 12:49:09 [INFO] APIVersion coordination.k8s.io/v1beta1 Kind Lease
2021/08/08 12:49:09 [INFO] APIVersion node.k8s.io/v1beta1 Kind RuntimeClass
2021/08/08 12:49:09 [INFO] APIVersion discovery.k8s.io/v1beta1 Kind EndpointSlice
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1 Kind Challenge
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1 Kind Order
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1beta1 Kind Order
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1beta1 Kind Challenge
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1alpha3 Kind Order
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1alpha3 Kind Challenge
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Challenge
2021/08/08 12:49:09 [INFO] APIVersion acme.cert-manager.io/v1alpha2 Kind Order
2021/08/08 12:49:09 [INFO] APIVersion catalog.cattle.io/v1 Kind ClusterRepo
2021/08/08 12:49:09 [INFO] APIVersion catalog.cattle.io/v1 Kind App
2021/08/08 12:49:09 [INFO] APIVersion catalog.cattle.io/v1 Kind Operation
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1 Kind ClusterIssuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1 Kind CertificateRequest
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1 Kind Issuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1 Kind Certificate
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1beta1 Kind CertificateRequest
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1beta1 Kind Issuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1beta1 Kind Certificate
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1beta1 Kind ClusterIssuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha3 Kind Issuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha3 Kind Certificate
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha3 Kind ClusterIssuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha3 Kind CertificateRequest
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Certificate
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha2 Kind ClusterIssuer
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha2 Kind CertificateRequest
2021/08/08 12:49:09 [INFO] APIVersion cert-manager.io/v1alpha2 Kind Issuer
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkSet
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMHandle
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPPeer
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkSet
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind NetworkPolicy
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMBlock
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind GlobalNetworkPolicy
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind FelixConfiguration
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind HostEndpoint
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPPool
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind IPAMConfig
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind ClusterInformation
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind BlockAffinity
2021/08/08 12:49:09 [INFO] APIVersion crd.projectcalico.org/v1 Kind BGPConfiguration
2021/08/08 12:49:09 [INFO] APIVersion gitjob.cattle.io/v1 Kind GitJob
2021/08/08 12:49:09 [INFO] APIVersion monitoring.coreos.com/v1 Kind PrometheusRule
2021/08/08 12:49:09 [INFO] APIVersion monitoring.coreos.com/v1 Kind Prometheus
2021/08/08 12:49:09 [INFO] APIVersion monitoring.coreos.com/v1 Kind Alertmanager
2021/08/08 12:49:09 [INFO] APIVersion monitoring.coreos.com/v1 Kind ServiceMonitor
2021/08/08 12:49:09 [INFO] APIVersion rancher.cattle.io/v1 Kind Project
2021/08/08 12:49:09 [INFO] APIVersion rancher.cattle.io/v1 Kind RoleTemplate
2021/08/08 12:49:09 [INFO] APIVersion rancher.cattle.io/v1 Kind RoleTemplateBinding
2021/08/08 12:49:09 [INFO] APIVersion rancher.cattle.io/v1 Kind Cluster
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind BundleDeployment
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind BundleNamespaceMapping
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Bundle
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Cluster
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind Content
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterRegistration
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind GitRepoRestriction
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterRegistrationToken
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind GitRepo
2021/08/08 12:49:09 [INFO] APIVersion fleet.cattle.io/v1alpha1 Kind ClusterGroup
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterAppRevision
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sSystemImage
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind EtcdBackup
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRoleBinding
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind RoleTemplate
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Cluster
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Catalog
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind MultiClusterApp
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind DynamicSchema
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind MonitorMetric
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRoleTemplateBinding
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertRule
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Feature
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Token
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlert
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind NodePool
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Project
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind TemplateVersion
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Preference
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplateRevision
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlert
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind RkeAddon
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterMonitorGraph
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Group
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplate
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplate
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterScan
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind RkeK8sServiceOption
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Setting
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectNetworkPolicy
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind SamlToken
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectLogging
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterTemplate
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind KontainerDriver
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind GlobalRole
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind CisBenchmarkVersion
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterAlertGroup
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterCatalog
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectCatalog
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind TemplateContent
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind GroupMember
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind User
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind UserAttribute
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind FleetWorkspace
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind PodSecurityPolicyTemplateProjectBinding
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterLogging
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertGroup
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Template
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ComposeConfig
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind CisConfig
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind NodeTemplate
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ClusterRegistrationToken
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Notifier
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind AuthConfig
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDnsProvider
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectMonitorGraph
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectAlertRule
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind ProjectRoleTemplateBinding
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind NodeDriver
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind Node
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind GlobalDns
2021/08/08 12:49:09 [INFO] APIVersion management.cattle.io/v3 Kind CatalogTemplateVersion
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind App
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind PipelineSetting
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeProviderConfig
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind AppRevision
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind Pipeline
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind PipelineExecution
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeRepository
2021/08/08 12:49:09 [INFO] APIVersion project.cattle.io/v3 Kind SourceCodeCredential
2021/08/08 12:49:09 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind NodeMetrics
2021/08/08 12:49:09 [INFO] APIVersion metrics.k8s.io/v1beta1 Kind PodMetrics
2021/08/08 12:49:11 [INFO] Running in clustered mode with ID 10.42.0.33, monitoring endpoint cattle-system/rancher
2021/08/08 12:49:35 [INFO] Starting cluster controllers for local
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=NodeTemplate controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
2021/08/08 12:49:39 [INFO] Starting project.cattle.io/v3, Kind=PipelineExecution controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ClusterRegistrationToken controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=DynamicSchema controller
2021/08/08 12:49:39 [INFO] Starting project.cattle.io/v3, Kind=Pipeline controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=CisConfig controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterApp controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=Token controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplate controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=NodeDriver controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=AuthConfig controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=KontainerDriver controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=GroupMember controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=UserAttribute controller
2021/08/08 12:49:39 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeRepository controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ClusterCatalog controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=GlobalRole controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=Project controller
2021/08/08 12:49:39 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeCredential controller
2021/08/08 12:49:39 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=User controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=Catalog controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2021/08/08 12:49:39 [INFO] Starting project.cattle.io/v3, Kind=App controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sServiceOption controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=RkeK8sSystemImage controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplate controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=ProjectLogging controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=CisBenchmarkVersion controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplate controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=MultiClusterAppRevision controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=RkeAddon controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=CatalogTemplateVersion controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=GlobalDns controller
2021/08/08 12:49:39 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:49:39 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=FelixConfiguration
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=EtcdBackup
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterLogging
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sServiceOption
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectNetworkPolicy
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Template
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeCredential
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Bundle
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkSet
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=ClusterInformation
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Cluster
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeRepository
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Node
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Catalog
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=ConfigMap
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=DynamicSchema
2021/08/08 12:49:41 [INFO] Watching metadata for apps/v1, Kind=ControllerRevision
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=ServiceAccount
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=MonitorMetric
2021/08/08 12:49:41 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=Operation
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=PodTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMBlock
2021/08/08 12:49:41 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Prometheus
2021/08/08 12:49:41 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration
2021/08/08 12:49:41 [INFO] Watching metadata for policy/v1beta1, Kind=PodSecurityPolicy
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=SamlToken
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Preference
2021/08/08 12:49:41 [INFO] Watching metadata for storage.k8s.io/v1beta1, Kind=CSIDriver
2021/08/08 12:49:41 [INFO] Watching metadata for cert-manager.io/v1, Kind=Issuer
2021/08/08 12:49:41 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=PrometheusRule
2021/08/08 12:49:41 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=Cluster
2021/08/08 12:49:41 [INFO] Watching metadata for certificates.k8s.io/v1beta1, Kind=CertificateSigningRequest
2021/08/08 12:49:41 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=Project
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterApp
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=PersistentVolume
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectLogging
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=RoleTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for networking.k8s.io/v1beta1, Kind=Ingress
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPPool
2021/08/08 12:49:41 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=ClusterRepo
2021/08/08 12:49:41 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Order
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectCatalog
2021/08/08 12:49:41 [INFO] Watching metadata for cert-manager.io/v1, Kind=CertificateRequest
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectMonitorGraph
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=FleetWorkspace
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlert
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleNamespaceMapping
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDnsProvider
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=LimitRange
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=AuthConfig
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodePool
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRole
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPPeer
2021/08/08 12:49:41 [INFO] Watching metadata for apps/v1, Kind=DaemonSet
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMHandle
2021/08/08 12:49:41 [INFO] Watching metadata for batch/v1beta1, Kind=CronJob
2021/08/08 12:49:41 [INFO] Watching metadata for batch/v1, Kind=Job
2021/08/08 12:49:41 [INFO] Watching metadata for admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkSet
2021/08/08 12:49:41 [INFO] Watching metadata for storage.k8s.io/v1, Kind=CSINode
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Service
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Cluster
2021/08/08 12:49:41 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=RoleTemplateBinding
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BlockAffinity
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=MultiClusterAppRevision
2021/08/08 12:49:41 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRole
2021/08/08 12:49:41 [INFO] Watching metadata for cert-manager.io/v1, Kind=ClusterIssuer
2021/08/08 12:49:41 [INFO] Watching metadata for networking.k8s.io/v1, Kind=NetworkPolicy
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterGroup
2021/08/08 12:49:41 [INFO] Watching metadata for apps/v1, Kind=Deployment
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=IPAMConfig
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterCatalog
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=KontainerDriver
2021/08/08 12:49:41 [INFO] Watching metadata for coordination.k8s.io/v1, Kind=Lease
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertGroup
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=User
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalDns
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlert
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=NodeDriver
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterScan
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=NetworkPolicy
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=CatalogTemplateVersion
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Node
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertGroup
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=PersistentVolumeClaim
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=UserAttribute
2021/08/08 12:49:41 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=Role
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineSetting
2021/08/08 12:49:41 [INFO] Watching metadata for apps/v1, Kind=ReplicaSet
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=BundleDeployment
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Endpoints
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Feature
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistrationToken
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ComposeConfig
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for gitjob.cattle.io/v1, Kind=GitJob
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=ResourceQuota
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateContent
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisConfig
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=TemplateVersion
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=BGPConfiguration
2021/08/08 12:49:41 [INFO] Watching metadata for policy/v1beta1, Kind=PodDisruptionBudget
2021/08/08 12:49:41 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=ServiceMonitor
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeK8sSystemImage
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=RkeAddon
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectAlertRule
2021/08/08 12:49:41 [INFO] Watching metadata for scheduling.k8s.io/v1, Kind=PriorityClass
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Secret
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepoRestriction
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Event
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=GroupMember
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=GlobalRoleBinding
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=CisBenchmarkVersion
2021/08/08 12:49:41 [INFO] Watching metadata for autoscaling/v2beta2, Kind=HorizontalPodAutoscaler
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=Pipeline
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=PipelineExecution
2021/08/08 12:49:41 [INFO] Watching metadata for acme.cert-manager.io/v1, Kind=Challenge
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterMonitorGraph
2021/08/08 12:49:41 [INFO] Watching metadata for storage.k8s.io/v1, Kind=VolumeAttachment
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=Content
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=ClusterRegistration
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Notifier
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Namespace
2021/08/08 12:49:41 [INFO] Watching metadata for events.k8s.io/v1beta1, Kind=Event
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Token
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterTemplateRevision
2021/08/08 12:49:41 [INFO] Watching metadata for rancher.cattle.io/v1, Kind=RoleTemplate
2021/08/08 12:49:41 [INFO] Watching metadata for storage.k8s.io/v1, Kind=StorageClass
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=Pod
2021/08/08 12:49:41 [INFO] Watching metadata for fleet.cattle.io/v1alpha1, Kind=GitRepo
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=AppRevision
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Setting
2021/08/08 12:49:41 [INFO] Watching metadata for monitoring.coreos.com/v1, Kind=Alertmanager
2021/08/08 12:49:41 [INFO] Watching metadata for discovery.k8s.io/v1beta1, Kind=EndpointSlice
2021/08/08 12:49:41 [INFO] Watching metadata for /v1, Kind=ReplicationController
2021/08/08 12:49:41 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding
2021/08/08 12:49:41 [INFO] Watching metadata for node.k8s.io/v1beta1, Kind=RuntimeClass
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRoleTemplateBinding
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterRegistrationToken
2021/08/08 12:49:41 [INFO] Watching metadata for apiregistration.k8s.io/v1, Kind=APIService
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Project
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=App
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ProjectRoleTemplateBinding
2021/08/08 12:49:41 [INFO] Watching metadata for catalog.cattle.io/v1, Kind=App
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding
2021/08/08 12:49:41 [INFO] Watching metadata for apiextensions.k8s.io/v1, Kind=CustomResourceDefinition
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=Group
2021/08/08 12:49:41 [INFO] Watching metadata for cert-manager.io/v1, Kind=Certificate
2021/08/08 12:49:41 [INFO] Watching metadata for project.cattle.io/v3, Kind=SourceCodeProviderConfig
2021/08/08 12:49:41 [INFO] Watching metadata for rbac.authorization.k8s.io/v1, Kind=RoleBinding
2021/08/08 12:49:41 [INFO] Watching metadata for management.cattle.io/v3, Kind=ClusterAlertRule
2021/08/08 12:49:41 [INFO] Watching metadata for apps/v1, Kind=StatefulSet
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=GlobalNetworkPolicy
2021/08/08 12:49:41 [INFO] Watching metadata for crd.projectcalico.org/v1, Kind=HostEndpoint
2021/08/08 12:49:50 [INFO] Starting cluster agent for local [owner=false]
2021/08/08 12:49:50 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:49:50 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:49:50 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:49:50 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:49:50 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:49:50 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:49:50 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:49:50 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:49:50 [INFO] Starting /v1, Kind=Service controller
2021/08/08 12:50:04 [INFO] Reconciling GlobalRoles
2021/08/08 12:50:04 [INFO] Reconciling RoleTemplates
2021/08/08 12:50:04 [INFO] Listening on :443
2021/08/08 12:50:04 [INFO] Listening on :80
2021/08/08 12:50:04 [INFO] Listening on :444
2021/08/08 12:50:04 [INFO] Updating TLS secret for serving-cert (count: 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:04 [INFO] Active TLS secret serving-cert (ver=36375) (count 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:05 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:50:05 [INFO] Active TLS secret serving-cert (ver=36375) (count 8): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:429e226403a5b18ac214ac57165b349fcd12a069caef133d72f269769e175315]
2021/08/08 12:50:05 [INFO] adding kontainer driver rancherKubernetesEngine
2021/08/08 12:50:05 [INFO] adding kontainer driver googleKubernetesEngine
2021/08/08 12:50:05 [INFO] adding kontainer driver azureKubernetesService
2021/08/08 12:50:05 [INFO] adding kontainer driver amazonElasticContainerService
2021/08/08 12:50:05 [INFO] adding kontainer driver baiducloudcontainerengine
2021/08/08 12:50:05 [INFO] adding kontainer driver aliyunkubernetescontainerservice
2021/08/08 12:50:05 [INFO] adding kontainer driver tencentkubernetesengine
2021/08/08 12:50:05 [INFO] adding kontainer driver huaweicontainercloudengine
2021/08/08 12:50:05 [INFO] adding kontainer driver oraclecontainerengine
2021/08/08 12:50:05 [INFO] adding kontainer driver linodekubernetesengine
2021/08/08 12:50:05 [INFO] adding kontainer driver opentelekomcloudcontainerengine
2021/08/08 12:50:05 [INFO] Starting catalog controller
2021/08/08 12:50:05 [INFO] Starting project-level catalog controller
2021/08/08 12:50:05 [INFO] Starting cluster-level catalog controller
2021/08/08 12:50:06 [INFO] Refreshing driverMetadata in 1440 minutes
2021/08/08 12:50:06 [INFO] update kontainerdriver opentelekomcloudcontainerengine
2021/08/08 12:50:06 [INFO] update kontainerdriver oraclecontainerengine
2021/08/08 12:50:06 [INFO] update kontainerdriver rancherkubernetesengine
2021/08/08 12:50:06 [INFO] update kontainerdriver amazonelasticcontainerservice
2021/08/08 12:50:06 [INFO] update kontainerdriver baiducloudcontainerengine
2021/08/08 12:50:06 [INFO] update kontainerdriver googlekubernetesengine
2021/08/08 12:50:06 [INFO] update kontainerdriver linodekubernetesengine
2021/08/08 12:50:06 [INFO] update kontainerdriver tencentkubernetesengine
2021/08/08 12:50:06 [INFO] update kontainerdriver aliyunkubernetescontainerservice
2021/08/08 12:50:06 [INFO] update kontainerdriver azurekubernetesservice
2021/08/08 12:50:06 [INFO] update kontainerdriver huaweicontainercloudengine
2021/08/08 12:50:06 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2021/08/08 12:50:06 [ERROR] error parsing azure-group-cache-size, skipping update strconv.Atoi: parsing "": invalid syntax
2021/08/08 12:50:07 [INFO] Active TLS secret serving-cert (ver=36388) (count 9): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:cbb8784b086c09c2d80b5a2c202cd7e584e513ad03f6a216463a80a2e76d0558]
2021/08/08 12:50:07 [INFO] Starting catalog.cattle.io/v1, Kind=App controller
2021/08/08 12:50:07 [INFO] Starting management.cattle.io/v3, Kind=ComposeConfig controller
2021/08/08 12:50:07 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertRule controller
2021/08/08 12:50:07 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:50:07 [INFO] Starting management.cattle.io/v3, Kind=EtcdBackup controller
2021/08/08 12:50:07 [INFO] Starting management.cattle.io/v3, Kind=GlobalDnsProvider controller
2021/08/08 12:50:07 [INFO] Starting management.cattle.io/v3, Kind=ClusterAlertGroup controller
2021/08/08 12:50:07 [INFO] Starting apps/v1, Kind=DaemonSet controller
2021/08/08 12:50:07 [INFO] Starting catalog.cattle.io/v1, Kind=Operation controller
2021/08/08 12:50:07 [INFO] Starting management.cattle.io/v3, Kind=ClusterScan controller
2021/08/08 12:50:07 [INFO] Rancher startup complete
2021/08/08 12:50:07 [INFO] CleanupDuplicateBindings, checking configmap
2021/08/08 12:50:07 [INFO] bindings cleanup already ran before, not calling again
2021/08/08 12:50:09 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.5/data.json
2021/08/08 12:50:09 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
2021/08/08 12:50:10 [INFO] Refreshing driverMetadata in 1440 minutes
2021/08/08 12:50:11 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:34217
2021/08/08 12:50:11 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:32817
2021/08/08 12:50:11 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:37729
2021/08/08 12:50:11 [INFO] kontainerdriver amazonelasticcontainerservice stopped
2021/08/08 12:50:11 [INFO] dynamic schema for kontainerdriver amazonelasticcontainerservice updating
2021/08/08 12:50:11 [INFO] kontainerdriver azurekubernetesservice stopped
2021/08/08 12:50:11 [INFO] dynamic schema for kontainerdriver azurekubernetesservice updating
2021/08/08 12:50:11 [INFO] kontainerdriver googlekubernetesengine stopped
2021/08/08 12:50:11 [INFO] dynamic schema for kontainerdriver googlekubernetesengine updating
2021/08/08 12:50:13 [INFO] driverMetadata: refreshing data from upstream https://releases.rancher.com/kontainer-driver-metadata/release-v2.5/data.json
2021/08/08 12:50:13 [INFO] Retrieve data.json from local path /var/lib/rancher-data/driver-metadata/data.json
2021/08/08 12:50:18 [INFO] Stopping cluster agent for local
2021/08/08 12:50:19 [INFO] Shutting down /v1, Kind=Service workers
2021/08/08 12:50:19 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2021/08/08 12:50:19 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding workers
2021/08/08 12:50:19 [INFO] Shutting down /v1, Kind=Pod workers
2021/08/08 12:50:19 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
2021/08/08 12:50:19 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2021/08/08 12:50:19 [INFO] Shutting down /v1, Kind=Endpoints workers
2021/08/08 12:50:19 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
2021/08/08 12:50:19 [INFO] Shutting down /v1, Kind=Namespace workers
W0808 12:50:19.018088       7 reflector.go:437] pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168: watch of *v1.Role ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2021/08/08 12:50:19 [INFO] Registering project network policy
2021/08/08 12:50:19 [INFO] Not registering CIS controller for non RKE cluster: local
2021/08/08 12:50:19 [INFO] registering podsecuritypolicy cluster handler for cluster local
2021/08/08 12:50:19 [INFO] registering podsecuritypolicy project handler for cluster local
2021/08/08 12:50:19 [INFO] registering podsecuritypolicy namespace handler for cluster local
2021/08/08 12:50:19 [INFO] registering podsecuritypolicy serviceaccount handler for cluster local
2021/08/08 12:50:19 [INFO] registering podsecuritypolicy template handler for cluster local
2021/08/08 12:50:19 [INFO] Registering monitoring for cluster "local"
2021/08/08 12:50:19 [INFO] Registering istio for cluster "local"
2021/08/08 12:50:20 [INFO] Registering namespaceHandler for adding labels
2021/08/08 12:50:20 [INFO] Starting cluster controllers for local
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=SamlToken controller
2021/08/08 12:50:20 [INFO] Starting project.cattle.io/v3, Kind=SourceCodeProviderConfig controller
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=ClusterLogging controller
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=ProjectAlert controller
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=Notifier controller
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=ProjectAlertGroup controller
2021/08/08 12:50:20 [INFO] Starting project.cattle.io/v3, Kind=PipelineSetting controller
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=ProjectNetworkPolicy controller
2021/08/08 12:50:20 [INFO] Starting management.cattle.io/v3, Kind=ProjectAlertRule controller
2021/08/08 12:50:21 [INFO] Starting cluster agent for local [owner=true]
2021/08/08 12:50:21 [INFO] Starting batch/v1beta1, Kind=CronJob controller
2021/08/08 12:50:21 [INFO] Starting extensions/v1beta1, Kind=Ingress controller
2021/08/08 12:50:21 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=LimitRange controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=ConfigMap controller
2021/08/08 12:50:21 [INFO] Starting monitoring.coreos.com/v1, Kind=Prometheus controller
2021/08/08 12:50:21 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:50:21 [INFO] Starting batch/v1, Kind=Job controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=ServiceAccount controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Service controller
2021/08/08 12:50:21 [INFO] Starting networking.k8s.io/v1, Kind=NetworkPolicy controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Secret controller
2021/08/08 12:50:21 [INFO] Starting policy/v1beta1, Kind=PodSecurityPolicy controller
2021/08/08 12:50:21 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:50:21 [INFO] Starting apps/v1, Kind=Deployment controller
2021/08/08 12:50:21 [INFO] Starting apps/v1, Kind=ReplicaSet controller
2021/08/08 12:50:21 [INFO] Starting apps/v1, Kind=StatefulSet controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=ResourceQuota controller
2021/08/08 12:50:21 [INFO] Starting apps/v1, Kind=DaemonSet controller
2021/08/08 12:50:21 [INFO] Updating workload [ingress-nginx/nginx-ingress-controller] with public endpoints [[{"nodeName":"local:machine-fhh7b","addresses":["192.168.157.141"],"port":80,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-2sd2j","allNodes":false},{"nodeName":"local:machine-fhh7b","addresses":["192.168.157.141"],"port":443,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-2sd2j","allNodes":false}]]
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Node controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Event controller
2021/08/08 12:50:21 [INFO] Starting monitoring.coreos.com/v1, Kind=ServiceMonitor controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=ReplicationController controller
2021/08/08 12:50:21 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:50:21 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:50:21 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:50:21 [INFO] Updating service [cm-acme-http-solver-gps6s] with public endpoints [[{"addresses":["192.168.157.141"],"port":32446,"protocol":"TCP","serviceName":"cattle-system:cm-acme-http-solver-gps6s","allNodes":true}]]
2021/08/08 12:50:24 [INFO] Adding peer wss://10.42.0.34/v3/connect, 10.42.0.34
2021/08/08 12:50:29 [INFO] Handling backend connection request [10.42.0.34]
2021/08/08 12:50:30 [INFO] Active TLS secret serving-cert (ver=36529) (count 10): map[field.cattle.io/projectId:local:p-dhzbq listener.cattle.io/cn-10.42.0.10:10.42.0.10 listener.cattle.io/cn-10.42.0.11:10.42.0.11 listener.cattle.io/cn-10.42.0.33:10.42.0.33 listener.cattle.io/cn-10.42.0.34:10.42.0.34 listener.cattle.io/cn-10.42.0.39:10.42.0.39 listener.cattle.io/cn-10.42.0.9:10.42.0.9 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-rancher.cattle-system:rancher.cattle-system listener.cattle.io/hash:886488085d31e3ebc49f83d909f43c692a45719f4293c566b41e72dd78dcf92c]
2021/08/08 12:50:31 [INFO] Adding peer wss://10.42.0.39/v3/connect, 10.42.0.39
2021/08/08 12:50:31 [INFO] Stopping cluster agent for local
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=RoleBinding workers
W0808 12:50:31.032683       7 reflector.go:437] pkg/mod/github.com/rancher/client-go@v1.20.0-rancher.1/tools/cache/reflector.go:168: watch of *v1.StatefulSet ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Namespace workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=ReplicationController workers
2021/08/08 12:50:31 [INFO] Shutting down monitoring.coreos.com/v1, Kind=ServiceMonitor workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Event workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Node workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Pod workers
2021/08/08 12:50:31 [INFO] Shutting down apps/v1, Kind=DaemonSet workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=ResourceQuota workers
2021/08/08 12:50:31 [INFO] Shutting down apps/v1, Kind=StatefulSet workers
2021/08/08 12:50:31 [INFO] Shutting down apps/v1, Kind=ReplicaSet workers
2021/08/08 12:50:31 [INFO] Shutting down apps/v1, Kind=Deployment workers
2021/08/08 12:50:31 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2021/08/08 12:50:31 [INFO] Shutting down policy/v1beta1, Kind=PodSecurityPolicy workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Secret workers
2021/08/08 12:50:31 [INFO] Shutting down batch/v1beta1, Kind=CronJob workers
2021/08/08 12:50:31 [INFO] Shutting down networking.k8s.io/v1, Kind=NetworkPolicy workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Service workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=Endpoints workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=ServiceAccount workers
2021/08/08 12:50:31 [INFO] Shutting down batch/v1, Kind=Job workers
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=Role workers
2021/08/08 12:50:31 [INFO] Shutting down monitoring.coreos.com/v1, Kind=Prometheus workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=ConfigMap workers
2021/08/08 12:50:31 [INFO] Shutting down extensions/v1beta1, Kind=Ingress workers
2021/08/08 12:50:31 [INFO] Shutting down /v1, Kind=LimitRange workers
2021/08/08 12:50:31 [INFO] Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding workers
2021/08/08 12:50:31 [INFO] Starting cluster controllers for local
2021/08/08 12:50:31 [INFO] Starting /v1, Kind=Service controller
2021/08/08 12:50:31 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller
2021/08/08 12:50:31 [INFO] Starting cluster agent for local [owner=false]
2021/08/08 12:50:31 [INFO] Starting /v1, Kind=Endpoints controller
2021/08/08 12:50:31 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller
2021/08/08 12:50:31 [INFO] Starting /v1, Kind=Namespace controller
2021/08/08 12:50:31 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller
2021/08/08 12:50:31 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2021/08/08 12:50:31 [INFO] Starting rbac.authorization.k8s.io/v1, Kind=Role controller
2021/08/08 12:50:31 [INFO] Starting /v1, Kind=Pod controller
2021/08/08 12:50:34 [INFO] Handling backend connection request [10.42.0.39]

I’m sensing that the samething is happening with all 3 pods yet I’m clueless to what is really going on.
I see a handshake failure there, but what could possibly be causing it?

kamal_boumahdi · August 10, 2021, 12:08pm

could it be that I miss configured the load balancer?
I’m using nginx version: nginx/1.20.1
this is the IP of the machine where I ran rke up : 192.168.157.140
and the cluster.yml shows :

$ cat cluster.yml
# If you intened to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
nodes:
- address: 192.168.157.141
  port: "22"
  internal_address: ""
  role:
  - controlplane
  - worker
  - etcd
  hostname_override: fuji1
  user: fuji
  docker_socket: /var/run/docker.sock
  ssh_key: ""
  ssh_key_path: ~/.ssh/id_rsa
  ssh_cert: ""
  ssh_cert_path: ""
  labels: {}
  taints: []
services:
  etcd:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    uid: 0
    gid: 0
    snapshot: null
    retention: ""
    creation: ""
    backup_config: null
  kube-api:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: ""
    pod_security_policy: false
    always_pull_images: false
    secrets_encryption_config: null
    audit_log: null
    admission_configuration: null
    event_rate_limit: null
  kube-controller:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  scheduler:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.43.0.10
    fail_swap_on: false
    generate_serving_certificate: false
  kubeproxy:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
network:
  plugin: canal
  options: {}
  mtu: 0
  node_selector: {}
  update_strategy: null
authentication:
  strategy: x509
  sans: []
  webhook: null
addons: ""
addons_include: []
system_images:
  etcd: rancher/coreos-etcd:v3.4.3-rancher1
  alpine: rancher/rke-tools:v0.1.56
  nginx_proxy: rancher/rke-tools:v0.1.56
  cert_downloader: rancher/rke-tools:v0.1.56
  kubernetes_services_sidecar: rancher/rke-tools:v0.1.56
  kubedns: rancher/k8s-dns-kube-dns:1.15.0
  dnsmasq: rancher/k8s-dns-dnsmasq-nanny:1.15.0
  kubedns_sidecar: rancher/k8s-dns-sidecar:1.15.0
  kubedns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1
  coredns: rancher/coredns-coredns:1.6.5
  coredns_autoscaler: rancher/cluster-proportional-autoscaler:1.7.1
  nodelocal: rancher/k8s-dns-node-cache:1.15.7
  kubernetes: rancher/hyperkube:v1.17.4-rancher1
  flannel: rancher/coreos-flannel:v0.11.0-rancher1
  flannel_cni: rancher/flannel-cni:v0.3.0-rancher5
  calico_node: rancher/calico-node:v3.13.0
  calico_cni: rancher/calico-cni:v3.13.0
  calico_controllers: rancher/calico-kube-controllers:v3.13.0
  calico_ctl: rancher/calico-ctl:v2.0.0
  calico_flexvol: rancher/calico-pod2daemon-flexvol:v3.13.0
  canal_node: rancher/calico-node:v3.13.0
  canal_cni: rancher/calico-cni:v3.13.0
  canal_flannel: rancher/coreos-flannel:v0.11.0
  canal_flexvol: rancher/calico-pod2daemon-flexvol:v3.13.0
  weave_node: weaveworks/weave-kube:2.5.2
  weave_cni: weaveworks/weave-npc:2.5.2
  pod_infra_container: rancher/pause:3.1
  ingress: rancher/nginx-ingress-controller:nginx-0.25.1-rancher1
  ingress_backend: rancher/nginx-ingress-controller-defaultbackend:1.5-rancher1
  metrics_server: rancher/metrics-server:v0.3.6
  windows_pod_infra_container: rancher/kubelet-pause:v0.1.3
ssh_key_path: ~/.ssh/id_rsa
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
  mode: rbac
  options: {}
ignore_docker_version: false
kubernetes_version: ""
private_registries: []
ingress:
  provider: ""
  options: {}
  node_selector: {}
  extra_args: {}
  dns_policy: ""
  extra_envs: []
  extra_volumes: []
  extra_volume_mounts: []
  update_strategy: null
cluster_name: ""
cloud_provider:
  name: ""
prefix_path: ""
addon_job_timeout: 0
bastion_host:
  address: ""
  port: ""
  user: ""
  ssh_key: ""
  ssh_key_path: ""
  ssh_cert: ""
  ssh_cert_path: ""
monitoring:
  provider: ""
  options: {}
  node_selector: {}
  update_strategy: null
  replicas: null
restore:
  restore: false
  snapshot_name: ""
dns: null

and this is what I did in nginx :

$ cat  /etc/nginx/nginx.conf
stream {

        upstream TCP80{
                server 192.168.157.141:80;
        }
        upstream TCP443{
                server 192.168.157.141:443;
        }


        server {
                listen 80;
                proxy_pass TCP80;
        }
        server{
                listen 443;
                proxy_pass TCP443;
        }
}
events { }

superseb · August 11, 2021, 10:41am

None of the logs show a restart, to know why its restarting, we need the logs from the pod that exited (this might need kubectl logs -p for previous pod logs. When you are using Rancher, what problems do you encounter?

kamal_boumahdi · August 11, 2021, 10:23pm

I get a blank page when I browse to the host name https://rancher.my.org

superseb · August 12, 2021, 8:28am

All the time or randomly? Simplest way to start debugging is to see what the response is on the command line, for example, curl -vLk https://rancher.my.org and start from there. Also, if you think the load balancer is the issue, exclude it by pointing the DNS to a node directly and see if that works.

The example NGINX configuration for layer 4 can be found here: Rancher Docs: Setting up an NGINX Load Balancer

kamal_boumahdi · August 16, 2021, 1:46am

well the blank page is all the time, the restarts are random.
as for debugging from within, here’s what I get:

~$ curl -vLk https://rancher.my.org
*   Trying 64.190.62.111:443...
* TCP_NODELAY set
* Connected to rancher.my.org (64.190.62.111) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=rancher.my.org
*  start date: Jul 26 00:00:00 2021 GMT
*  expire date: Jul 26 23:59:59 2022 GMT
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=Encryption Everywhere DV TLS CA - G1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x558754f60e10)
> GET / HTTP/2
> Host: rancher.my.org
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 200
< cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< content-type: text/html; charset=UTF-8
< date: Mon, 16 Aug 2021 01:10:27 GMT
< expires: Mon, 26 Jul 1997 05:00:00 GMT
< last-modified: Mon, 16 Aug 2021 01:10:27 GMT
< pragma: no-cache
< server: NginX
< vary: Accept-Encoding
< x-adblock-key: MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBANnylWw2vLY4hUn9w06zQKbhKBfvjFUCsdFlb6TdQhxb9RXWXuI4t31c+o8fYOv/s8q1LGPga3DE1L/tHU4LENMCAwEAAQ==_xd/MGsOJ8/p4oGkEaB6Vmp67y3786AgZqepPA/7wf5voD5M/bARtM6XqJg1ah8THtifPlehNIVapKbYwIsJ52Q==
< x-cache-miss-from: parking-7d65c95449-4j5qd
<
<!DOCTYPE html><html lang="en"  data-adblockkey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBANnylWw2vLY4hUn9w06zQKbhKBfvjFUCsdFlb6TdQhxb9RXWXuI4t31c+o8fYOv/s8q1LGPga3DE1L/tHU4LENMCAwEAAQ==_xd/MGsOJ8/p4oGkEaB6Vmp67y3786AgZqepPA/7wf5voD5M/bARtM6XqJg1ah8THtifPlehNIVapKbYwIsJ52Q==><head><meta charset="utf-8"><title>my.org&nbsp;-&nbspSocial networking المصادر والمعلومات.</title><meta name="viewport" content="width=device-width,initial-scale=1.0,maximum-scale=1.0,user-scalable=0"><meta name="description" content="my.org هل هو أول وأفضل مصادر جميع المعلومات التي تبحث عنها. من ضوء الموضوعات العامة إلى مزيد من الموضوعات التي كنت لتتوقعها هنا، my.org تمتلكها جميعًا. نأمل أن تجد ما تبحث عنه!"><link
        rel="icon"
        type="image/png"
        href="//img.sedoparking.com/templates/logos/sedo_logo.png"
/><style>
        /*! normalize.css v7.0.0 | MIT License | github.com/necolas/normalize.css */html{line-height:1.15;-ms-text-size-adjust:100%;-webkit-text-size-adjust:100%}body{margin:0}article,aside,footer,header,nav,section{display:block}h1{font-size:2em;margin:.67em 0}figcaption,figure,main{display:block}figure{margin:1em 40px}hr{box-sizing:content-box;height:0;overflow:visible}pre{font-family:monospace,monospace;font-size:1em}a{background-color:transparent;-webkit-text-decoration-skip:objects}abbr[title]{border-bottom:none;text-decoration:underline;text-decoration:underline dotted}b,strong{font-weight:inherit}b,strong{font-weight:bolder}code,kbd,samp{font-family:monospace,monospace;font-size:1em}dfn{font-style:italic}mark{background-color:#ff0;color:#000}small{font-size:80%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sub{bottom:-0.25em}sup{top:-0.5em}audio,video{display:inline-block}audio:not([controls]){display:none;height:0}img{border-style:none}svg:not(:root){overflow:hidden}button,input,optgroup,select,textarea{font-family:sans-serif;font-size:100%;line-height:1.15;margin:0}button,input{overflow:visible}button,select{text-transform:none}button,html [type=button],[type=reset],[type=submit]{-webkit-appearance:button}button::-moz-focus-inner,[type=button]::-moz-focus-inner,[type=reset]::-moz-focus-inner,[type=submit]::-moz-focus-inner{border-style:none;padding:0}button:-moz-focusring,[type=button]:-moz-focusring,[type=reset]:-moz-focusring,[type=submit]:-moz-focusring{outline:1px dotted ButtonText}fieldset{padding:.35em .75em .625em}legend{box-sizing:border-box;color:inherit;display:table;max-width:100%;padding:0;white-space:normal}progress{display:inline-block;vertical-align:baseline}textarea{overflow:auto}[type=checkbox],[type=radio]{box-sizing:border-box;padding:0}[type=number]::-webkit-inner-spin-button,[type=number]::-webkit-outer-spin-button{height:auto}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}[type=search]::-webkit-search-cancel-button,[type=search]::-webkit-search-decoration{-webkit-appearance:none}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}details,menu{display:block}summary{display:list-item}canvas{display:inline-block}template{display:none}[hidden]{display:none}.announcement{background:#262626;text-align:center;padding:0 5px}.announcement p{color:#717171}.announcement a{color:#717171}.container-header{margin:0 auto 0 auto;text-align:center}.container-header__content{color:#717171}.container-content{margin:25px auto 20px auto;text-align:center;background:url("//img.sedoparking.com/templates/bg/arrows-1-colors-3.png") #fbfbfb no-repeat center top;background-size:100%}.container-content__container-relatedlinks,.container-content__container-ads,.container-content__webarchive{width:30%;display:inline-block}.container-content__container-relatedlinks{margin-top:9%}.container-content__container-ads{margin-top:8%}.container-content__container-ads--twot{margin-top:7%}.container-content__webarchive{margin-top:8%}.container-content__header{color:#717171;font-size:15px;margin:0}.container-content--lp{width:90%;min-height:820px}.container-content--rp{width:90%;min-height:820px}.container-content--wa{width:90%}.container-content--twot{width:90%;min-height:820px}.two-tier-ads-list{padding:0 0 1.6em 0}.two-tier-ads-list__list-element{list-style:none;padding:10px 0 5px 0;display:inline-block}.two-tier-ads-list__list-element-image{content:url("//img.sedoparking.com/templates/images/bullet_justads.gif");float:left;padding-top:32px}.two-tier-ads-list__list-element-content{display:inline-block}.two-tier-ads-list__list-element-header-link{font-size:37px;font-weight:bold;text-decoration:underline;color:#0a48ff}.two-tier-ads-list__list-element-text{padding:3px 0 6px 0;margin:.11em 0;line-height:18px;color:#000}.two-tier-ads-list__list-element-link{font-size:1em;text-decoration:underline;color:#0a48ff}.two-tier-ads-list__list-element-link:link,.two-tier-ads-list__list-element-link:visited{text-decoration:underline}.two-tier-ads-list__list-element-link:hover,.two-tier-ads-list__list-element-link:active,.two-tier-ads-list__list-element-link:focus{text-decoration:none}.webarchive-block{text-align:center}.webarchive-block__header-link{color:#0a48ff;font-size:20px}.webarchive-block__list{padding:0}.webarchive-block__list-element{word-wrap:break-word;list-style:none}.webarchive-block__list-element-link{line-height:30px;font-size:20px;color:rgba(10,72,255,.7)}.webarchive-block__list-element-link:link,.webarchive-block__list-element-link:visited{text-decoration:none}.webarchive-block__list-element-link:hover,.webarchive-block__list-element-link:active,.webarchive-block__list-element-link:focus{text-decoration:underline}.container-buybox{text-align:center}.container-buybox__content-buybox{display:inline-block;text-align:left}.container-buybox__content-heading{font-size:15px}.container-buybox__content-text{font-size:12px}.container-buybox__content-link{color:#919da6}.container-buybox__content-link--no-decoration{text-decoration:none}.container-searchbox{margin-bottom:50px;text-align:center}.container-searchbox__content{display:inline-block;font-family:arial,sans-serif;font-size:12px}.container-searchbox__searchtext-label{display:none}.container-searchbox__input,.container-searchbox__button{border:0 none}.container-searchbox__button{cursor:pointer;font-size:12px;margin-left:15px;border:0 none;padding:2px 8px;color:#638296}.container-disclaimer{text-align:center}.container-disclaimer__content{display:inline-block}.container-disclaimer__content-text,.container-disclaimer a{font-size:10px}.container-disclaimer__content-text{color:#555}.container-disclaimer a{color:#555}.container-imprint{text-align:center}.container-imprint__content{display:inline-block}.container-imprint__content-text,.container-imprint__content-link{font-size:10px;color:#555}.container-privacyPolicy{text-align:center}.container-privacyPolicy__content{display:inline-block}.container-privacyPolicy__content-link{font-size:10px;color:#555}.container-cookie-message{position:fixed;bottom:0;width:100%;background:#5f5f5f;font-size:12px;padding-top:15px;padding-bottom:15px}.container-cookie-message__content-text{color:#fff}.container-cookie-message__content-text{margin-left:15%;margin-right:15%}.container-cookie-message__content-interactive{text-align:left;margin:0 15px;font-size:10px}.container-cookie-message__content-interactive-header,.container-cookie-message__content-interactive-text{color:#fff}.container-cookie-message__content-interactive-header{font-size:small}.container-cookie-message__content-interactive-text{margin-top:10px;margin-right:0px;margin-bottom:5px;margin-left:0px;font-size:larger}.container-cookie-message a{color:#fff}.cookie-modal-window{position:fixed;background-color:rgba(200,200,200,.75);top:0;right:0;bottom:0;left:0;-webkit-transition:all .3s;-moz-transition:all .3s;transition:all .3s;text-align:center}.cookie-modal-window__content-header{font-size:150%;margin:0 0 15px}.cookie-modal-window__content{text-align:initial;margin:10% auto;padding:40px;background:#fff;display:inline-block;max-width:550px}.cookie-modal-window__content-text{line-height:1.5em}.cookie-modal-window__close{width:100%;margin:0}.cookie-modal-window__content-body table{width:100%;border-collapse:collapse}.cookie-modal-window__content-body table td{padding-left:15px}.cookie-modal-window__content-necessary-cookies-row{background-color:#dee1e3}.disabled{display:none;z-index:-999}.btn{display:inline-block;border-style:solid;border-radius:5px;padding:15px 25px;text-align:center;text-decoration:none;cursor:pointer;margin:5px;transition:.3s}.btn--success{background-color:#218838;border-color:#218838;color:#fff;font-size:x-large}.btn--success:hover{background-color:#1a6b2c;border-color:#1a6b2c;color:#fff;font-size:x-large}.btn--success-sm{background-color:#218838;border-color:#218838;color:#fff;font-size:initial}.btn--success-sm:hover{background-color:#1a6b2c;border-color:#1a6b2c;color:#fff;font-size:initial}.btn--secondary{background-color:#8c959c;border-color:#8c959c;color:#fff;font-size:medium}.btn--secondary:hover{background-color:#727c83;border-color:#727c83;color:#fff;font-size:medium}.btn--secondary-sm{background-color:#8c959c;border-color:#8c959c;color:#fff;font-size:initial}.btn--secondary-sm:hover{background-color:#727c83;border-color:#727c83;color:#fff;font-size:initial}.switch input{opacity:0;width:0;height:0}.switch{position:relative;display:inline-block;width:60px;height:34px}.switch__slider{position:absolute;cursor:pointer;top:0;left:0;right:0;bottom:0;background-color:#5a6268;-webkit-transition:.4s;transition:.4s}.switch__slider:before{position:absolute;content:"";height:26px;width:26px;left:4px;bottom:4px;background-color:#fff;-webkit-transition:.4s;transition:.4s}.switch__slider--round{border-radius:34px}.switch__slider--round:before{border-radius:50%}input:checked+.switch__slider{background-color:#007bff}input:focus+.switch__slider{box-shadow:0 0 1px #007bff}input:checked+.switch__slider:before{-webkit-transform:translateX(26px);-ms-transform:translateX(26px);transform:translateX(26px)}body{background-color:#262626;font-family:Arial,Helvetica,Verdana,"Lucida Grande",sans-serif}body.cookie-message-enabled{padding-bottom:300px}.container-footer{padding-top:0;padding-left:5%;padding-right:5%;padding-bottom:10px}

    </style><script type="text/javascript">
        var dto = {"uiOptimize":false,"singleDomainName":"my.org","domainName":"my.org","domainPrice":0,"domainCurrency":"","adultFlag":false,"pu":"//rancher.my.org","dnsh":true,"dpsh":false,"toSell":false,"cdnHost":"http://img.sedoparking.com","adblockkey":" data-adblockkey=MFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBANnylWw2vLY4hUn9w06zQKbhKBfvjFUCsdFlb6TdQhxb9RXWXuI4t31c+o8fYOv/s8q1LGPga3DE1L/tHU4LENMCAwEAAQ==_xd/MGsOJ8/p4oGkEaB6Vmp67y3786AgZqepPA/7wf5voD5M/bARtM6XqJg1ah8THtifPlehNIVapKbYwIsJ52Q==","tid":3032,"buybox":false,"buyboxTopic":true,"disclaimer":true,"imprint":false,"searchbox":true,"noFollow":false,"slsh":false,"ppsh":true,"dnhlsh":true,"toSellUrl":"","toSellText":"","searchboxPath":"//rancher.my.org/parking.php","searchParams":{"ses":"Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0xJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejYmdHJhY2txdWVyeT0x"},"gts":false,"imprintUrl":false,"contentType":2,"t":"content","pus":"ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=","postActionParameter":false,"gFeedSES":{"default":"OAlmYThmNWY3MzlkM2YxNDQ2ZWM3ZjQzMTZiMjRkYWI1ZAkJMwkwCQkxMDcxMjE3NzgJU29jaWFsIG5ldHdvcmtpbmcJMzAzMgkJNDcJOTIJMTYyOTA3NjIyNwkwCU4JMAkxCTAJMTI5OAk4MTU4OTE4NgkxOTYuNjUuMjI4LjE0Mwkx","alternate":"OAk0OTFlZjFkN2FjMjlhZGM2OWM5MDNkMDdkY2E5ZmZmOQkwCTMJMAkJMTA3MTIxNzc4CVNvY2lhbCBuZXR3b3JraW5nCTMwMzIJCTQ3CTkyCTE2MjkwNzYyMjcJMAlOCTAJMQkwCTEyOTgJODE1ODkxODYJMTk2LjY1LjIyOC4xNDMJMQ%3D%3D"},"visitorViewIdJsAds":false,"jsParameter":false,"ads":[],"adv":0,"advt":2,"rls":[{"term":"Dating and Relationships"}],"numberRelatedLinks":6,"waUrl":"/search/portal.php?l=OAlmYThmNWY3MzlkM2YxNDQ2ZWM3ZjQzMTZiMjRkYWI1ZAkJMwkwCQkxMDcxMjE3NzgJU29jaWFsIG5ldHdvcmtpbmcJMzAzMgkJNDcJOTIJMTYyOTA3NjIyNwkwCU4JMAkxCTAJMTI5OAk4MTU4OTE4NgkxOTYuNjUuMjI4LjE0Mwkx","tsc":true,"tscQs":"200=MTA3MTIxNzc4&21=MTk2LjY1LjIyOC4xNDM=&681=MTYyOTA3NjIyNzczNjgzYTU2NDI3Yzk0OWEwZjc3MGU0MGQzYTNjM2Zm&crc=2408e98de656f67a89fd5c1a4a4c2d5d7a72a87d&cv=1","lang":"ar","maid":92,"sedoParkingUrl":"https://www.sedo.com/services/parking.php3","dbg":false,"signedLink":"l","visitorViewId":"v","registrar_params":[],"clickTrack":[],"cookieMessage":false,"cookieMessageInteractive":false,"executeTrackingPixels":false,"bannerTypes":[]};
    </script></head><body ><div class="container-header" id="container-header"><h1 class="container-header__content">my.org</h1></div><div id="container-content" class="container-content container-content--wa"><div class="container-content__webarchive" id="container-webarchive"><div class="container-content__content-webarchive" id="wa-default"><h2 class="container-content__header">
                الفئات الأكثر شعبية
            </h2><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%AA%D9%85%D9%88%D9%8A%D9%84.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تمويل&amp;keyword=تمويل">
                            تمويل
                        </a></h3><ul class="webarchive-block__list"><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AA%D8%A3%D9%85%D9%8A%D9%86-%D8%A7%D9%84%D8%A3%D9%85%D9%84%D8%A7%D9%83.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تأمين الأملاك&amp;keyword=تأمين الأملاك">
                                    تأمين الأملاك
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A3%D8%B3%D9%87%D9%85.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=أسهم&amp;keyword=أسهم">
                                    أسهم
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D8%B3%D8%AA%D8%AB%D9%85%D8%A7%D8%B1%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=استثمارات&amp;keyword=استثمارات">
                                    استثمارات
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AA%D8%A3%D9%85%D9%8A%D9%86.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تأمين&amp;keyword=تأمين">
                                    تأمين
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A8%D8%B7%D8%A7%D9%82%D8%A7%D8%AA-%D8%A5%D8%A6%D8%AA%D9%85.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=بطاقات إئتم&amp;keyword=بطاقات إئتم">
                                    بطاقات إئتم
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A7%D9%84%D9%83%D8%AA%D8%B1%D9%88%D9%86%D9%8A%D8%A7%D8%AA.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الكترونيات&amp;keyword=الكترونيات">
                            الكترونيات
                        </a></h3><ul class="webarchive-block__list"><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AF%D9%8A-%D9%81%D9%8A-%D8%AF%D9%8A.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=دي في دي&amp;keyword=دي في دي">
                                    دي في دي
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%87%D9%88%D8%A7%D8%AA%D9%81-%D8%A7%D9%84%D9%85%D8%AD%D9%85%D9%88%D9%84.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=هواتف المحمول&amp;keyword=هواتف المحمول">
                                    هواتف المحمول
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D9%83%D8%A7%D9%85%D9%8A%D8%B1%D8%A7%D8%AA-%D8%A7%D9%84%D8%B1%D9%82%D9%85%D9%8A%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الكاميرات الرقمية&amp;keyword=الكاميرات الرقمية">
                                    الكاميرات الرقمية
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D9%87%D9%88%D8%A7%D8%AA%D9%81.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الهواتف&amp;keyword=الهواتف">
                                    الهواتف
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D8%AD%D9%88%D8%A7%D8%B3%D8%A8-%D8%A7%D9%84%D9%83%D9%81%D9%8A%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الحواسب الكفية&amp;keyword=الحواسب الكفية">
                                    الحواسب الكفية
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%B3%D9%8A%D8%A7%D8%AD%D8%A9.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=سياحة&amp;keyword=سياحة">
                            سياحة
                        </a></h3><ul class="webarchive-block__list"><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%81%D9%86%D8%A7%D8%AF%D9%82.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=فنادق&amp;keyword=فنادق">
                                    فنادق
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A2%D8%AE%D8%B1-%D9%84%D8%AD%D8%B8%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=آخر لحظة&amp;keyword=آخر لحظة">
                                    آخر لحظة
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B9%D8%B7%D9%84%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=عطلات&amp;keyword=عطلات">
                                    عطلات
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AA%D8%A3%D8%AC%D9%8A%D8%B1-%D8%B3%D9%8A%D8%A7%D8%B1%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تأجير سيارات&amp;keyword=تأجير سيارات">
                                    تأجير سيارات
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B1%D8%AD%D9%84%D8%A7%D8%AA-%D8%A8%D8%AD%D8%B1%D9%8A%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=رحلات بحرية&amp;keyword=رحلات بحرية">
                                    رحلات بحرية
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A3%D9%84%D8%B9%D8%A7%D8%A8.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=ألعاب&amp;keyword=ألعاب">
                            ألعاب
                        </a></h3><ul class="webarchive-block__list"><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/Casino.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=Casino&amp;keyword=Casino">
                                    Casino
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/Poker.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=Poker&amp;keyword=Poker">
                                    Poker
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/Sports-Betting.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=Sports Betting&amp;keyword=Sports Betting">
                                    Sports Betting
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%83%D8%B3-%D8%A8%D9%88%D9%83%D8%B3.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=اكس بوكس&amp;keyword=اكس بوكس">
                                    اكس بوكس
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A8%D9%84%D8%A7%D9%8A-%D8%B3%D8%AA%D9%8A%D8%B4%D9%86.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=بلاي ستيشن&amp;keyword=بلاي ستيشن">
                                    بلاي ستيشن
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A5%D9%86%D8%AA%D8%B1%D9%86%D8%AA%21.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=إنترنت!&amp;keyword=إنترنت!">
                            إنترنت!
                        </a></h3><ul class="webarchive-block__list"><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A3%D9%85%D8%A7%D9%86-%D8%A7%D9%84%D8%B4%D8%A8%D9%83%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=أمان الشبكة&amp;keyword=أمان الشبكة">
                                    أمان الشبكة
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%85%D8%B2%D9%88%D8%AF%D9%8A-%D8%AE%D8%AF%D9%85%D8%A9-%D8%A7%D9%84%D8%A5%D9%86%D8%AA%D8%B1%D9%86%D8.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=مزودي خدمة الإنترنت&amp;keyword=مزودي خدمة الإنترنت">
                                    مزودي خدمة الإنترنت
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AF%D9%8A-%D8%A7%D8%B3-%D8%A7%D9%84.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=دي اس ال&amp;keyword=دي اس ال">
                                    دي اس ال
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A5%D8%B3%D8%AA%D8%B6%D8%A7%D9%81%D8%A9-%D9%85%D9%88%D8%A7%D9%82%D8%B9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=إستضافة مواقع&amp;keyword=إستضافة مواقع">
                                    إستضافة مواقع
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AA%D8%B5%D9%85%D9%8A%D9%85-%D9%85%D9%88%D8%A7%D9%82%D8%B9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تصميم مواقع&amp;keyword=تصميم مواقع">
                                    تصميم مواقع
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A3%D8%B3%D9%84%D9%88%D8%A8-%D8%A7%D9%84%D8%AD%D9%8A%D8%A7%D8%A9.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=أسلوب الحياة&amp;keyword=أسلوب الحياة">
                            أسلوب الحياة
                        </a></h3><ul class="webarchive-block__list"><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A3%D8%AF%D9%88%D9%8A%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=أدوية&amp;keyword=أدوية">
                                    أدوية
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%81%D9%8A%D8%AA%D8%A7%D9%85%D9%8A%D9%86%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=فيتامينات&amp;keyword=فيتامينات">
                                    فيتامينات
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D9%86%D9%88%D9%85.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=النوم&amp;keyword=النوم">
                                    النوم
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B1%D8%AC%D9%8A%D9%85.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=رجيم&amp;keyword=رجيم">
                                    رجيم
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%81%D9%82%D8%AF%D8%A7%D9%86-%D8%A7%D9%84%D9%88%D8%B2%D9%86.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=فقدان الوزن&amp;keyword=فقدان الوزن">
                                    فقدان الوزن
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D9%87%D8%AF%D8%A7%D9%8A%D8%A7.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=هدايا&amp;keyword=هدايا">
                            هدايا
                        </a></h3><ul class="webarchive-block__list"><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AC%D9%88%D9%84%D9%81.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=جولف&amp;keyword=جولف">
                                    جولف
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B3%D9%84%D8%A7%D8%AA-%D9%87%D8%AF%D8%A7%D9%8A%D8%A7.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=سلات هدايا&amp;keyword=سلات هدايا">
                                    سلات هدايا
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%85%D8%AC%D9%88%D9%87%D8%B1%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=مجوهرات&amp;keyword=مجوهرات">
                                    مجوهرات
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B2%D9%87%D9%88%D8%B1.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=زهور&amp;keyword=زهور">
                                    زهور
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D8%B2%D9%88%D8%A7%D8%AC.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الزواج&amp;keyword=الزواج">
                                    الزواج
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%AD%D9%88%D8%A7%D8%B3%D8%A8-%D8%A2%D9%84%D9%8A%D8%A9.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=حواسب آلية&amp;keyword=حواسب آلية">
                            حواسب آلية
                        </a></h3><ul class="webarchive-block__list"><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A3%D9%84%D8%B9%D8%A7%D8%A8-%D8%A7%D9%84%D8%AD%D8%A7%D8%B3%D8%A8-%D8%A7%D9%84%D8%A2%D9%84%D9%8A.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=ألعاب الحاسب الآلي&amp;keyword=ألعاب الحاسب الآلي">
                                    ألعاب الحاسب الآلي
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AE%D9%88%D8%A7%D8%AF%D9%85.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=خوادم&amp;keyword=خوادم">
                                    خوادم
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D8%AD%D8%A7%D8%B3%D8%A8-%D8%A7%D9%84%D9%86%D9%82%D8%A7%D9%84.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الحاسب النقال&amp;keyword=الحاسب النقال">
                                    الحاسب النقال
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%82%D8%B7%D8%B9-%D8%A3%D8%AC%D9%87%D8%B2%D8%A9-%D8%A7%D9%84%D8%AD%D8%A7%D8%B3%D8%A8-%D8%A7%D9%84.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=قطع أجهزة الحاسب الآلي&amp;keyword=قطع أجهزة الحاسب الآلي">
                                    قطع أجهزة الحاسب الآلي
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A8%D8%B1%D8%A7%D9%85%D8%AC.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=برامج&amp;keyword=برامج">
                                    برامج
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A3%D8%B5%D8%AF%D9%82%D8%A7%D8%A1.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=أصدقاء&amp;keyword=أصدقاء">
                            أصدقاء
                        </a></h3><ul class="webarchive-block__list"><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D8%B2%D9%88%D8%A7%D8%AC.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الزواج&amp;keyword=الزواج">
                                    الزواج
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/Eastern.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=Eastern&amp;keyword=Eastern">
                                    Eastern
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/Christmas.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=Christmas&amp;keyword=Christmas">
                                    Christmas
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B9%D9%8A%D8%AF-%D9%85%D9%8A%D9%84%D8%A7%D8%AF.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=عيد ميلاد&amp;keyword=عيد ميلاد">
                                    عيد ميلاد
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A8%D8%B7%D8%A7%D9%82%D8%A7%D8%AA-%D9%85%D8%B9%D8%A7%D9%8A%D8%AF%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=بطاقات معايدة&amp;keyword=بطاقات معايدة">
                                    بطاقات معايدة
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A7%D9%84%D8%AA%D8%B1%D9%81%D9%8A%D9%87.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الترفيه&amp;keyword=الترفيه">
                            الترفيه
                        </a></h3><ul class="webarchive-block__list"><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%85%D9%88%D8%B3%D9%8A%D9%82%D9%89.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=موسيقى&amp;keyword=موسيقى">
                                    موسيقى
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AA%D8%B0%D8%A7%D9%83%D8%B1.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تذاكر&amp;keyword=تذاكر">
                                    تذاكر
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/MP3s.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=MP3s&amp;keyword=MP3s">
                                    MP3s
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A3%D9%81%D9%84%D8%A7%D9%85.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=أفلام&amp;keyword=أفلام">
                                    أفلام
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B1%D9%8A%D8%A7%D8%B6%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=رياضة&amp;keyword=رياضة">
                                    رياضة
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%A7%D9%84%D8%AA%D8%B3%D9%88%D9%82.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=التسوق&amp;keyword=التسوق">
                            التسوق
                        </a></h3><ul class="webarchive-block__list"><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D9%85%D9%88%D8%B6%D9%87.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=الموضه&amp;keyword=الموضه">
                                    الموضه
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%B2%D9%87%D9%88%D8%B1.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=زهور&amp;keyword=زهور">
                                    زهور
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%85%D8%A8%D9%8A%D8%B9%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=مبيعات&amp;keyword=مبيعات">
                                    مبيعات
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%85%D9%84%D8%A7%D8%A8%D8%B3-%D9%86%D8%B3%D8%A7%D8%A6%D9%8A%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=ملابس نسائية&amp;keyword=ملابس نسائية">
                                    ملابس نسائية
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%85%D8%B2%D8%A7%D8%AF%D8%A7%D8%AA.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=مزادات&amp;keyword=مزادات">
                                    مزادات
                                </a></li></ul></div><div class="webarchive-block"><h3 class="webarchive-block__header"><a
                                class="webarchive-block__header-link"
                                href="//rancher.my.org/info/%D8%AA%D8%AC%D8%A7%D8%B1%D8%A9.html>?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تجارة&amp;keyword=تجارة">
                            تجارة
                        </a></h3><ul class="webarchive-block__list"><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AE%D8%B7%D8%A9-%D8%AA%D8%AC%D8%A7%D8%B1%D9%8A%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=خطة تجارية&amp;keyword=خطة تجارية">
                                    خطة تجارية
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D9%88%D8%B8%D8%A7%D8%A6%D9%81.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=وظائف&amp;keyword=وظائف">
                                    وظائف
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%AA%D8%AF%D8%B1%D9%8A%D8%A8-%D8%AA%D9%82%D9%86%D9%8A.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=تدريب تقني&amp;keyword=تدريب تقني">
                                    تدريب تقني
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A5%D8%AF%D8%A7%D8%B1%D8%A9.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=إدارة&amp;keyword=إدارة">
                                    إدارة
                                </a></li><li class="webarchive-block__list-element"><a
                                        class="webarchive-block__list-element-link"
                                        href="//rancher.my.org/info/%D8%A7%D9%84%D8%B9%D9%85%D9%84-%D9%85%D9%86-%D8%A7%D9%84%D9%85%D9%86%D8%B2%D9%84.html?ses=Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0zJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejY=&amp;category=العمل من المنزل&amp;keyword=العمل من المنزل">
                                    العمل من المنزل
                                </a></li></ul></div></div></div></div><div class="container-searchbox" id="container-searchbox"><div class="container-searchbox__content" id="sb-default"><form class="searchform" id="searchform" action="//rancher.my.org/parking.php" method="get" name="searchform"><input
                        class="container-searchbox__input"
                        type="hidden"
                        name="ses"
                        value="Y3JlPTE2MjkwNzYyMjcmdGNpZD1yYW5jaGVyLm15Lm9yZzYxMTliYjAzYTEzOTUyLjE4NTczNDc3JnRhc2s9c2VhcmNoJmRvbWFpbj1teS5vcmcmbGFuZ3VhZ2U9YXImYV9pZD0xJnNlc3Npb249MVBucWNzWFpyYXFNbmRzYUxfejYmdHJhY2txdWVyeT0x"><input
                        class="container-searchbox__input"
                        type="hidden"
                        name="token"
                        value=""><label
                        class="container-searchbox__searchtext-label"
                        for="searchtext">
                    Search
                </label><input
                        class="container-searchbox__input"
                        type="text"
                        name="keyword"
                        value=""
                        id="searchtext"
                        maxlength="70"><button
                        class="container-searchbox__button"
                        type="submit">
                    بحث
                </button></form></div></div><div class="container-buybox" id="container-buybox"></div><div class="container-footer" id="container-footer"><div class="container-disclaimer" id="container-disclaimer"><div class="container-disclaimer__content"><p class="container-disclaimer__content-text">
        This webpage was generated by the domain owner using <a href="https://www.sedo.com/services/parking.php3">Sedo Domain Parking</a>. Disclaimer: Sedo maintains no relationship with third party advertisers. Reference to any specific service or trade mark is not controlled by Sedo nor does it constitute or imply its association, endorsement or recommendation.
    </p></div></div><div class="container-imprint" id="container-imprint"></div><div class="container-privacyPolicy" id="container-privacyPolicy"><div class="container-privacyPolicy__content"><div id="privacy-policy-link"><a
                class="container-privacyPolicy__content-link"
                href="#"
                onclick="window.open(
                        '//sedoparking.com/privacy-policy/us/index.html',
                        'privacy-policy','width=600,height=400');
                        return false;">
            نهج الخصوصية
        </a></div></div></div></div><!-- CAF --><script type="text/javascript">
                        if (dto.tsc) {
    var request = new XMLHttpRequest();
    request.open('GET', dto.pu + '/search/tsc.php?' + dto.tscQs);
    request.send();
}
* Connection #0 to host rancher.my.org left intact
            </script></body></html>

which I don’t know what to make of other than what appears to be some sort of redirecting towards an html page.
I believe that the Load balancer is set up correctly, because it goes along with the example you shared. even more, when I browse https://192.168.157.140, I get the warning about the Rancher generated cert. which implies that the load balancer is working properly.
I also find it strange that after acknowledging the warning, I get :

default backend - 404

and also :

~$ curl -vLk https://192.168.157.140
*   Trying 192.168.157.140:443...
* TCP_NODELAY set
* Connected to 192.168.157.140 (192.168.157.140) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Aug  6 15:16:44 2021 GMT
*  expire date: Aug  6 15:16:44 2022 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x557b50b72e10)
> GET / HTTP/2
> Host: 192.168.157.140
> user-agent: curl/7.68.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 404
< server: openresty/1.15.8.1
< date: Mon, 16 Aug 2021 01:42:41 GMT
< content-type: text/plain; charset=utf-8
< content-length: 21
< strict-transport-security: max-age=15724800; includeSubDomains
<
* Connection #0 to host 192.168.157.140 left intact
default backend - 404

which I don’t get when browsing http://rancher.my.org
I don’t know how to diagnose the situation, so I’d like to know what you think.
I want to try excluding the load balancer by pointing the DNS to a node directly, I don’t know how but I’ll look it up.
thanks for the assistance you have provided already, I’ve been learning through you.

kamal_boumahdi · August 31, 2021, 12:00pm

Solved!
To everyone who might run into this problem in the future.
These Liveliness probe and Readiness probe kept failing due to connection being refused, which is due to a certain port being closed.
You have to actually open them in order for connection to be accepted.
You’ll find the list of all the ports that need to be opened as well as how to open them in this documentation page: Rancher Docs: Requirements
PLEASE don’t repeat my mistake

Tulio_Manrique · October 12, 2021, 5:43pm

could you tell me which ports was the problem,right now i have a cluster and the prometheus restart becasuse dont pass the health check but its fine when you go on the sites and saw the metrics

kamal_boumahdi · October 18, 2021, 12:09am

if I remember correctly, it was 6443, 80 or 443

Ravi_Sagar · November 10, 2022, 1:11pm

open the ports where?

user449993 · November 23, 2022, 8:33am

I have just encountered similar problems. in my case it was not the ports that I had to open but to add the configuration that is mentioned here for NetworkManager

https://docs.rke2.io/known_issues/#networkmanager

I also disabled selinux and rebooted the machine. after doing this, the errors with the liveness probes and other connectivity issues I’ve had disappeared.

Topic		Replies	Views
Cattle-pods failing Rancher	2	1772	October 25, 2019
A lot of Issues after Server-Downtime Rancher	5	1924	November 2, 2021
Error uninstall rancher helm	1	3244	June 26, 2022
Rancher in Docker - helm-operation error Rancher	2	6448	December 6, 2022
Deployment in progress but no pods created - logs error	0	18	September 6, 2024

Liveness probe failed connection refused

Related topics