Hi,
I have a rancher node and launched kubernetes with 3 nodes (1 master, 2 workers). However, in rancher, Launh Kubectl gives an error Closed Code 1006
Any suggestions?
Thanks
Same problem here - all of a sudden…
Same here but different code:
Closed Code: 1015
I have the same problem with Rancher 2.2.2
root@server-01:~# kubectl get po -n grafana-mm01
NAME READY STATUS RESTARTS AGE
grafana-01-dccff9fbf-vv6zp 1/1 Running 1 42m
root@server-01:~# kubectl logs grafana-01-dccff9fbf-vv6zp
Error from server (NotFound): pods “grafana-01-dccff9fbf-vv6zp” not found
Very strange…
root@server-01:~# kubectl exec grafana-01-dccff9fbf-vv6zp -n grafana-mm01 – /bin/bash
error: unable to upgrade connection: pod does not exist
root@server-01:~# kubectl -v=7 exec grafana-01-dccff9fbf-vv6zp -n grafana-mm01 – /bin/bash
I0501 23:34:38.780428 5311 loader.go:359] Config loaded from file /root/.kube/config
I0501 23:34:38.781943 5311 round_trippers.go:416] GET https://192.168.178.111:6443/api/v1/namespaces/grafana-mm01/pods/grafana-01-dccff9fbf-vv6zp
I0501 23:34:38.782344 5311 round_trippers.go:423] Request Headers:
I0501 23:34:38.782585 5311 round_trippers.go:426] Accept: application/json, /
I0501 23:34:38.782791 5311 round_trippers.go:426] User-Agent: kubectl/v1.14.1 (linux/amd64) kubernetes/b739410
I0501 23:34:38.792236 5311 round_trippers.go:441] Response Status: 200 OK in 9 milliseconds
I0501 23:34:38.799915 5311 round_trippers.go:416] POST https://192.168.178.111:6443/api/v1/namespaces/grafana-mm01/pods/grafana-01-dccff9fbf-vv6zp/exec?command=%2Fbin%2Fbash&container=grafana&stderr=true&stdout=true
I0501 23:34:38.802768 5311 round_trippers.go:423] Request Headers:
I0501 23:34:38.803426 5311 round_trippers.go:426] X-Stream-Protocol-Version: v4.channel.k8s.io
I0501 23:34:38.803684 5311 round_trippers.go:426] X-Stream-Protocol-Version: v3.channel.k8s.io
I0501 23:34:38.803960 5311 round_trippers.go:426] X-Stream-Protocol-Version: v2.channel.k8s.io
I0501 23:34:38.804233 5311 round_trippers.go:426] X-Stream-Protocol-Version: channel.k8s.io
I0501 23:34:38.804480 5311 round_trippers.go:426] User-Agent: kubectl/v1.14.1 (linux/amd64) kubernetes/b739410
I0501 23:34:38.822631 5311 round_trippers.go:441] Response Status: 404 Not Found in 17 milliseconds
F0501 23:34:38.823119 5311 helpers.go:114] error: unable to upgrade connection: pod does not exist
root@server-01:~#
Also via the Rancher UI no Logs and no Shell…
1006 just means the connection was closed without sending a close frame/message first (which is the normal way of closing the connection).
So the cause can be anything, and is probably different for each of you, but likely involve something like a proxy/balancer/enterprise-tls-man-in-the-middle device that does not know how to speak websockets properly or issues with the TLS setup (1015 means that the cert verification failed).
I have done now a new installation of Rancher 2.2.2.
I can use kubectl via Rancher UI.
I can list Pods
BUT:
I can’t view Logs from the Pods
I can’t ssh into the Pods
SEE Attached Screenshot.
Any Idea how to troubleshoot this behavior?
Without Pod Logs and SSH Login to the Pods I cant use this installation…
Thanks in advance
Can you try kubectl -n kube-system logs metrics-server-58bd5dd9d7-2mt7w for the logs and kubectl -n kube-system exec -it metrics-server-58bd5dd9d7-2mt7w -- /bin/sh for exec (which is not really SSH)?
First command is using kubectl get logs where logs is not a k8s resource and it is also not specifying the namespace it is in, kubectl logs and adding -n kube-system should resolve that.
Second one, is trying to exec into a pod of a job which is Completed (nothing to exec into) and missing the namespace as well. Using a Running pod and specifying the namespace should resolve that.
Same problem here with version v2.2.2, tried two different setups, one is working, the other one (production setup) not:
Cluster 3 nodes (one nodepool): Launch Kubectl from UI is working
Cluster 3 nodes (2 nodepools, separate etcd, worker): Launch Kubectl from UI not working, indicating “Code 1006” in the UI
Note: my rancher container is living behind a traefik proxy with LE enabled.
1 Like
Hey folks!
In my case it was the firewall.
Check yours.
I had the same issue and I managed to fix it by increasing load balancer’s timeout to 3600. I hope this might help someone else in future too.
Qual alteração realizou no firewall para solucionar?