OK, I failed at that … I can’t seem to update the cluster.
- When I try to get the kubeconfig_admin.yaml, I get an error;
Unable to connect to the server: net/http: TLS handshake timeout
- I found an old kubeconfig_admin.yaml file and I used that, but I still get the error:
Unable to connect to the server: net/http: TLS handshake timeout
When I look at the docker status, I can see that kube-apiserver is constantly restarting
ubuntu@ranchertest1:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d704073bc13b rancher/rancher-agent:v2.1.1 "run.sh --server h..." 5 days ago Up 5 days silly_haibt
8f9079771aa2 rancher/hyperkube:v1.11.2-rancher1 "/opt/rke-tools/en..." 12 months ago Up 2 weeks kube-proxy
79d667615d77 rancher/hyperkube:v1.11.2-rancher1 "/opt/rke-tools/en..." 12 months ago Up 2 weeks kubelet
c4b134475cc1 rancher/hyperkube:v1.11.2-rancher1 "/opt/rke-tools/en..." 12 months ago Up 2 weeks kube-scheduler
0d06281ce7f4 rancher/hyperkube:v1.11.2-rancher1 "/opt/rke-tools/en..." 12 months ago Up 2 weeks kube-controller-manager
e18b3ebbacc7 rancher/hyperkube:v1.11.2-rancher1 "/opt/rke-tools/en..." 12 months ago Up 2 seconds kube-apiserver
f304f98d0270 rancher/coreos-etcd:v3.2.18 "/usr/local/bin/et..." 12 months ago Up 2 weeks etcd
I took a look at the logs for the kube-apiserver and saw some nice errors…
{
"log": "F1030 13:37:51.820860 1 storage_decorator.go:57] Unable to create storage backend: config (&{etcd3 /registry [https://10.0.134.133:2379 https://10.0.141.59:2379 https://10.0.140.63:2379] /etc/kubernetes/ssl/kube-node-key.pem /etc/kubernetes/ssl/kube-node.pem /etc/kubernetes/ssl/kube-ca.pem true false 1000 0xc420261500 <nil> 5m0s 1m0s}), err (context deadline exceeded)\n",
"stream": "stderr",
"time": "2019-10-30T13:37:51.821152667Z"
}
{
"log": "+ grep -q cloud-provider=azure\n",
"stream": "stderr",
"time": "2019-10-30T13:37:52.29488014Z"
}
{
"log": "+ echo kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --allow-privileged=true --service-cluster-ip-range=10.43.0.0/16 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --requestheader-allowed-names=kube-apiserver-proxy-client --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --requestheader-username-headers=X-Remote-User --service-node-port-range=30000-32767 --etcd-prefix=/registry --admission-control=ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds --requestheader-group-headers=X-Remote-Group --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --secure-port=6443 --storage-backend=etcd3 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --insecure-port=0 --service-account-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authorization-mode=Node,RBAC --etcd-servers=https://10.0.134.133:2379,https://10.0.141.59:2379,https://10.0.140.63:2379 --insecure-bind-address=127.0.0.1 --cloud-provider=aws\n",
"stream": "stderr",
"time": "2019-10-30T13:37:52.29569754Z"
}
{
"log": "+ '[' kube-apiserver = kubelet ']'\n",
"stream": "stderr",
"time": "2019-10-30T13:37:52.296391272Z"
}
{
"log": "+ exec kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --allow-privileged=true --service-cluster-ip-range=10.43.0.0/16 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --requestheader-allowed-names=kube-apiserver-proxy-client --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --requestheader-username-headers=X-Remote-User --service-node-port-range=30000-32767 --etcd-prefix=/registry --admission-control=ServiceAccount,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds --requestheader-group-headers=X-Remote-Group --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --secure-port=6443 --storage-backend=etcd3 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --insecure-port=0 --service-account-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authorization-mode=Node,RBAC --etcd-servers=https://10.0.134.133:2379,https://10.0.141.59:2379,https://10.0.140.63:2379 --insecure-bind-address=127.0.0.1 --cloud-provider=aws\n",
"stream": "stderr",
"time": "2019-10-30T13:37:52.296404885Z"
}
{
"log": "Flag --admission-control has been deprecated, Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.\n",
"stream": "stderr",
"time": "2019-10-30T13:37:52.446063372Z"
}
{
"log": "Flag --insecure-port has been deprecated, This flag will be removed in a future version.\n",
"stream": "stderr",
"time": "2019-10-30T13:37:52.44609073Z"
}
{
"log": "Flag --insecure-bind-address has been deprecated, This flag will be removed in a future version.\n",
"stream": "stderr",
"time": "2019-10-30T13:37:52.44609495Z"
}
{
"log": "I1030 13:37:52.446264 1 server.go:703] external host was not specified, using 10.0.141.59\n",
"stream": "stderr",
"time": "2019-10-30T13:37:52.446441805Z"
}
{
"log": "I1030 13:37:52.446400 1 server.go:145] Version: v1.11.2\n",
"stream": "stderr",
"time": "2019-10-30T13:37:52.446449752Z"
}
{
"log": "W1030 13:37:53.047054 1 admission.go:71] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.\n",
"stream": "stderr",
"time": "2019-10-30T13:37:53.047375752Z"
}
{
"log": "I1030 13:37:53.047307 1 plugins.go:158] Loaded 6 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass.\n",
"stream": "stderr",
"time": "2019-10-30T13:37:53.047400206Z"
}
{
"log": "I1030 13:37:53.047326 1 plugins.go:161] Loaded 3 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,ResourceQuota.\n",
"stream": "stderr",
"time": "2019-10-30T13:37:53.047404911Z"
}
{
"log": "W1030 13:37:53.048803 1 admission.go:71] PersistentVolumeLabel admission controller is deprecated. Please remove this controller from your configuration files and scripts.\n",
"stream": "stderr",
"time": "2019-10-30T13:37:53.049040344Z"
}
{
"log": "I1030 13:37:53.048975 1 plugins.go:158] Loaded 6 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultTolerationSeconds,PersistentVolumeLabel,DefaultStorageClass.\n",
"stream": "stderr",
"time": "2019-10-30T13:37:53.049051415Z"
}
{
"log": "I1030 13:37:53.048989 1 plugins.go:161] Loaded 3 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,ResourceQuota.\n",
"stream": "stderr",
"time": "2019-10-30T13:37:53.049055263Z"
}
So, that makes sense.
The kube API server is responsible for connecting on port 6443, but it keeps crashing for some reason … maybe it’s the certificates.
@superseb can I replace the certificates directly and stop it from crashing?
