Kubernetes 1.3 kubelet fails to start

Rancher 1.x
1

Created a new kubernetes cluster on:

RancherOS 0.70
Rancher v1.2.0-pre4-rc5

When I attempt to create a kubernetes 1.3 stack, the kubelets fail to start with the error:

10/25/2016 3:34:15 PMWaiting for metadata
10/25/2016 3:34:16 PM+ curl -s -f http://rancher-metadata/2015-12-19/stacks/Kubernetes/services/kubernetes/uuid
10/25/2016 3:34:16 PM++ curl -s http://rancher-metadata/2015-12-19/stacks/Kubernetes/services/kubernetes/uuid
10/25/2016 3:34:16 PM+ UUID=764d98df-667f-43b1-ac2e-36b20ae0e46b
10/25/2016 3:34:16 PM++ curl -s -u 20E67152138B33496EC3:QrUFQaFwGeNLK6qTcxLdJ9VC5DnsLwW5Yw3Yk1bt ‘http://192.168.2.219:8080/v1/services?uuid=764d98df-667f-43b1-ac2e-36b20ae0e46b’
10/25/2016 3:34:16 PM++ jq -r ‘.data[0].actions.certificate’
10/25/2016 3:34:16 PM+ ACTION=‘http://192.168.2.219:8080/v1/services/1s2/?action=certificate’
10/25/2016 3:34:16 PM+ ‘[’ -n ‘http://192.168.2.219:8080/v1/services/1s2/?action=certificate’ ‘]’
10/25/2016 3:34:16 PM+ mkdir -p /etc/kubernetes/ssl
10/25/2016 3:34:16 PM+ cd /etc/kubernetes/ssl
10/25/2016 3:34:16 PM+ curl -s -u 20E67152138B33496EC3:QrUFQaFwGeNLK6qTcxLdJ9VC5DnsLwW5Yw3Yk1bt -X POST ‘http://192.168.2.219:8080/v1/services/1s2/?action=certificate’
10/25/2016 3:34:16 PM+ unzip -o certs.zip
10/25/2016 3:34:16 PM764d98df-667f-43b1-ac2e-36b20ae0e46bArchive: certs.zip
10/25/2016 3:34:16 PM inflating: ca.pem
10/25/2016 3:34:16 PM inflating: cert.pem
10/25/2016 3:34:16 PM inflating: key.pem
10/25/2016 3:34:16 PM+ cd /
10/25/2016 3:34:16 PM+ cat
10/25/2016 3:34:16 PM+ ‘[’ kubelet == kubelet ‘]’
10/25/2016 3:34:16 PM+ /usr/bin/share-mnt /var/lib/kubelet /sys – kubelet-start.sh kubelet --kubeconfig=/etc/kubernetes/ssl/kubeconfig --api_servers=https://kubernetes:6443 --allow-privileged=true --register-node=true --cloud-provider=rancher --healthz-bind-address=0.0.0.0 --cluster-dns=169.254.169.250 --cluster-domain=cluster.local --network-plugin=cni --network-plugin-dir=/etc/cni/net.d
10/25/2016 3:34:16 PMtime=“2016-10-25T21:34:16Z” level=info msg=“Execing [/usr/bin/nsenter --mount=/proc/29050/ns/mnt -F – /var/lib/docker/overlay/573bcae900b18e4c6a1ef29fd4cb1b7c256a36aeae14b0969d34f15cbc9f588d/merged/usr/bin/share-mnt --stage2 /var/lib/kubelet /sys – kubelet-start.sh kubelet --kubeconfig=/etc/kubernetes/ssl/kubeconfig --api_servers=https://kubernetes:6443 --allow-privileged=true --register-node=true --cloud-provider=rancher --healthz-bind-address=0.0.0.0 --cluster-dns=169.254.169.250 --cluster-domain=cluster.local --network-plugin=cni --network-plugin-dir=/etc/cni/net.d]”
10/25/2016 3:34:16 PMI1025 21:34:16.893414 29242 docker.go:327] Start docker client with request timeout=2m0s
10/25/2016 3:34:16 PMI1025 21:34:16.938391 29242 server.go:768] Watching apiserver
10/25/2016 3:34:16 PMW1025 21:34:16.938644 29242 kubelet.go:561] Hairpin mode set to “promiscuous-bridge” but configureCBR0 is false, falling back to “hairpin-veth”
10/25/2016 3:34:16 PMI1025 21:34:16.938746 29242 kubelet.go:384] Hairpin mode set to “hairpin-veth”
10/25/2016 3:34:16 PME1025 21:34:16.939322 29242 server.go:293] Failed running kubelet: failed to create kubelet: Unable to post “machine info” to “http://127.0.0.1:9344/api/v1.3/machine”: Get http://127.0.0.1:9344/api/v1.3/machine: dial tcp 127.0.0.1:9344: getsockopt: connection refused
10/25/2016 3:34:16 PMfailed to create kubelet: Unable to post “machine info” to “http://127.0.0.1:9344/api/v1.3/machine”: Get http://127.0.0.1:9344/api/v1.3/machine: dial tcp 127.0.0.1:9344: getsockopt: connection refused

Am I missing something?

2

I run in the same issue, independently I used RancherOS v0.6.1 or v0.7.0, Centos 7, Kubernetes 1.2 or 1.3. I’m a bit disappointed.

3

Finally I’ve switched from master to qa branch on infra catalog. The problem doesn’t appear in k8s 1.4.4

4

@saphoooo Can you elaborate how you did that? I am running everything via docker. Is there an alternate tag I can pull?

5

I have the same issues. Everithing spins up except the kubelet container.

9-11-2016 16:29:40time=“2016-11-09T15:29:40Z” level=info msg="Execing [/usr/bin/nsenter --mount=/proc/9569/ns/mnt -F – /var/lib/docker/devicemapper/mnt/fb50a30919a22e74ea5ecddd5357e376f9a91bd91f2d105582b87145fee51527/rootfs/usr/bin/share-mnt --stage2 /var/lib/kubelet /sys – kubelet --api_servers=http://master --allow-privileged=true --register-node=true --cloud-provider=rancher --healthz-bind-address=0.0.0.0 --cluster-dns=169.254.169.250 --cluster-domain=cluster.local]"
9-11-2016 16:29:40W1109 15:29:40.997774 8613 server.go:445] Could not load kubeconfig file /var/lib/kubelet/kubeconfig: stat /var/lib/kubelet/kubeconfig: no such file or directory. Trying auth path instead.
9-11-2016 16:29:40W1109 15:29:40.997825 8613 server.go:406] Could not load kubernetes auth path /var/lib/kubelet/kubernetes_auth: stat /var/lib/kubelet/kubernetes_auth: no such file or directory. Continuing with defaults.
9-11-2016 16:29:41I1109 15:29:41.129883 8613 server.go:683] Watching apiserver
9-11-2016 16:29:41E1109 15:29:41.141760 8613 server.go:270] Failed running kubelet: failed to create kubelet: Unable to post “machine info” to “http://127.0.0.1:9344/api/v1.3/machine”: Get http://127.0.0.1:9344/api/v1.3/machine: dial tcp 127.0.0.1:9344: connection refused
9-11-2016 16:29:41failed to create kubelet: Unable to post “machine info” to “http://127.0.0.1:9344/api/v1.3/machine”: Get http://127.0.0.1:9344/api/v1.3/machine: dial tcp 127.0.0.1:9344: connection refused