Rancher k8s ingress-controller container failing

Rancher 1.x
1

Hello,

I’m running a standalone rancher server for a POC and I’m trying to create a new k8s environment on the same host. The issue is that 8/9 services appear to be running, but the ingress-controller container is failing. Here’s the output from the container logs:

8/14/2016 1:23:27 PMtime="2016-08-13T18:43:27Z" level=info msg="Starting Rancher LB service"
8/14/2016 1:23:27 PMtime="2016-08-13T18:43:27Z" level=info msg="LB controller: kubernetes"
8/14/2016 1:23:27 PMtime="2016-08-13T18:43:27Z" level=info msg="LB provider: rancher"
8/14/2016 1:23:27 PMtime="2016-08-13T18:43:27Z" level=info msg="starting kubernetes-ingress-controller"
8/14/2016 1:23:27 PMtime="2016-08-13T18:43:27Z" level=info msg="Healthcheck handler is listening on :10241"
8/14/2016 1:23:30 PMtime="2016-08-13T18:43:30Z" level=error msg="Health check failed: unable to reach Kubernetes. Error: &url.Error{Op:\"Get\", URL:\"http://master/apis/extensions/v1beta1/ingresses\", Err:(*net.OpError)(0xc8202c2cd0)}"
8/14/2016 1:23:30 PMtime="2016-08-13T18:43:30Z" level=error msg="Healtcheck failed for LB controller"
Kubernetes. Error: &url.Error{Op:\"Get\", URL:\"http://master/apis/extensions/v1beta1/ingresses\", Err:(*net.OpError)(0xc8202c2cd0)}"
8/14/2016 1:23:30 PMtime="2016-08-13T18:43:30Z" level=error msg="Healtcheck failed for LB controller"
Kubernetes. Error: &url.Error{Op:\"Get\", URL:\"http://master/apis/extensions/v1beta1/ingresses\", Err:(*net.OpError)(0xc8202c2cd0)}"
8/14/2016 1:23:30 PMtime="2016-08-13T18:43:30Z" level=error msg="Healtcheck failed for LB controller"

My host is behind an http proxy but I’ve set no_proxy for this specific host. I’m using docker 1.10.3 and the latest and stable tags of the ranche/server image. Shouldn’t the ingress container be trying to reach http:///apis instead of master? Is “master” some kind of alias?

By searching online I found a suggestion that this might be resource related, but I’ve tried this setup on multiple hosts and haven’t seen any issue with cpu/memory/io congestion.

Thanks

2

The kube-proxy is able to talk to the master without issues and is running normally. It’s started pointing to master:

kube-proxy --master=http://master --v=2 --healthz-bind-address=0.0.0.0 --proxy-mode=userspace
3

Do you have only one host in your k8s environment?

4

Yes. I’m using the same host to run the rancher/server and the agent. If I delete the ingress controller the setup finishes normally and I can use kubectl as if everything went fine.

5

The ingress controller is used in k8s if you are using ingress in k8s.

http://docs.rancher.com/rancher/v1.2/en/kubernetes/ingress/

Without the ingress controller, k8s should work just fine, but you won’t be able to add in ingress.

I haven’t had issues setting up k8s in v1.1.2 (the stable/latest verison) but haven’t tried with using the same host for server/agent.

6

Hi denise,

I am seeing the same messages in the logs for the rancher-ingress-controler as nkaravias in my single node RancherOS 0.5 /Rancher setup.

I am looking to do a minimal Rancher/k8s setup: single RancherOS 0.5 in ESX via ISO, login, then run the docker command to bringup Rancher. Log into Rancher, go straight to new k8s environment.

As mentioned, getting the same ingress logs, but also kublet is looping with these logs:

8/15/2016 12:36:27 PMTimestamp:
8/15/2016 12:40:26 PMArchive: certs.zip
8/15/2016 12:40:26 PM inflating: ca.pem
8/15/2016 12:40:26 PM inflating: cert.pem
8/15/2016 12:40:26 PM inflating: key.pem
8/15/2016 12:40:27 PMtime=“2016-08-15T19:40:27Z” level=info msg=“Execing [/usr/bin/nsenter --mount=/proc/1712/ns/mnt -F – /var/lib/docker/overlay/e7dd710e668361b4cd3fd751eec61d2f8d3876e92bd85de5083d254560c5799c/merged/usr/bin/share-mnt --stage2 /var/lib/kubelet /sys – kubelet-start.sh kubelet --kubeconfig=/etc/kubernetes/ssl/kubeconfig --api_servers=https://kubernetes:6443 --allow-privileged=true --register-node=true --cloud-provider=rancher --healthz-bind-address=0.0.0.0 --cluster-dns=169.254.169.250 --cluster-domain=cluster.local]“
8/15/2016 12:40:27 PMtime=“2016-08-15T19:40:27Z” level=warning msg=“signal: killed"
8/15/2016 12:40:27 PMtime=“2016-08-15T19:40:27Z” level=error msg=“Failed to start (pid 1748) &libcontainer.Process{Args:[]string{“kubelet-start.sh”, “kubelet”, “–kubeconfig=/etc/kubernetes/ssl/kubeconfig”, “–api_servers=https://kubernetes:6443”, “–allow-privileged=true”, “–register-node=true”, “–cloud-provider=rancher”, “–healthz-bind-address=0.0.0.0”, “–cluster-dns=169.254.169.250”, “–cluster-domain=cluster.local”}, Env:[]string{“CATTLE_CONFIG_URL=http://172.27.0.140:8080/v1”, “HOSTNAME=rancher”, “CATTLE_ACCESS_KEY=E342C371A69635E40C44”, “CATTLE_AGENT_INSTANCE_AUTH=Basic RTM0MkMzNzFBNjk2MzVFNDBDNDQ6VGZ1ZmRpdzVOZUxNb0ZoUUVDa0JWamZFWG9jZ2ZxRldwMmVHeENiQw==”, “CATTLE_SECRET_KEY=Tfufdiw5NeLMoFhQECkBVjfEXocgfqFWp2eGxCbC”, “PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin”, “PWD=/”, “SHLVL=1”, “HOME=/root”, “CATTLE_URL=http://172.27.0.140:8080/v1”, “_=/usr/bin/share-mnt”, “OLDPWD=/etc/kubernetes/ssl”}, User:“0:0”, Cwd:””, Stdin:(*os.File)(0xc820026128), Stdout:(*os.File)(0xc820026140), Stderr:(*os.File)(0xc820026150), ExtraFiles:[]*os.File(nil), consolePath:””, Capabilities:[]string(nil), ops:libcontainer.processOperations(nil)}: [8] System error: pivot_root invalid argument"
8/15/2016 12:40:27 PMtime=“2016-08-15T19:40:27Z” level=fatal msg="[8] System error: pivot_root invalid argument"

Thanks for any guidance.

7

@xb12pilot are you also running behind an http proxy?

8

I am not, so I think my setup is a bit simpler than yours. My environment is just ESX (in vSphere, but only using 1 host) in a lab network. Running the latest RancherOS 0.5 single node.

9

What version of Rancher server are you using?

Also, can you tell me the tag version on your kubernetes service? This would help me determine which template you launched.

10

Here’s a list of all images for each kubernetes component, in case it helps:

  • rancher/k8s:v1.2.4-rancher9

  • etcd: rancher/etcd:v2.3.7-6	 

  • kubectld: rancher/kubectld:v0.2.1

  • kubelet: rancher/k8s:v1.2.4-rancher9	 

  • kubernetes: rancher/k8s:v1.2.4-rancher9	 

  • proxy: rancher/k8s:v1.2.4-rancher9	 

  • rancher-ingress-controller: rancher/ingress-controller:v0.1.3

  • rancher-kubernetes-agent: rancher/kubernetes-agent:v0.2.1	 

  • scheduler: rancher/k8s:v1.2.4-rancher9
11

hi,
I have the same problem.

i run rancher server and agent using vmware in my computer.
and i find there is a host named ‘master’ in my network.
when i disconnect my computer 's network with the host named ‘master’, the setup is ok.