Cannot make kubernetes cluster work in rancher - kube-proxy issue

1

First of all - hello all :slight_smile:

I love Rancher - looks really promising - but have one issue.

i am trying to setup kubernetes cluster on such machines:

  • Red Hat 7.2
    • user namespace enabled in kernel
  • docker 1.10.3
    • selinux enabled

I have 3 hosts with such configuration. On one I just run rancher/server 1.1.2- nothing more.
The other two are ment to be kubernetes cluster.

Now - If I add hosts to the Kubernetes environment they create cluster with no issues. BUT if I create service with one container running on one host I can connect to this service only via host where it runs.
Service and RC are configured with NodePort and 1 replica.
kube-proxy says : Dial failed: dial tcp 172.17.0.6:8080: i/o timeout

Both docker engines have default network configuration.

I tried also with servers with selinux and user namespace disbled - still no luck.
I tried add flannel to docker and then setup kubernetes cluster from rancher but then kubernetes went mad :smiley:

Could you tell me what is that thing ?

If you need some more information just tell me what is needed.

Thanks!

edit :
fixed it by adding static routes on the nodes.

2

Hi xifedi
did you get an answer for your issue? I’m running into sort of the same issue. I can’t access the cluster from my workstation even after using the config file generated by Rancher. Let me know. Thanks

3

@go4875 I’m not sure if you saw @xifedi 's edit on how he fixed it.

4

sorry for late response - was a bit busy.

So yes - i have fixed it by changing docker network on nodes to:
node 1 (lets say node IP 192.168.10.11/24) : docker net :172.17.1.0/24
node 2 (192.168.10.12/24) : 172.17.2.0/24
node 3 (192.168.10.13/24) : 172.17.3.0/24
and by adding static routes :
on node 1:

  • route to 172.17.2.0/24 via 182.168.10.12
  •                 route to 172.17.3.0/24 via 182.168.10.13
    on node 2:
  • route to 172.17.1.0/24 via 182.168.10.11
  • route to 172.17.3.0/24 via 182.168.10.13
    on node 3:
  • route to 172.17.1.0/24 via 182.168.10.11
  • route to 172.17.2.0/24 via 182.168.10.12

As I said kubernetes cluster services/containers in archer all were green but proxy could not reach pod/container on the other nodes.

or maybe simply I should configure network as subnets of kubernetes cluster network ? 10.42.x.x/24 or something like that ?

I wanted to see how rancher works before I recommend it for QC/Prod environment. For now it is a bit disappointing.

The next thing is when I create service in particular namespace in kubernetes it is shown as standalone until I restart the whole cluster. Moreover when I take node where pod is running down it is moved to another node (as it should) but it is shown as standalone again… I am talking about host view .

If somebody knows what I am doing wrong I would appreciate some hints :wink: These artificial static routes are not a fix in fact but workaround.

Thanks!

ps: I didn’t try to use kubectl from my workstation.

5

hi @xifedi. Thank You so much for the feedback. I agree with you, i don’t really think that is the solution. I am looking at Rancher as the main solution for our CaaS, what i was looking forward also, was using the docker-machine drivers to use on our internally developed IaaS. Will continue testing further, hoping Rancher can provide a more automated solution for deploying Kubernetes environment.
@denise - thoughts on your side? Thanks
Have a good weekend to all

6

Hi,

after hours of digging it looks like io.rancher.container.network=true label is not set for docker containers created by kubernetes.

My docker network is default on all hosts : 172.17.0.0/16
rancher network is set to 172.18.0.0/16.
Kubernetes cluster components are assigned IP addresses from rancher network - so it is fine.
When I create service and replication controller pod always get IP address from 172.17.0.0/16 range - WRONG.

Any idea how to add it durig RC / SVC creation ? I do not plan to run any containers as standalone so setting this label for all newly created containers is fine as well.

Seriously I am really tired to make it work so this is the last thing I want to try before I give up …

7

any thoughts on this issue ?
I tried with rancher 1.1.2, 1.1.3 and v1.2.0-pre2 but still no luck.
Pods always gets IP from docker network instead of rancher network ;/
Only if I create container it gets rancher IP…

8

uninstalling docker-selinux package from red hat 7.2 and removing --selinux-enable from docker daemon options solved problem but exposed host filesystem as unrestricted to containers. It is possible to modify host filesystem without any restrictions including whole /etc … This is unacceptable

Question to Rancher guys - what selinux settings are required for rancher to run correctly ?

1 Like
9

I have same problem on Ubuntu 14.04.4.
I don’t have any selinux installed on host servers. Still every pod i creating gets ip from 172.17.0.0/16 range 8(

10

Has there been any resolution to this? I’m trying to get this to work with ubuntu 16 and it always dies on the proxy.

Should I be setting io.rancher.container.network=true somewhere in the config of each of the hosts? I tried adding that without any success. I should also mention that all of my ip addresses are assigned prior to the deployment and I’m also having trouble with the kubelet process.