Hello,
I try to setup an Rancher Cluster using RKE. The cluster.yml contains three nodes “rancher-mgmt1-3” which all have internal IP Adresses and corresponing DNS entries. There are no local firewall rules.
The cluster.yml is basically a minimal yml as I’ve tried to nail the problem down by removing more and more and redeploying the cluster completely.
The cluster.yml looks like this:
nodes:
- address: rancher-mgmt1.some.domain
port: “22”
internal_address: “”
role:
- controlplane
- worker
- etcd
user: stefans
docker_socket: /var/run/docker.sock
ssh_key: “”
ssh_key_path: ~/.ssh/id_ed25519
ssh_cert: “”
ssh_cert_path: “”
labels: {}
taints: []
- address: rancher-mgmt2.some.domain
port: “22”
internal_address: “”
role:
- controlplane
- worker
- etcd
user: stefans
docker_socket: /var/run/docker.sock
ssh_key: “”
ssh_key_path: ~/.ssh/id_ed25519
ssh_cert: “”
ssh_cert_path: “”
labels: {}
taints: []
- address: rancher-mgmt3.some.domain
port: “22”
internal_address: “”
role:
- controlplane
- worker
- etcd
user: stefans
docker_socket: /var/run/docker.sock
ssh_key: “”
ssh_key_path: ~/.ssh/id_ed25519
ssh_cert: “”
ssh_cert_path: “”
labels: {}
taints: []
authentication:
strategy: x509
sans: [ “rancher-mgmt.some.domain” ]
ssh_agent_auth: true
authorization:
mode: rbac
ignore_docker_version: false
private_registries: []
ingress:
provider: “nginx”
cluster_name: “rancher-mgmt”
When I am running “rke up” the process starts and normally I am getting one Failure.
FATA[0483] Failed to get job complete status for job rke-network-plugin-deploy-job in namespace kube-system
Waiting some seconds and running “rke up” again everything seems to be fine.
The logs for the deployment of the “rke-network-plugin-deploy-job” show now Problems…but I am not sure if they get cleaned between the rke runs.
clusterrole.rbac.authorization.k8s.io/calico created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/canal-flannel created
clusterrolebinding.rbac.authorization.k8s.io/canal-calico created
configmap/canal-config created
daemonset.apps/canal created
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
After that my Kubernetes Cluster seems to be happy/healthy and usable.
kubectl get nodes
NAME STATUS ROLES AGE VERSION
rancher-mgmt1.some.domain Ready controlplane,etcd,worker 67m v1.17.14
rancher-mgmt2.some.domain Ready controlplane,etcd,worker 67m v1.17.14
rancher-mgmt3.some.domain Ready controlplane,etcd,worker 67m v1.17.14
After that I deploy the Network Test containers and run the provided test script one can find here.
The result is:
=> Start network overlay test
rancher-mgmt2.ops.some.domain can reach rancher-mgmt2.ops.some.domain
command terminated with exit code 1
FAIL: overlaytest-kz49x on rancher-mgmt2.ops.some.domain cannot reach pod IP 10.42.1.4 on rancher-mgmt3.ops.some.domain
command terminated with exit code 1
FAIL: overlaytest-kz49x on rancher-mgmt2.ops.some.domain cannot reach pod IP 10.42.0.3 on rancher-mgmt1.ops.some.domain
command terminated with exit code 1
FAIL: overlaytest-t27gf on rancher-mgmt3.ops.some.domain cannot reach pod IP 10.42.2.4 on rancher-mgmt2.ops.some.domain
rancher-mgmt3.ops.some.domain can reach rancher-mgmt3.ops.some.domain
command terminated with exit code 1
FAIL: overlaytest-t27gf on rancher-mgmt3.ops.some.domain cannot reach pod IP 10.42.0.3 on rancher-mgmt1.ops.some.domain
command terminated with exit code 1
FAIL: overlaytest-tg4l8 on rancher-mgmt1.ops.some.domain cannot reach pod IP 10.42.2.4 on rancher-mgmt2.ops.some.domain
command terminated with exit code 1
FAIL: overlaytest-tg4l8 on rancher-mgmt1.ops.some.domain cannot reach pod IP 10.42.1.4 on rancher-mgmt3.ops.some.domain
rancher-mgmt1.ops.some.domain can reach rancher-mgmt1.ops.some.domain
=> End network overlay test
I could also find the reason why it is not working. It looks like the packages never get marked for forwarding in iptables. The default rules/chains seem to be present:
Chain KUBE-FORWARD (1 references)
target prot opt source destination
DROP all – 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
ACCEPT all – 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules / mark match 0x4000/0x4000
ACCEPT all – 10.42.0.0/16 0.0.0.0/0 / kubernetes forwarding conntrack pod source rule / ctstate RELATED,ESTABLISHED
ACCEPT all – 0.0.0.0/0 10.42.0.0/16 / kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED
When I add an ACCEPT rule to the KUBE-FORWARD chain which is not checking for the mark it looks like this:
iptables -A KUBE-FORWARD -s 0.0.0.0/0 -d 0.0.0.0/0 -j ACCEPT
=> Start network overlay test
rancher-mgmt2.ops.some.domain can reach rancher-mgmt2.ops.some.domain
rancher-mgmt2.ops.some.domain can reach rancher-mgmt3.ops.some.domain
rancher-mgmt2.ops.some.domain can reach rancher-mgmt1.ops.some.domain
rancher-mgmt3.ops.some.domain can reach rancher-mgmt2.ops.some.domain
rancher-mgmt3.ops.some.domain can reach rancher-mgmt3.ops.some.domain
rancher-mgmt3.ops.some.domain can reach rancher-mgmt1.ops.some.domain
rancher-mgmt1.ops.some.domain can reach rancher-mgmt2.ops.some.domain
rancher-mgmt1.ops.some.domain can reach rancher-mgmt3.ops.some.domain
rancher-mgmt1.ops.some.domain can reach rancher-mgmt1.ops.some.domain
=> End network overlay test
The Rules gets of course re-writen after some time by kubeproxy so adding that does not seem to be a solution.
So the question is now -> what did I miss? Is there a problem with my cluster.yml (I don’t think so because it is pretty minimal).
I can’t think of some missconfigurations of the nodes. No own firewall rules, all located in the same subnet and so on. It just looks like the mark rules for new containers do not get created and so the forwarding is not allowed.
I also did a lot of tcpdumping and there as well everything looks fine. When connecting to a container on another node the traffic gets routed into the flannel interface. When forwarding is allowed everything work.
What have I missed?
What configuration could I check to solve the problem. The cluster is completely new…so I guess it just should work.
Many thanks for any hint or solution!
best regards,
Stefan