Wrong IP given to Pod

Rancher 2.x
1

Hello,

I’m using Calico as the CNI in my Rancher launched K8s cluster.

image
image706×273 8.34 KB

The calico IP pool is the following :
image

When I deploy a Pod, it gets a correct IP (10.42.X.X). However, sometime the Pod is restarted and gets a wriong IP (172.17.X.X.). It seems that 172.17.X.X is the docker0 bridge ip pool.

Have you an idea on why it gives sometimes a wrong ip ? Have you got the same issue ?

Regards

2

Can you do a kubectl cluster-info dump | grep -m 1 cluster-cidr

3

Here is the cluster-cidr :

# ps -ef | grep "cluster-cidr"
root      1918  1839  0 Nov19 ?        00:07:43 kube-proxy --v=2 --healthz-bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 ... ... ...
root      5909  5859  1 08:37 ?        00:00:05 kube-controller-manager --cloud-provider= --cluster-cidr=10.42.0.0/16 ... ... ...
4

No one has got the same kind of issue ?

Regards

5

Ok, so it is a bug related to Rancher :


6

Hi @pcasis
we had the same issue on one cluster. Pods running on one specific worker-node got the pod IP from docker-bridge. Drain, delete and reconfigure this worker-node helped.
Maybe that’s related to Kubernetes upgrade to 1.16.3.

How can we monitor stuff like this? We came only across this issue because of application connection errors.

7

Our Kubernetes version is 1.15.

Good question :slight_smile: We’ve enabled Monitoring and Logs tools (Grafana/Prometheus/Loki). But those containers are impacted by the IP issue so we can’t really investigate on it … :cry: