Upgraded to v0.56 and hosts are getting local addresses

Rancher 1.x
1

Good morning,

I kicked off the upgrade to v0.56 this morning. Everything seemed to come up fine but my containers are now getting a local 172 address instead of 10.42.

Anyone have any insight or where to look to troubleshoot this?

Thanks

2

Looks like the IP address wasn’t the issue. I need to learn to us “ip addr show” instead of ifconfig.
Still troubleshooting an issue with LB connections

3

What OS are you running? If you are running CoreOS, please upgrade to v0.56.1.

Have you read the notes on how to troubleshoot if you are having networking issues from the release notes?

4

I’m running on Ubuntu. I’ve tried server restarts but it did not resolve the issue.
I’ve also upgraded to v0.56.1 just to see if that would help.

This is in the server logs.

INFO: Updating ipsec
INFO: Downloading http://xxxxxxxx:8080/v1//configcontent//ipsec current=ipsec-1-d22e5c81b7176d7d19dc905df94d640253c260d96dd697bfdcaa5fb8a047da47
/var/lib/cattle/common/scripts.sh: line 62: 12957 Terminated ${CATTLE_HOME}/config.sh “$@”
/var/lib/cattle/config.sh: line 8: 1886 Terminated $curl -H “Authorization: $CATTLE_AGENT_INSTANCE_AUTH” "$@"1.000

5

@fnord I found out that there were some issues where network agent restarts/host reboots weren’t fixing connectivity sometimes and this was fixed in v0.56.1.

Can you also do this command for us? Exec into the network agent and do a ps-eaf | grep rancher-net. You should get something like below. Ideally, for ubuntu, centos, rancheros, it should have --gcm=true and for coreos, it should have --gcm=false. I know you’re running ubuntu for your hosts, but wanted to share the information for anyone else who looks at the issue.


root      3659     1  0 Jan26 ?        00:01:57 /var/lib/cattle/bin/rancher-net 
--log /var/log/rancher-net.log -f /var/lib/cattle/etc/cattle/ipsec/config.json -
c /var/lib/cattle/etc/cattle/ipsec -i 172.17.0.2/16 --pid-file /var/run/rancher-
net.pid --gcm=true                                                              
root     18025 18021  0 20:10 pts/0    00:00:00 grep rancher-net

In v0.56.1, this was fixed, so just want to make sure this isn’t your issue.

6

Interesting. Mine is set to false even though it is Ubuntu.

root@d432eefa20de:/# ps -eaf | grep rancher-net
root 1247 1 0 19:40 ? 00:00:03 /var/lib/cattle/bin/rancher-net
–log /var/log/rancher-net.log -f /var/lib/cattle/etc/cattle/ipsec/config.json -
c /var/lib/cattle/etc/cattle/ipsec -i 172.17.0.24/16 --pid-file /var/run/rancher
-net.pid --gcm=false
root 1946 1942 0 20:35 pts/0 00:00:00 grep rancher-net

7

The issue was fixed be deleting the network agents and having Rancher recreate them.
Thanks for everyones help!

8

Glad you got it fixed!