Troubleshooting cross host communication - SOLVED

Rancher 1.x
1

I’ve been trying for a few days.

I am running rancher agent v 1.2.1 on docker 17.03.1-ce on Ubuntu 16.04.2 LTS in OpenStack.

I have enabled DEBUG logs for IPSEC, the only mildly useful logs I get from the IPSEC router image are:

10/04/2017 15:20:12+ curl http://localhost:8111
10/04/2017 15:20:12+ export CHARON_PID_FILE=/var/run/charon.pid
10/04/2017 15:20:12+ CHARON_PID_FILE=/var/run/charon.pid
10/04/2017 15:20:12+ rm -f /var/run/charon.pid
10/04/2017 15:20:12+ export PIDFILE=/var/run/rancher-net.pid
10/04/2017 15:20:12+ PIDFILE=/var/run/rancher-net.pid
10/04/2017 15:20:12+ GCM=false
10/04/2017 15:20:12+ (( i=0 ))
10/04/2017 15:20:12+ (( i<6 ))
10/04/2017 15:20:12+ ip xfrm state add src 1.1.1.1 dst 1.1.1.1 spi 42 proto esp mode tunnel aead 'rfc4106(gcm(aes))' 0x0000000000000000000000000000000000000001 128 sel src 1.1.1.1 dst 1.1.1.1
10/04/2017 15:20:12/usr/bin/start.sh: line 26:     9 Segmentation fault      ip xfrm state add src 1.1.1.1 dst 1.1.1.1 spi 42 proto esp mode tunnel aead "rfc4106(gcm(aes))" 0x0000000000000000000000000000000000000001 128 sel src 1.1.1.1 dst 1.1.1.1
10/04/2017 15:20:12+ ip xfrm state del src 1.1.1.1 dst 1.1.1.1 spi 42 proto esp```

Any suggestions on where to start looking to debug this?
2

https://pastebin.com/5K8K6ee1

Here’s the dmesg output from the segfault!

3

Solved!

I downgraded to Ubuntu 14… The problem was Openstack poorly supporting the AVX extensions in our environment.

2 Likes
4

Hey, thanks man, you saved my live. I spend the whole two days on this error, tried different cloud provider, just didn’t try different operation system(or system version). Ubuntu 16.04 works on hosts of virtualbox on my Mac, works on hosts of DigitOcean, doesn’t work on UCloud, when I downgrade system to 14.04, it works!!! Thanks again.

I wonder how you figured out this problem, I just saw ipsec-ipsec-router error message:

9/17/2017 10:18:03 PM+ trap 'exit 1' SIGTERM SIGINT
9/17/2017 10:18:03 PM+ curl http://localhost:8111
9/17/2017 10:18:03 PM+ export CHARON_PID_FILE=/var/run/charon.pid
9/17/2017 10:18:03 PM+ CHARON_PID_FILE=/var/run/charon.pid
9/17/2017 10:18:03 PM+ rm -f /var/run/charon.pid
9/17/2017 10:18:03 PM+ export PIDFILE=/var/run/rancher-net.pid
9/17/2017 10:18:03 PM+ PIDFILE=/var/run/rancher-net.pid
9/17/2017 10:18:03 PM+ GCM=false
9/17/2017 10:18:03 PM+ (( i=0 ))
9/17/2017 10:18:03 PM+ (( i<6 ))
9/17/2017 10:18:03 PM+ ip xfrm state add src 1.1.1.1 dst 1.1.1.1 spi 42 proto esp mode tunnel aead 'rfc4106(gcm(aes))' 0x0000000000000000000000000000000000000001 128 sel src 1.1.1.1 dst 1.1.1.1
9/17/2017 10:18:03 PM/usr/bin/start.sh: line 26:     9 Segmentation fault      ip xfrm state add src 1.1.1.1 dst 1.1.1.1 spi 42 proto esp mode tunnel aead "rfc4106(gcm(aes))" 0x0000000000000000000000000000000000000001 128 sel src 1.1.1.1 dst 1.1.1.1
9/17/2017 10:18:03 PM+ ip xfrm state del src 1.1.1.1 dst 1.1.1.1 spi 42 proto esp
5

I got to the same error. The segmentation fault was what got me looking
towards the kernel and then to instruction sets after I did some research
on how IPSEC worked.

6

Ok, thank you very much.

7

MartinL what version of OpenStack were you running this on? I’m on ocata and also getting issues with the ipsec routers.