[ppc64le] IPSEC infrastructure stack service unable to start on added host

Hi All,

Am newbie to rancher , have added a new environment and hosts to my rancher server and m trying to install infrastructure service ( ipsec ) , on added rancher hosts added. I have been able to see infra services like “network-services and network policy manager” running successfully on rancher/agent. However while booting ipsec , it fails to boot successfully. My environment details are .
On Rancher/server version : v1.6.7 - Docker version : 1.12.6 - ip address : 10.51.226.189
host version :rancher/agent version v1.2.5 , Docker version : 17.06.1-ce - ip address : 10.88.67.96 , so basically both rancher/server and hosts are on the same network.

Inside IPsec stack too , am able to successfully see “cni-driver in ipsec” running , however ‘ipsec in ipsec’ shows -
"ipsec + 1 Sidekick (Expected state running but got stopped) "

On hosts
root@pts00449-vm29:~# docker ps -a | grep ipsec
44525e5dc7be xxxxxx “/rancher-entrypoi…” 46 hours ago Exited (2) About a minute ago r-ipsec-ipsec-router-1-6e11bd00
abdc07b8dc9f xxxx “/.r/r /rancher-en…” 46 hours ago Exited (1) About a minute ago r-ipsec-ipsec-1-b32671d4
883e4fbe0b95 xxxx “/rancher-entrypoi…” 7 days ago Up 7 days r-ipsec-cni-driver-1-591a2b5b

As can be seen above ipsec-cni driver is only running and other 2 containers of ipsec are exiting.

Also docker logs for above container
root@pts00449-vm29:~# docker exec -it $(docker ps -a | grep r-ipsec-ipsec-router-1-6e11bd00 | awk ‘{print $1}’) bash
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:255: creating new parent process caused “container_linux.go:1462: running lstat on namespace path “/proc/322498/ns/net” caused “lstat /proc/322498/ns/net: no such file or directory””

root@pts00449-vm29:~# docker exec -it $(docker ps -a | grep r-ipsec-ipsec-router-1-6e11bd00 | awk ‘{print $1}’) bash
Error response from daemon: Container 44525e5dc7bef723b03a07c566028101d6b28639251257464aef50be7672ec25 is not running
root@pts00449-vm29:~# docker exec -it $(docker ps -a | grep r-ipsec-ipsec-router-1-6e11bd00 | awk ‘{print $1}’) bash
Error response from daemon: Container 44525e5dc7bef723b03a07c566028101d6b28639251257464aef50be7672ec25 is not running

root@pts00449-vm29:~# docker exec -it $(docker ps -a | grep r-ipsec-ipsec-1-b32671d4 | awk ‘{print $1}’) bash
Error response from daemon: Container abdc07b8dc9f0b10a1ea8e742844c52e0d546141d6deee5b62989c9bc7d90349 is not running

@leodotcloud - saw ur name for few of github issues on rancher/ipsec + forums , so thought of tagging you here.

let me know if u need any more information on above or pointers if any.

@ghatwala Can you please collect logs using https://github.com/leodotcloud/rancher-logs-collector and share? What kind of OS are you using? How many hosts do you have? Is this the first host or a host added to an existing cluster? Do you have any kind of firewall running on your host? Where are these hosts running? AWS or GCE or private datacenter?
Also can you please file a github issue with all the details?

What kind of OS are you using?
A: Am using Ubuntu-16.04 on both server+client and my set up is such that rancher/server-v1.6.7 is running on x86_64 and my hosts added are of “ppc64le” architecture.

How many hosts do you have? Is this the first host or a host added to an existing cluster?
A: This was the very first host (ppc64le) added to server.

Do you have any kind of firewall running on your host?
A : No -
root@pts00449-vm29:~# ufw status
Status: inactive

Where are these hosts running? AWS or GCE or private datacenter?
A: ppc64le hosts are nothing but plain vm guests on a power KVM ( x86 server and ppc64le hosts are in the same network with no firewall ).

As regards to the readme steps here https://github.com/rancher/rancher-logs-collector . am going to try it on server (x86_64 ) and get back to u on git issue

just a heads up on rancher cli @leodotcloud .
So i installed rancher-cli on x86_64 acting as server
root@hj-ibmibm519:~# rancher --version
rancher version v1.0.0-alpha1

while configuring the cli using steps here : http://rancher.com/docs/rancher/v1.2/en/cli/
getting below error
root@hj-ibmibm519:~# rancher config
URL []: http://10.51.226.189:8080
Access Key []: "3E4802DE76719C04823B"
Secret Key []: "aZfEpWGBLiYu6UP3YTMJqUjZaRSsBgpwMeKkmwD9"
FATA[0030] Failed to find schema at [http://10.51.226.189:8080/v3]

Any idea how to get over this error so that i can fetch the using ur script : ./rancher_logs_collector.sh. ? Any pointers are welcome here.

update : after installing cli for rancher/server 1.6.7 .
$ rancher --version
rancher version v0.6.3

now rancher-cli is working

moved the discussion to git issue : https://github.com/rancher/rancher/issues/9925

@ghatwala Just saw you mentioned Architecture: ppc64le. I don’t think we have support for PPC. I am not sure if there is full support from Docker itself. If you need special support for PPC, please feel free to reach out to our Sales team.

There is a lot of work involved to get all the microservices cross compiled and rebuild all of the infra containers using the PPC Base images.

@leodotcloud could you provide a phone number for sales, I have a question for them. TIA, Todd

problem has been fixed now ( ipsec is successfully booting up on added ppc64le host ), last update here : https://github.com/rancher/rancher/issues/9925