Cattle Agent server "is not accessible"

Rancher 1.x
1

Following the quick start guide and the agent/cattle container can’t see the rancher/host. I’m guessing there’s an issue with DNS between the container and looping back to the other container on port 8080? It works when I spin it up locally on my own virtual machine lab in the office using the hostname/address rancher.local:8080 .

Bone-stock brand new and up to date Ubuntu 14.04. Used Docker’s getting started guide: https://docs.docker.com/engine/installation/linux/ubuntulinux/

UFW is enabled and forwarding is turned on: https://docs.docker.com/engine/installation/linux/ubuntulinux/#/enable-ufw-forwarding

Followed the Rancher getting started guide: h ttp://rancher.com/getting-started/

Cool it’s up, UI works etc:

Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 4.5.5-x86_64-linode69 x86_64)

* Documentation: https://help.ubuntu.com/ hadlock@argonaut:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 6fc3c451c433 rancher/server "/usr/bin/s6-svscan /" 3 days ago Up 2 days 3306/tcp, 0.0.0.0:8080->8080/tcp kickass_bardeen

Step 3 Add Hosts:

sudo docker run -e CATTLE_AGENT_IP="123.456.789.000" -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.0.2 http://domainname.us:8080/v1/scripts/70998A3EDF502798141B:1468742400000:UhA4SpXtlp817ExkmGMMYPXt3yS

output looks good:

[psudo] password for hadlock: 50d2a19ee43a312da7b822468779472add387312154a4f45ce025351f64d9d27 hadlock@argonaut:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 50d2a19ee43a rancher/agent:v1.0.2 "/run.sh http://domai" 6 seconds ago Up 5 seconds jolly_hopper 6fc3c451c433 rancher/server "/usr/bin/s6-svscan /" 3 days ago Up 2 days 3306/tcp, 0.0.0.0:8080->8080/tcp kickass_bardeen

Hmm it’s not showing up in the hosts… let’s look at the logs…

hadlock@argonaut:~$ docker logs -f 50d2a19ee43a

INFO: Running Agent Registration Process, CATTLE_URL=http://domainname.us:8080/v1 INFO: Checking for Docker version >= 1.6.0 INFO: Found Server version: 1.11.2 INFO: docker version: Client version: 1.6.0 INFO: docker version: Client API version: 1.18 INFO: docker version: Go version (client): go1.4.2 INFO: docker version: Git commit (client): 4749651 INFO: docker version: OS/Arch (client): linux/amd64 INFO: docker version: Server version: 1.11.2 INFO: docker version: Server API version: 1.23 INFO: docker version: Go version (server): go1.5.4 INFO: docker version: Git commit (server): b9f10c9 INFO: docker version: OS/Arch (server): linux/amd64 INFO: docker info: Containers: 17 INFO: docker info: Images: 4 INFO: docker info: Storage Driver: devicemapper INFO: docker info: Pool Name: docker-8:0-257306-pool INFO: docker info: Pool Blocksize: 65.54 kB INFO: docker info: Base Device Size: 10.74 GB INFO: docker info: Backing Filesystem: ext4 INFO: docker info: Data file: /dev/loop0 INFO: docker info: Metadata file: /dev/loop1 INFO: docker info: Data Space Used: 2.146 GB INFO: docker info: Data Space Total: 107.4 GB INFO: docker info: Data Space Available: 16.17 GB INFO: docker info: Metadata Space Used: 3.465 MB INFO: docker info: Metadata Space Total: 2.147 GB INFO: docker info: Metadata Space Available: 2.144 GB INFO: docker info: Udev Sync Supported: true INFO: docker info: Deferred Removal Enabled: false INFO: docker info: Deferred Deletion Enabled: false INFO: docker info: Deferred Deleted Device Count: 0 INFO: docker info: Data loop file: /var/lib/docker/devicemapper/devicemapper/data INFO: docker info: Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata INFO: docker info: Library Version: 1.02.77 (2012-10-15) INFO: docker info: Execution Driver: INFO: docker info: Kernel Version: 4.5.5-x86_64-linode69 INFO: docker info: Operating System: Ubuntu 14.04.4 LTS INFO: docker info: CPUs: 1 INFO: docker info: Total Memory: 1.953 GiB INFO: docker info: Name: argonaut.domainname.us INFO: docker info: ID: IOXD:WPMY:RK5G:ZHGV:ZORI:BJAT:VXWV:DNIW:ADK7:MSXB:5XKE:XBOF INFO: docker info: Http Proxy: INFO: docker info: Https Proxy: INFO: docker info: No Proxy: WARNING: No swap limit support INFO: Attempting to connect to: http://domainname.us:8080/v1 ERROR: http://domainname.us:8080/v1 is not accessible ^C hadlock@argonaut:~$ ^C

2

Ok and the first several questions will be “well can you even connect?” Here is some output from running docker exec

root@af8e72016649:/# ping rancher.com PING rancher.com (104.24.17.51) 56(84) bytes of data. 64 bytes from 104.24.17.51: icmp_seq=1 ttl=58 time=1.33 ms 64 bytes from 104.24.17.51: icmp_seq=2 ttl=58 time=1.43 ms ^C --- rancher.com ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1000ms rtt min/avg/max/mdev = 1.337/1.385/1.434/0.061 ms root@af8e72016649:/# ping domainname.us PING domainname.us (74.207.123.123) 56(84) bytes of data. 64 bytes from li89-11.members.linode.com (74.207.123.123): icmp_seq=1 ttl=64 time=0.047 ms 64 bytes from li89-11.members.linode.com (74.207.123.123): icmp_seq=2 ttl=64 time=0.091 ms ^C --- domsinnsmr.us ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 0.047/0.069/0.091/0.022 ms

Ok I can at least ping it. Cool. Can I hit the API endpoint?

root@af8e72016649:/# curl http://domainname.us:8080/v1 ^C root@af8e72016649:/# exit exit

Hmm it can’t talk to the api endpoint. Weird! If I curl google.com I have no problems. Well maybe the rancher endpoint is not accessible. Let’s exit and see what’s going on from the host, since I’m running the server and agent on the same box, as is explained in the quick start guide linked in the first post:

hadlock@argonaut:~$ curl http://domainname.us:8080/v1 {"id":"ee10fc84-79b7-4fde-bd92-068b74b07027","type":"error","links":{},"actions":{},"status":401,"code":"Unauthorized","message":"Unauthorized","detail":null}

Huh, works! Strange. So it’s definitely some kind of connectivity problem between the agent container and the server. But I have no idea what exactly.

3

And if from the command line of your host, you run a completely vanilla ubuntu container via:
docker run -it ubuntu bash, are you able to curl from it to the rancher server?

4

Hello,

I have same problem with rancher agent on CentOS 7. But if I use rancher-agent in version 1.0.1, it’s work.

5

So, I ended up just hard-coding everything. I pulled a vanilla ubuntu image and it did not work; however I went to Admin/Settings/Host Registration URL and set it to “Something else:”. I got the ip by doing docker exec -it rancher/server /bin/bash and running ifconfig and got the internal IP (in my case, 172.17.0.2) and set it to http://172.17.0.2:8080

Then I went to infrastructure/hosts/add host/custom/ and copied the text from step #5 (ignoring optional step #4) which looked something like this:

sudo docker run -d --privileged -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/rancher:/var/lib/rancher rancher/agent:v1.0.2 http://172.17.0.2:8080/v1/scripts/BF404332E6E09BBD6C59:1471744800000:ux0j3QaFj8XhnospXFFimN8CQ

Pasted that in to the console of my server and now it seems to be working just fine.

Kudos to your network guy for throwing up the big scary message about it being a private ip :wink:

But now it works, so, on to continuing to evaluate the product.