I am currently working on a build process that leverages a docker server and a handful of docker agents. If I use IP address for the Host Registration URL setting on the server, everything works fine. As soon as I use the hostname of the Rancher server as the URL (rancher.server.hostname:8080), everything fails. As long as I have the setting on the server using the IP, I can even use the hostname in the registration command when registering an agent when it works. I have tried sudo docker exec ping rancher.server.hostname from the agent and it resolves and pings and vice versa. I have opened up all firewall rules between the endpoints. My OS is Ubuntu 16.04. Docker version 1.13.0. Rancher server version is 1.3.3. Rancher agent version is 1.1.3.
I can curl from in the agent container to the server. Everything works on the same servers in the same setup as long as I flip the server setting from using the hostname to the IP address. I know this is a lot, but I figured more information would be better than not enough. I appreciate any help someone can provide. Another note is that I’ve removed the http://'s from certain spots due to posting rules.
When the Hostname is set in the Server, I see the logs below on the server:
2017-01-31 16:09:47,116 ERROR [:] [] [] [] [TaskScheduler-2] [i.c.p.a.s.ping.impl.PingMonitorImpl ] Failed to get ping from agent [6] count [3]
2017-01-31 16:09:52,118 ERROR [:] [] [] [] [TaskScheduler-2] [i.c.p.a.s.ping.impl.PingMonitorImpl ] Failed to get ping from agent [6] count [4]
2017-01-31 16:09:57,119 ERROR [:] [] [] [] [TaskScheduler-2] [i.c.p.a.s.ping.impl.PingMonitorImpl ] Failed to get ping from agent [6] count [5]
2017-01-31 16:10:02,120 ERROR [:] [] [] [] [TaskScheduler-2] [i.c.p.a.s.ping.impl.PingMonitorImpl ] Failed to get ping from agent [6] count [6]
2017-01-31 16:10:02,121 ERROR [:] [] [] [] [TaskScheduler-2] [i.c.p.a.s.ping.impl.PingMonitorImpl ] Scheduling reconnect for [6]
2017-01-31 16:10:07,130 ERROR [:] [] [] [] [TaskScheduler-2] [i.c.p.a.s.ping.impl.PingMonitorImpl ] Failed to get ping from agent [6] count [7]
2017-01-31 16:10:12,132 ERROR [:] [] [] [] [TaskScheduler-2] [i.c.p.a.s.ping.impl.PingMonitorImpl ] Failed to get ping from agent [6] count [8]
2017-01-31 16:10:17,134 ERROR [:] [] [] [] [TaskScheduler-2] [i.c.p.a.s.ping.impl.PingMonitorImpl ] Failed to get ping from agent [6] count [9]
Below (with DNS Names removed) are the logs I see on the Running rancher agent:
INFO: Starting agent for 8389590038D4CC68839D
INFO: Access Key: 8389590038D4CC68839D
INFO: Config URL: rancher.server.hostname:8080/v1
INFO: Storage URL: rancher.server.hostname:8080/v1
INFO: API URL: rancher.server.hostname:8080/v1
INFO: IP: 10.101.1.148
INFO: Port:
INFO: Required Image: rancher/agent:v1.1.3
INFO: Current Image: rancher/agent:v1.1.3
INFO: Using image rancher/agent:v1.1.3
INFO: Downloading agent rancher.server.hostname:8080/v1/configcontent/configscripts
INFO: Updating configscripts
INFO: Downloading rancher.server.hostname:8080/v1//configcontent//configscripts current=
INFO: Running /var/lib/cattle/download/configscripts/configscripts-1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319/apply.sh
INFO: Sending configscripts applied 1-f0f3fb2e1110b5ada7c441705981f93a480313a324294321cff467f0c3e12319
INFO: Updating pyagent
INFO: Downloading rancher.server.hostname:8080/v1//configcontent//pyagent current=
INFO: Running /var/lib/cattle/download/pyagent/pyagent-1-9e3dcab12585149a65d818082279708965cd826df1937a55b24f48428f8f0c0644fc1033b5be15c88dcf9540b1c04c67/apply.sh
INFO: Updating host-api
INFO: Downloading rancher.server.hostname:8080/v1//configcontent//host-api current=1-9608ead804f8cebbfe95b9f114701d8a5ac4244d603d196068043c902b1a0bdf
INFO: Already up to date
INFO: Sending host-api applied 1-9608ead804f8cebbfe95b9f114701d8a5ac4244d603d196068043c902b1a0bdf
INFO: Updating python-agent
INFO: Downloading rancher.server.hostname:8080/v1//configcontent//python-agent current=1-8f7ddbf845f3f27063a382644792cfa8bcbac969d38627eeafcdfb452d498458
INFO: Already up to date
INFO: Sending python-agent applied 1-8f7ddbf845f3f27063a382644792cfa8bcbac969d38627eeafcdfb452d498458
INFO: Sending pyagent applied 1-9e3dcab12585149a65d818082279708965cd826df1937a55b24f48428f8f0c0644fc1033b5be15c88dcf9540b1c04c67
INFO: Starting agent /var/lib/cattle/pyagent/apply.sh
INFO: Executing /var/lib/cattle/pyagent/agent
INFO: Updating host-config
INFO: Downloading rancher.server.hostname:8080/v1//configcontent//host-config current=host-config-1-7995525957ce359257e5e90e972d995f111f9bb1505f01da9ff1b499d71e6e0e
INFO: Already up to date
INFO: Sending host-config applied 1-7995525957ce359257e5e90e972d995f111f9bb1505f01da9ff1b499d71e6e0e
time=“2017-01-31T16:21:13Z” level=info msg="Launching agent"
time=“2017-01-31T16:21:13Z” level=info msg="rancher.server.hostname:8080/v18389590038D4CC68839D51FJfk6hexyEyn4wr4oNCM88XGfqNmgADe99mXAc"
time=“2017-01-31T16:21:13Z” level=info msg="Listening for events on rancher.server.hostname:8080/v1"
time=“2017-01-31T16:21:13Z” level=info msg="launching hostapi"
fatal error: unexpected signal during runtime execution
runtime stack:
runtime.throw(0x9cf035, 0x2a)
/usr/local/go/src/runtime/panic.go:566 +0x95
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:12 +0x2cc
goroutine 31 [syscall, locked to thread]:
Followed by a number of goroutine syscall.locked to thread errors.
The logs on the stopped rancher/agent container are below:
INFO: Running Agent Registration Process, CATTLE_URL=rancher.server.hostname:8080/v1
INFO: Attempting to connect to: rancher.server.hostname:8080/v1
INFO: rancher.server.hostname:8080/v1 is accessible
INFO: Inspecting host capabilities
INFO: Boot2Docker: false
INFO: Host writable: true
INFO: Token: xxxxxxxx
INFO: Running registration
INFO: Printing Environment
INFO: ENV: CATTLE_ACCESS_KEY=8389590038D4CC68839D
INFO: ENV: CATTLE_HOME=/var/lib/cattle
INFO: ENV: CATTLE_HOST_LABELS=label1=1
INFO: ENV: CATTLE_REGISTRATION_ACCESS_KEY=registrationToken
INFO: ENV: CATTLE_REGISTRATION_SECRET_KEY=xxxxxxx
INFO: ENV: CATTLE_SECRET_KEY=xxxxxxx
INFO: ENV: CATTLE_URL=http://rancher.server.hostname:8080/v1
INFO: ENV: DETECTED_CATTLE_AGENT_IP=1.2.3.4
INFO: ENV: RANCHER_AGENT_IMAGE=rancher/agent:v1.1.3
INFO: Launched Rancher Agent: 7e9fc24dc8fee47b155f1269c8ef264522ac071233c66e3b8556d8a3adf0b83f