I don’t seem to be able to access containers via the host’s IP although the ports are mapped.
I want to use AWS AutoScaling groups to spin up servers. The AS has an ELB attached to it, with health checks.
When AS spins up a new server, it installs docker and runs the rancher/agent to register with rancher and set up some labels. This works great.
Rancher then installs some service on the hosts based on the labels, works great. In this particular case, I’m running consul. The docker-compose.yml properly maps ports to the host and looks like:
consul: image: progrium/consul ports: - 8400:8400 - 8500:8500 - 8300:8300 - 8301:8301 - 8301:8301/udp - 8302:8302 - 8302:8302/udp - 53:53/udp command: -server -bootstrap -ui-dir /ui labels: io.rancher.scheduler.global: 'true' io.rancher.scheduler.affinity:host_label: consul=true
Now I’m testing the connection to consul, using a simple curl call:
If I ssh to the host running the consul container and curl the IP of the container (assigned by rancher in the 10.42 range) then this works fine. From the same host, if I try to curl the IP of the host itself, I get:
curl: (7) Failed to connect to <IP> port 8500: Connection refused
The hosts are in a private subnet, and the attached security groups allow all incoming traffic between hosts themselves, all incoming traffic from the bastion server (which runs rancher server), and all incoming traffic from an ELB security group. I’ve also tried spinning up a new host directly from rancher ui, and I get the exact same behavior.
So the ELB/AS will eventually fail the health checks and teardown the host and spin a new one up and start again.
When I inspect the consul container in rancher ui, all the ports are mapped and active.
UFW is disabled and here’s iptables from the host:
root@ip-10-0-2-88:~# iptables -L -n Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain DOCKER (1 references) target prot opt source destination ACCEPT udp -- 0.0.0.0/0 172.17.0.4 udp dpt:4500 ACCEPT udp -- 0.0.0.0/0 172.17.0.4 udp dpt:500
I found a topic that describes a similar problem to mine but the solution there didn’t help me as ufw was already disabled
Rancher v0.31.0 Cattle v0.73.0 User Interface v0.39.0 Rancher Compose beta/latest