'Failed to find a placement' error for port 5000

celine · September 16, 2015, 3:21pm

I observed something weird on Rancher 0.37.0
My environment:
Rancher v0.37.0
Cattle v0.91.0
UI v0.47.0
Docker 1.8.2

I have a haproxy container that I use as a SSL proxy for a Docker private registry.
Both services (haproxy + registry) are running on Rancher on 1 host, both configured to use Managed Network.
The registry container runs on port 5000 by default (I didn’t add any port/network configuration for this container)
The haproxy container listens on port 5000 but I’m mapping that port to port 5000 on the host.

The weird thing is that haproxy wouldn’t start. I’ll get these errors in the rancher server:

2015-09-15 15:48:43,884 ERROR [7bf876d9-9542-45a1-9deb-e0d19756339e:199] [instance:5] [instance.start->(InstanceStart)] [] [ecutorService-2] [i.c.p.process.instance.InstanceStart] Failed to Networking for instance [5] 
2015-09-15 15:48:48,811 ERROR [8488b132-1e9e-45d6-9904-b07dfafc2b7f:212] [instance:7] [instance.start->(InstanceStart)->instance.allocate->(InstanceAllocate)] [] [utorService-155] [c.p.a.e.i.AllocatorEventListenerImpl] No allocator handled [EventVO [id=510cb808-83f7-464e-935e-c3fbabac2b13, name=instance.allocate, previousNames=null, replyTo=reply.1532895759587658000, resourceId=7, resourceType=instance, publisher=null, transitioning=null, transitioningMessage=null, transitioningInternalMessage=null, previousIds=null, data={}, time=Tue Sep 15 15:48:48 UTC 2015, listenerKey=null, transitioningProgress=null]] 

2015-09-15 15:48:48,814 ERROR [8488b132-1e9e-45d6-9904-b07dfafc2b7f:212] [instance:7] [instance.start->(InstanceStart)->instance.allocate] [] [ecutorService-8] [c.p.e.p.i.DefaultProcessInstanceImpl] Unknown exception io.cattle.platform.eventing.exception.EventExecutionException: Failed to find a placement

It’s reacting as if port 5000 was taken on the host… which is not the case. Nothing was using that port on the server (I didn’t find anything) but impossible to start the service.
I have another haproxy container configured the same way but with port 80 (apps running on port 80 and haproxy listening on port 80 and mapping the port on the host) and it’s working.
I changed the mapping so haproxy still listens on port 5000 but maps to port 443 on the host. And it is working.

This is not an issue as I don’t care using port 443 instead of 5000 but I couldn’t explain why it wouldn’t work on port 5000… so that’s why I’m sharing it with you.

Notes:
I could make it work on port 5000 on Rancher 0.28.0
It might be possible to replace that haproxy service by a Rancher LoadBalancer but I haven’t tried it.

denise · September 16, 2015, 8:55pm

Can you share the iptables on the host to see what is using port 5000?

iptables-save on the host should provide you those details. You’ll need to run as root.

celine · September 17, 2015, 8:54am

It seems that the Docker registry was taking the port even if the port mapping wasn’t set up in the service:

# Generated by iptables-save v1.4.21 on Thu Sep 17 08:27:12 2015
*mangle
:PREROUTING ACCEPT [5703302:2616278424]
:INPUT ACCEPT [506149:1616674245]
:FORWARD ACCEPT [5202300:999948199]
:OUTPUT ACCEPT [477795:1158566597]
:POSTROUTING ACCEPT [5680086:2158513843]
COMMIT
# Completed on Thu Sep 17 08:27:12 2015
# Generated by iptables-save v1.4.21 on Thu Sep 17 08:27:12 2015
*nat
:PREROUTING ACCEPT [1:60]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [3:180]
:CATTLE_POSTROUTING - [0:0]
:CATTLE_PREROUTING - [0:0]
:DOCKER - [0:0]
-A PREROUTING -j CATTLE_PREROUTING
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -j CATTLE_POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p udp -m udp --dport 4500 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p udp -m udp --dport 500 -j MASQUERADE
-A POSTROUTING -s 172.17.0.27/32 -d 172.17.0.27/32 -p tcp -m tcp --dport 8080 -j MASQUERADE
-A CATTLE_POSTROUTING -s 10.42.0.0/16 -d 169.254.169.250/32 -j ACCEPT
-A CATTLE_POSTROUTING -s 10.42.0.0/16 ! -d 10.42.0.0/16 -p tcp -j MASQUERADE --to-ports 1024-65535
-A CATTLE_POSTROUTING -s 10.42.0.0/16 ! -d 10.42.0.0/16 -p udp -j MASQUERADE --to-ports 1024-65535
-A CATTLE_POSTROUTING -s 10.42.0.0/16 ! -d 10.42.0.0/16 -j MASQUERADE
-A CATTLE_POSTROUTING -s 172.17.0.0/16 ! -o docker0 -p tcp -j MASQUERADE --to-ports 1024-65535
-A CATTLE_POSTROUTING -s 172.17.0.0/16 ! -o docker0 -p udp -j MASQUERADE --to-ports 1024-65535
-A CATTLE_PREROUTING -p udp -m addrtype --dst-type LOCAL -m udp --dport 4500 -j DNAT --to-destination 10.42.187.72:4500
-A CATTLE_PREROUTING -p udp -m addrtype --dst-type LOCAL -m udp --dport 500 -j DNAT --to-destination 10.42.187.72:500
-A CATTLE_PREROUTING -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 5000 -j DNAT --to-destination 10.42.9.17:5000
-A CATTLE_PREROUTING -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 27017 -j DNAT --to-destination 10.42.5.234:27017
-A CATTLE_PREROUTING -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 80 -j DNAT --to-destination 10.42.116.66:80
-A DOCKER ! -i docker0 -p udp -m udp --dport 4500 -j DNAT --to-destination 172.17.0.2:4500
-A DOCKER ! -i docker0 -p udp -m udp --dport 500 -j DNAT --to-destination 172.17.0.2:500
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.17.0.27:8080
COMMIT
# Completed on Thu Sep 17 08:27:12 2015
# Generated by iptables-save v1.4.21 on Thu Sep 17 08:27:12 2015
*filter
:INPUT ACCEPT [225278:345112214]
:FORWARD ACCEPT [3768:232564]
:OUTPUT ACCEPT [220390:433976407]
:DOCKER - [0:0]
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p udp -m udp --dport 4500 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p udp -m udp --dport 500 -j ACCEPT
-A DOCKER -d 172.17.0.27/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8080 -j ACCEPT
COMMIT
# Completed on Thu Sep 17 08:27:12 2015

I scaled up/down the service (I wanted to start/stop but clicked on “+ Scale Up” by mistake). So a new container was created for the service and it freed the port:

# Generated by iptables-save v1.4.21 on Thu Sep 17 08:29:32 2015
*mangle
:PREROUTING ACCEPT [5711894:2619299821]
:INPUT ACCEPT [507473:1618081369]
:FORWARD ACCEPT [5209568:1001562472]
:OUTPUT ACCEPT [479259:1159952981]
:POSTROUTING ACCEPT [5688818:2161514500]
COMMIT
# Completed on Thu Sep 17 08:29:32 2015
# Generated by iptables-save v1.4.21 on Thu Sep 17 08:29:32 2015
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [1:60]
:CATTLE_POSTROUTING - [0:0]
:CATTLE_PREROUTING - [0:0]
:DOCKER - [0:0]
-A PREROUTING -j CATTLE_PREROUTING
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -j CATTLE_POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p udp -m udp --dport 4500 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p udp -m udp --dport 500 -j MASQUERADE
-A POSTROUTING -s 172.17.0.27/32 -d 172.17.0.27/32 -p tcp -m tcp --dport 8080 -j MASQUERADE
-A CATTLE_POSTROUTING -s 10.42.0.0/16 -d 169.254.169.250/32 -j ACCEPT
-A CATTLE_POSTROUTING -s 10.42.0.0/16 ! -d 10.42.0.0/16 -p tcp -j MASQUERADE --to-ports 1024-65535
-A CATTLE_POSTROUTING -s 10.42.0.0/16 ! -d 10.42.0.0/16 -p udp -j MASQUERADE --to-ports 1024-65535
-A CATTLE_POSTROUTING -s 10.42.0.0/16 ! -d 10.42.0.0/16 -j MASQUERADE
-A CATTLE_POSTROUTING -s 172.17.0.0/16 ! -o docker0 -p tcp -j MASQUERADE --to-ports 1024-65535
-A CATTLE_POSTROUTING -s 172.17.0.0/16 ! -o docker0 -p udp -j MASQUERADE --to-ports 1024-65535
-A CATTLE_PREROUTING -p udp -m addrtype --dst-type LOCAL -m udp --dport 4500 -j DNAT --to-destination 10.42.187.72:4500
-A CATTLE_PREROUTING -p udp -m addrtype --dst-type LOCAL -m udp --dport 500 -j DNAT --to-destination 10.42.187.72:500
-A CATTLE_PREROUTING -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 27017 -j DNAT --to-destination 10.42.5.234:27017
-A CATTLE_PREROUTING -p tcp -m addrtype --dst-type LOCAL -m tcp --dport 80 -j DNAT --to-destination 10.42.116.66:80
-A DOCKER ! -i docker0 -p udp -m udp --dport 4500 -j DNAT --to-destination 172.17.0.2:4500
-A DOCKER ! -i docker0 -p udp -m udp --dport 500 -j DNAT --to-destination 172.17.0.2:500
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.17.0.27:8080
COMMIT
# Completed on Thu Sep 17 08:29:32 2015
# Generated by iptables-save v1.4.21 on Thu Sep 17 08:29:32 2015
*filter
:INPUT ACCEPT [226602:346519338]
:FORWARD ACCEPT [3772:232804]
:OUTPUT ACCEPT [221854:435362791]
:DOCKER - [0:0]
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p udp -m udp --dport 4500 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p udp -m udp --dport 500 -j ACCEPT
-A DOCKER -d 172.17.0.27/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8080 -j ACCEPT
COMMIT
# Completed on Thu Sep 17 08:29:32 2015

Still weird that the previous container was taking the port but I can now map Haproxy to port 5000. Thanks for your answer.

Topic		Replies	Views
Rancher Server behind haproxy Rancher 1.x	2	1092	April 11, 2017
Trying to run-my-own-elk-stack-with-docker-and-rancher Rancher 1.x	5	1855	August 23, 2017
Network Agent fails to start Rancher 1.x	1	1939	October 26, 2015
Failed to find nic Rancher 1.x	3	1348	August 29, 2015
Rancher HA and Docker versions Rancher 1.x	8	1532	February 14, 2017

'Failed to find a placement' error for port 5000

Related topics