We are using the Docker distribution installed from the Containers_Module_12_x86_64 package.
We have containers running locally on the host with mapped ports and everything works fine.
When we attempt to run the containers as a service under swarm, the service starts up fine but the mapped ports are not exposed on the localhost.
We have attempted this with very simple services and gotten the same results (including the docker tutorial).
Example:
Commands:
docker swarm init
docker service create --name my-web --publish published=8080,target=80 --replicas 1 nginx
Both successful.
Result from docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
ze88zz31le31 my-web replicated 1/1 nginx:latest *:8080->80/tcp
Result attempting to attach to localhost:8080 where nginx should be accessible
curl localhost:8080
curl: (7) Failed to connect to localhost port 8080: Connection refused
Research on forums seems to indicate that this might be a potential missing dependency in our SLES.
We are not running SuSEfirewall2 and iptables appears to be populated correctly:
sudo iptables-save | grep DOCKER
: DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i docker_gwbridge -j RETURN
: DOCKER - [0:0]
: DOCKER-ISOLATION - [0:0]
: DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker_gwbridge -j DOCKER
-A DOCKER-ISOLATION -i docker_gwbridge -o docker0 -j DROP
-A DOCKER-ISOLATION -i docker0 -o docker_gwbridge -j DROP
-A DOCKER-ISOLATION -j RETURN
-A DOCKER-USER -j RETURN
I have tried enabling ipv4 forwarding as recommended in a forum, with no seeming effect.
The docker swarm management ports are available when swarm is running:
netstat -nlp | grep 7946
tcp 0 0 :::7946 :::* LISTEN 1828/dockerd
udp 0 0 :::7946 :::* 1828/dockerd
netstat -nlp | grep 4789
udp 0 0 0.0.0.0:4789 0.0.0.0:* -
I’m at a bit of a dead-end. I read one post that said that LVS / SLES HA Package may be a dependency but I found no corroboration of this.
We are using the Docker distribution installed from the Containers_Module_12_x86_64 package.
We have containers running locally on the host with mapped ports and everything works fine.
When we attempt to run the containers as a service under swarm, the service starts up fine but the mapped ports are not exposed on the localhost.
[…]
I’m at a bit of a dead-end. I read one post that said that LVS / SLES HA Package may be a dependency but I found no corroboration of this.
I’m not sure what tools you might need to add to your current setup, but many cluster and load balancer programs are indeed included in the SLES HAE add-on and your descriptions seem to point to problems creating the ipvs load-balancer setup. But on the other hand, I’d expect to see error messages in the Docker-related logs if such tools were not found while setting up the swarm.
Thanks for reference on networking. Here are more details (should have included in edited original post) of docker logs from /var/messages that show a failure in modifying iptables? despite dockerd clearly being able to update iptables.
Here is a snip of logs that contain an error related to iptables (note that docker appears able to modify iptables per additional info below):
2018-10-18T19:25:19.287893-04:00 dockerd[1828]: time=â2018-10-18T19:25:19.287650051-04:00â level=info msg=âNetworkDB stats - netID:lxy5sbaswro9qxqa0vt6uq7ob leaving:false netPeers:1 entries:4 Queue qLen:0 netMsg/s:0â
2018-10-18T19:30:19.487727-04:00 dockerd[1828]: time=â2018-10-18T19:30:19.487514638-04:00â level=info msg=âNetworkDB stats - netID:lxy5sbaswro9qxqa0vt6uq7ob leaving:false netPeers:1 entries:4 Queue qLen:0 netMsg/s:0â
2018-10-18T19:31:47.425687-04:00 dockerd[1828]: time=â2018-10-18T19:31:47-04:00â level=info msg=âSUSE:secrets :: enabledâ
2018-10-18T19:31:47.540558-04:00 dockerd[1828]: time=â2018-10-18T19:31:47-04:00â level=error msg=âsetting up rule failed, [-t mangle -D PREROUTING -p tcp --dport 8080 -j MARK --set-mark 262]: (iptables failed: iptables --wait -t mangle -D PREROUTING -p tcp --dport 8080 -j MARK --set-mark 262: iptables: No chain/target/match by that name.
2018-10-18T19:31:47.540979-04:00 dockerd[1828]: (exit status 1))â
2018-10-18T19:31:47.545054-04:00 dockerd[1828]: time=â2018-10-18T19:31:47.542668859-04:00â level=error msg=âFailed to delete firewall mark rule in sbox ingress (ingress): reexec failed: exit status 5â
2018-10-18T19:31:47.745356-04:00 kernel: [548403.162337] [2768(dockerd)]: gsch_umount_hook_fn(/var/run/docker/netns/cf5394f87151,2) doing
2
I have now gone through a restart of Docker and Swarm and watched the logs during startup. I found what appear to be the errors at the root of the problem - confirming that it looks like IPVS is required to run Swarm networking:
2018-10-19T12:23:29.447004-04:00 dockerd[21642]: time=“2018-10-19T12:23:29.446754417-04:00” level=warning msg=“Running modprobe ip_vs failed with message: modprobe: WARNING: Module ip_vs not found., error: exit status 1”
2018-10-19T12:23:29.449931-04:00 dockerd[21642]: time=“2018-10-19T12:23:29.449047777-04:00” level=error msg=“Could not get ipvs family information from the kernel. It is possible that ipvs is not enabled in your kernel. Native loadbalancing will not work until this is fixed.”
The HA package seems quite heavy-weight when ip_vs is the one missing dependency. Is there an alternative to installing the SLES HA package (or is it not as heavy as I perceive)?
The HA package seems quite heavy-weight when ip_vs is the one missing dependency. Is there an alternative to installing the SLES HA package (or is it not as heavy as I perceive)?
Actually, that’s more of a licensing question. Because technically, you can limit yourself to only install the according kernel (and its modules) plus whatever else might be needed for ipvs. OTOH, HAE is a non-free SLES add-on.