Does the Swarm environment work? I’m unable to keep the swarmkit-mon containers in a stable state which causes the environment to always being an an setup state, no CLI. I’ve tested it with different versions of Docker and Rancher and all have failed with the same issue. Trying to figure out what I’m over looking here. I’ve rebuilt this setup multiple times with VMs and each time it ends with the same results. The initial setup will sometimes come up stable, but reboot any of the client VMs and this swarmkit-mon will start to fail on all machines. I understand the Swarm support is now listed as experimental. I didn’t experience this issue with the Swarm environment when I initially tested Rancher six months ago.
Nothing out of the normal with the setup of the VMs. Five VMs in VirtualBox on the same host. One MySQL DB, one Rancher server 1.3.4, and three clients
All three clients setup the same - Centos 7
Docker version
Client:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built:
OS/Arch: linux/amd64
Server:
Version: 1.12.3
API version: 1.24
Go version: go1.6.3
Git commit: 6b644ec
Built:
OS/Arch: linux/amd64
Firewalls (IPtables, Firewalld) disabled.
SELinux disabled
Rancher server 1.3.4 currently. I’ve tested it with every version from 1.3.0 all the way to 1.4.0, same results.
Logs from swarmkit-mon-1
2/9/2017 12:12:25 PMtime="2017-02-09T19:12:25Z" level=info msg="Listening on port: 2378"
2/9/2017 1:43:30 PMtime="2017-02-09T20:43:30Z" level=info msg="Listening on port: 2378"
Logs from swarmkit-mon-3
2/9/2017 1:40:05 PMDeleted host label swarm
2/9/2017 1:40:35 PMDeleted host label swarm
2/9/2017 1:41:06 PMDeleted host label swarm
2/9/2017 1:41:36 PMDeleted host label swarm
2/9/2017 1:42:06 PMDeleted host label swarm
2/9/2017 1:42:36 PMDeleted host label swarm
2/9/2017 1:43:06 PMDeleted host label swarm
2/9/2017 1:43:32 PMtime="2017-02-09T20:43:32Z" level=info msg="Listening on port: 2378"
2/9/2017 1:43:33 PMDeleted host label swarm
Logs from swarmkit-mon-2
/9/2017 2:09:05 PMNo active workers present for promotion, add more nodes to enable reconciliation.
2/9/2017 2:09:35 PMError response from daemon: rpc error: code = 9 desc = attempting to demote the last manager of the swarm
2/9/2017 2:09:35 PMError response from daemon: rpc error: code = 9 desc = node 4w8j9psi970rfqs7ag3uargxh is a cluster manager and is a member of the raft cluster. It must be demoted to worker before removal
2/9/2017 2:09:35 PMRemoved 4w8j9psi970rfqs7ag3uargxh from the swarm.
2/9/2017 2:09:35 PM1 of 1 manager(s) reachable, 0 worker(s) active
2/9/2017 2:09:35 PMNo active workers present for promotion, add more nodes to enable reconciliation.
2/9/2017 2:10:05 PMError response from daemon: rpc error: code = 9 desc = attempting to demote the last manager of the swarm
2/9/2017 2:10:05 PMError response from daemon: rpc error: code = 9 desc = node 4w8j9psi970rfqs7ag3uargxh is a cluster manager and is a member of the raft cluster. It must be demoted to worker before removal
2/9/2017 2:10:05 PMRemoved 4w8j9psi970rfqs7ag3uargxh from the swarm.
2/9/2017 2:10:05 PM1 of 1 manager(s) reachable, 0 worker(s) active
2/9/2017 2:10:05 PMNo active workers present for promotion, add more nodes to enable reconciliation.
2/9/2017 2:10:35 PMError response from daemon: rpc error: code = 9 desc = attempting to demote the last manager of the swarm
2/9/2017 2:10:35 PMError response from daemon: rpc error: code = 9 desc = node 4w8j9psi970rfqs7ag3uargxh is a cluster manager and is a member of the raft cluster. It must be demoted to worker before removal
2/9/2017 2:10:35 PMRemoved 4w8j9psi970rfqs7ag3uargxh from the swarm.
2/9/2017 2:10:35 PM1 of 1 manager(s) reachable, 0 worker(s) active
2/9/2017 2:10:35 PMNo active workers present for promotion, add more nodes to enable reconciliation.
2/9/2017 2:11:06 PMError response from daemon: rpc error: code = 9 desc = attempting to demote the last manager of the swarm
2/9/2017 2:11:06 PMError response from daemon: rpc error: code = 9 desc = node 4w8j9psi970rfqs7ag3uargxh is a cluster manager and is a member of the raft cluster. It must be demoted to worker before removal
2/9/2017 2:11:06 PMRemoved 4w8j9psi970rfqs7ag3uargxh from the swarm.
2/9/2017 2:11:06 PM1 of 1 manager(s) reachable, 0 worker(s) active
2/9/2017 2:11:06 PMNo active workers present for promotion, add more nodes to enable reconciliation.
This is found in the logs for the network-services-metadata containers on the failing hosts
2/9/2017 2:28:39 PMtime="2017-02-09T21:28:39Z" level=info msg="Error: /self/host/labels/swarm" client=172.17.0.1 version=2015-12-19
2/9/2017 2:29:09 PMtime="2017-02-09T21:29:09Z" level=info msg="Error: /self/host/labels/swarm" client=172.17.0.1 version=2015-12-19
2/9/2017 2:29:39 PMtime="2017-02-09T21:29:39Z" level=info msg="Error: /self/host/labels/swarm" client=172.17.0.1 version=2015-12-19
2/9/2017 2:30:09 PMtime="2017-02-09T21:30:09Z" level=info msg="Error: /self/host/labels/swarm" client=172.17.0.1 version=2015-12-19
2/9/2017 2:30:39 PMtime="2017-02-09T21:30:39Z" level=info msg="Error: /self/host/labels/swarm" client=172.17.0.1 version=2015-12-19