Trouble connecting to additional nodes when creating HA Rancher cluster

Rancher 1.x
1

I am trying to set up a Rancher HA cluster. For this purpose i set up a MySQL server on an ubuntu core 16.04 and have set it up as follows using the Rancher documentation.

CREATE DATABASE IF NOT EXISTS cattle COLLATE = 'utf8_general_ci' CHARACTER SET = 'utf8';
GRANT ALL ON cattle.* TO 'cattle'@'%' IDENTIFIED BY 'SomePassword123-';
GRANT ALL ON cattle.* TO 'cattle'@'localhost' IDENTIFIED BY 'SomePassword123-';

I have then on my first docker node started the first instance of Rancher.

sudo docker run -d --restart=unless-stopped -p 8081:8080 -p 9345:9345 rancher/server --db-host 192.168.1.145 --db-port 3306 --db-user cattle --db-pass SomePassword123- --db-name cattle --advertise-address 192.168.1.77 --advertise-http-port 8081

And on the second docker node.

sudo docker run -d --restart=unless-stopped -p 8081:8080 -p 9345:9345 rancher/server --db-host 192.168.1.145 --db-port 3306 --db-user cattle --db-pass SomePassword123- --db-name cattle --advertise-address 192.168.1.78 --advertise-http-port 8081

Now, i can access the first instance just fine, and if i go under Admin–>High Availability, i can see the second node.
But i can not access the second node on its HTTP port 192.168.1.78:8081

If i look in the logs for the first node i see this when i try running a second one.

W2017-06-06T09:17:20.626254660Z 2017-06-06 09:17:20,625 ERROR [:] [] [] [] [ration.thread-0] [c.h.s.i.o.impl.OperationRunnerImpl  ] [192.168.1.77]:9345 [8d558832-8c50-45db-98f7-c378fa99bd3c] [3.7.1] com.hazelcast.internal.cluster.impl.ClusterDataSerializerHook$ClusterDataSerializerFactoryImpl@72304d30 is not be able to create an instance for id: 18 on factoryId: 0 com.hazelcast.nio.serialization.HazelcastSerializationException: com.hazelcast.internal.cluster.impl.ClusterDataSerializerHook$ClusterDataSerializerFactoryImpl@72304d30 is not be able to create an instance for id: 18 on factoryId: 0
2017-06-06T09:17:20.626312987Z 	at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:119) ~[hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626327124Z 	at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:52) ~[hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626337783Z 	at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:46) ~[hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626348135Z 	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:172) ~[hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626358794Z 	at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:307) ~[hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626369172Z 	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:381) ~[hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626379524Z 	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:117) [hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626390086Z 	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:102) [hazelcast-3.7.1.jar:3.7.1]
 2017-06-06T09:17:20.626400372Z 
2017-06-06T09:17:20.626636935Z 2017-06-06 09:17:20,625 ERROR [:] [] [] [] [ration.thread-0] [c.h.s.i.o.impl.OperationExecutorImpl] [192.168.1.77]:9345 [8d558832-8c50-45db-98f7-c378fa99bd3c] [3.7.1] Failed to process packet: Packet{flags=17, isResponse=false, isOperation=true, isEvent=false, partitionId=-1, conn=Connection[id=22, /172.17.0.9:9345->/192.168.1.79:45977, endpoint=[192.168.1.79]:9345, alive=true, type=MEMBER]} on hz._hzInstance_1_8d558832-8c50-45db-98f7-c378fa99bd3c.generic-operation.thread-0 com.hazelcast.nio.serialization.HazelcastSerializationException: com.hazelcast.internal.cluster.impl.ClusterDataSerializerHook$ClusterDataSerializerFactoryImpl@72304d30 is not be able to create an instance for id: 18 on factoryId: 0
2017-06-06T09:17:20.626690471Z 	at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:119) ~[hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626704360Z 	at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:52) ~[hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626715273Z 	at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:46) ~[hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626738333Z 	at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:172) ~[hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626748675Z 	at com.hazelcast.spi.impl.NodeEngineImpl.toObject(NodeEngineImpl.java:307) ~[hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626757101Z 	at com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl.run(OperationRunnerImpl.java:381) ~[hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626770093Z 	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.process(OperationThread.java:117) [hazelcast-3.7.1.jar:3.7.1]
2017-06-06T09:17:20.626778716Z 	at com.hazelcast.spi.impl.operationexecutor.impl.OperationThread.run(OperationThread.java:102) [hazelcast-3.7.1.jar:3.7.1]
 2017-06-06T09:17:20.626788451Z

On the secondary node(s) i get this error repeated.

-2017-06-06T09:10:28.665389662Z 09:10:28.664 [hz._hzInstance_1_8d558832-8c50-45db-98f7-c378fa99bd3c.response] WARN  c.h.s.i.o.i.InboundResponseHandler - [192.168.1.78]:9345 [8d558832-8c50-45db-98f7-c378fa99bd3c] [3.8] No Invocation found for error response with callId: 0 sent from [192.168.1.77]:9345
-2017-06-06T09:10:28.665389662Z 09:10:28.664 [hz._hzInstance_1_8d558832-8c50-45db-98f7-c378fa99bd3c.response] WARN  c.h.s.i.o.i.InboundResponseHandler - [192.168.1.78]:9345 [8d558832-8c50-45db-98f7-c378fa99bd3c] [3.8] No Invocation found for error response with callId: 0 sent from [192.168.1.77]:9345
-2017-06-06T09:10:28.665389662Z 09:10:28.664 [hz._hzInstance_1_8d558832-8c50-45db-98f7-c378fa99bd3c.response] WARN  c.h.s.i.o.i.InboundResponseHandler - [192.168.1.78]:9345 [8d558832-8c50-45db-98f7-c378fa99bd3c] [3.8] No Invocation found for error response with callId: 0 sent from [192.168.1.77]:9345

Can anyone point me in the right direction. I feel like i followed the official guide very closely.

2

I didn’t fix the problem yet but i got some info. Im running my servers on an esxi cluster controlled by a vcenter server locally. It’s my testing environment. When i tried running the rancher containers on a server in the live environment it worked fine. So perhaps it’s a problem with multicast on my testing environments network?

3

I am running an HA Rancher cluster on ESXI (managed by vCenter) and it works for me. I am using the default port of 8080, so maybe try using the default port and see if that works.