the best practice is to have an odd number of nodes in order to avoid split brain. With even number of nodes the risk is very high.
Of course there are ways to mitigate splitbrain ,but depends on multiple factors:
- You can use a simple 1 vCPU 1 GB ram Virtual machine and this is enough to have an odd number of nodes.
- You can define how a quorum is being calculated. For example in a 4 node cluster you can combine the following options in order to define which pair of nodes will have quorum:
- auto_tie_breaker: 1
- auto_tie_breaker_node: lowest|highest|
In such case , if you have 2 nodes in DC - A and 2 nodes in DC -B , the nodes selected by the “auto_tie_breaker_node” will have a quorum and take all resources, while the nodes without that node will not have quorum.
- For your custom service I can recommend you that:
- the service should be under the control of the cluster. You can create your own pacemaker resource , but you should not activate the service by any other way.
- a service is any script that allows the cluster to “start”, “stop” and query “status”
- Any configuration should be on a shared storage (no matter FC, iSCSI , NFS, etc) in order to avoid differences in the service configuration
The simplest method for creating such service is to use systemd.service that is not enabled on any node ,as pacemaker has a built-in method to manage systemd services.
With the different constrains (order of start/stop, collocation - Service B runs where Service A is running, location - Service A must always run on node3),you have a lot of capabilities and a detailed plan is needed in advance.
Also if you have CPU & RAM requirements for your service , pacemaker can take care of that , too - which is especially useful when you have nodes with different size.