Buildin a 2-node active/standby cluster with SLES 11 SP3 HAE

SLES High Availability Extension
1

Hi,

I would like to create 2 VM guest to build a 2-node active/standby cluster with SLES 11 SP3 and High Availability.

The requirement for the cluster:
[INDENT]

  1. Two servers will use a virtual IP (ocf:IPaddr2).
  2. Each server will install tomcat and mysql. The tomcat will only connect to the mysql database in localhost.
  3. The tomcat and mysql will not be cluster resources since it may sometimes require to manually stop/start them.
  4. The server will create and save files to a data directory. The data directory should be always synchronized between manster and slave.
  5. The cluster fail over time should be fast (within 1 minute).
  6. The cluster will auto fail back if the master server is available again (the data directory and mysql database must be synchronized before fail back).
  7. The cluster environment should be as simple as possible.
  8. Easy backup and restore.
    [/INDENT]

I hava a doubt about which means of data synchornization (mainly for mysql database and file saving directory) is better for the system.

[LIST]
[]For mysql HA, is mysql replication better than drbd?
[]For file saving directory, is it suitable for using Cluster Clone Resources with SAN + iSCSI + OCSF2?
[/LIST]

Since I have no experience in building cluster before, there should be lots of issues that I haven’t been considered yet.

Could any one share his/her experience?
Thanks!

Best Regards,
Tony

2

Hi Tony,

[QUOTE=tonychao;25920]Hi,

I would like to create 2 VM guest to build a 2-node active/standby cluster with SLES 11 SP3 and High Availability.

The requirement for the cluster:[INDENT]

  1. Two servers will use a virtual IP (ocf:IPaddr2).
  2. Each server will install tomcat and mysql. The tomcat will only connect to the mysql database in localhost.
  3. The tomcat and mysql will not be cluster resources since it may sometimes require to manually stop/start them.
  4. The server will create and save files to a data directory. The data directory should be always synchronized between manster and slave.
  5. The cluster fail over time should be fast (within 1 minute).
  6. The cluster will auto fail back if the master server is available again (the data directory and mysql database must be synchronized before fail back).
  7. The cluster environment should be as simple as possible.
  8. Easy backup and restore.
    [/INDENT]

I hava a doubt about which means of data synchornization (mainly for mysql database and file saving directory) is better for the system.

[LIST]
[]For mysql HA, is mysql replication better than drbd?
[]For file saving directory, is it suitable for using Cluster Clone Resources with SAN + iSCSI + OCSF2?
[/LIST]

Since I have no experience in building cluster before, there should be lots of issues that I haven’t been considered yet.

Could any one share his/her experience?[/QUOTE]

concerning MySQL vs. DRBD, I don’t see how MySQL replication and DRBD compare at all. DRBD would create a “virtual block device”, that under complex conditions could be accessed from both nodes simultaneously. But running two MySQL daemons on the same set of database files will not work at all, so that’s of no use to you. MySQL replication will give you redundancy, but setting up an active/active MySQL cluster is another story.

What does “3.” mean precisely? Do you need to support running Tomcat/MySQL concurrently on both nodes? Looking at the other requirements, you’d no longer be talking about an “active/passive cluster”… Which would make things a bit more difficult to set up and run.

If it’s more about control (manual stop/start) rather than concurrently active systems, you might want to look into using SAN LUns for one or two OCFS2 clustered file systems.

MySQL would get started on a node once that node switched to “active” and stopped before dropping back to “passive”. If you include mount/umount of the file system in that sequence, you can better prepare against corruption by accidentially running to active MySQL instances on one set of data files.

You might be able to do similar things using DRBD instead of SAN LUNs, but from my experience this is more hassle than it’s worth, as you can provide “active/active” LUNs to your servers using the separate SAN resource.

Regards,
Jens

3

[QUOTE=jmozdzen;25934]Hi Tony,

concerning MySQL vs. DRBD, I don’t see how MySQL replication and DRBD compare at all. DRBD would create a “virtual block device”, that under complex conditions could be accessed from both nodes simultaneously. But running two MySQL daemons on the same set of database files will not work at all, so that’s of no use to you. MySQL replication will give you redundancy, but setting up an active/active MySQL cluster is another story.

What does “3.” mean precisely? Do you need to support running Tomcat/MySQL concurrently on both nodes? Looking at the other requirements, you’d no longer be talking about an “active/passive cluster”… Which would make things a bit more difficult to set up and run.

If it’s more about control (manual stop/start) rather than concurrently active systems, you might want to look into using SAN LUns for one or two OCFS2 clustered file systems.

MySQL would get started on a node once that node switched to “active” and stopped before dropping back to “passive”. If you include mount/umount of the file system in that sequence, you can better prepare against corruption by accidentially running to active MySQL instances on one set of data files.

You might be able to do similar things using DRBD instead of SAN LUNs, but from my experience this is more hassle than it’s worth, as you can provide “active/active” LUNs to your servers using the separate SAN resource.

Regards,
Jens[/QUOTE]

Hi Jens,

Thanks very much for your reply.

According to MySQL official document http://dev.mysql.com/doc/mysql-ha-scalability/en/ha-overview.html, MySQL Replication and DRBD are both MySQL HA Solutions.

In my HA environment, only one site will serve public request at the same time. What my mainly concerns are data integrity and the time of fail over.

If using the DRDB solution, I am afraid it may take a very long time for crash recovery happens on the passive server.
So I prefer to use MySQL Replication (using master/master mode for auto fail back).

For files data synchronization, is using SAN LUNs with OCFS2 a common solution?

On the other hand, is it able to automatically confirm that both the files data and MySQL are synchronized before fail back?

Regards,
Tony

4

Hi Tony,

I misinterpreted your original question - you’re comparing “MySQL with built-in replication” and “MySQL fail-over with DRDB-based data replication”, which of course is valid… to the point that MySQL+DRBD doesn’t match your stated requirements: From the docs you referenced: “[DRBD] acts in an active-standby mode: at any point in time, the directories being managed by DRBD are accessible for reads and writes on exactly one of the two hosts and inaccessible (even for reads) on the other.”. Hence you can only have a single active MySQL node at any point in time - versus your requirement “3. The tomcat and mysql will not be cluster resources since it may sometimes require to manually stop/start them.”, in case that was meant as “concurrently active on both nodes”.

[QUOTE=tonychao;25966]In my HA environment, only one site will serve public request at the same time. What my mainly concerns are data integrity and the time of fail over.

If using the DRDB solution, I am afraid it may take a very long time for crash recovery happens on the passive server.
So I prefer to use MySQL Replication (using master/master mode for auto fail back).[/QUOTE]
Yes, down-time can become a concern if using MySQL on DRBD protecting against node failure. The shared file system is another SPOF - although it’s mirrored, it may get corrupted, bringing down both MySQL nodes at the same time (if MySQL could handle working on a clustered FS).

Do you really mean “master/master” mode for replication? I’m only aware of a “master/slaves” mode. If you need master/master, I believe you’d have to look into “MySQL Cluster”.

Yes, definitely.

Nope, neither this==“OCFS2” nor this==“Pacemaker” will provide that functionality. Personally, I believe that in a (symmetric) cluster, resources can be anywhere and a “switch back after recovery” is just an additional source of potential trouble.

Regards,
Jens

5

Hi Jens,

MySQL Replication could be set to master/master. here is an example http://www.lefred.be/node/45.

Thanks with your explanation. Now I can decide to use SAN LUNs with OCFS2 for file data and MySQL Replication for MySQL.

I would like to ask 2 questions more:

[LIST=1]
[]Before I stop the Tomcat or MySQL manually in the primary node (or set it to maintenance mode), I would like to confirm that there is no connection for primary node any more. Is it possible to keep serving the current connections until they are finished, but the new connections will be served in the secondary node? It seems that the IPaddr2 does not have such feature.
[]Could I disable fencing in this architecture?
[/LIST]

Regards,
Tony

6

Hi Tony,

[QUOTE=tonychao;25983]Hi Jens,

MySQL Replication could be set to master/master. here is an example http://www.lefred.be/node/45.[/QUOTE]

that’s definitely interesting to read, but I must admit, to me it sounds & looks like providing a rather instable solution to a problem asking for MySQL Cluster. Setting that up is more work, but targets a multi-master scenario by design (i.e. synchronous updates).

[QUOTE=tonychao;25983]Thanks with your explanation. Now I can decide to use SAN LUNs with OCFS2 for file data and MySQL Replication for MySQL.

I would like to ask 2 questions more:

[LIST=1]
[*]Before I stop the Tomcat or MySQL manually in the primary node (or set it to maintenance mode), I would like to confirm that there is no connection for primary node any more. Is it possible to keep serving the current connections until they are finished, but the new connections will be served in the secondary node? It seems that the IPaddr2 does not have such feature.
[/LIST]
[/QUOTE]When I look at it from the technical point of view - how should IPaddr2 be able to do so? The clients are addressing the cluster IP address, which will always be active on a single node only… else the clients/routers wouldn’t know where to send the packets to.

You can disable fencing in every architecture :wink: but why would you want to do so? Fencing is a way to make sure no resource is active on a failing node, so it can be safely restarted on one of the remaining nodes. You’ll only need to provide this feature when you’re running active/passive resources. If, on the other hand, you’re running only active/active resources, there’s no need for Pacemaker, as you can simply start the services on all nodes and use local monitoring to keep them active. So IMO, you either use Pacemaker (to run a/p resources) and you should make use of fencing - or you run no Pacemaker at all (when having only a/a resources) and fencing (in the Pacemaker sense) won’t be part of the picture, either.

Regards,
Jens