Every HA architecture has common problems to solve or common questions to respond to:
- How the connection can be handled
- How the failover can be managed
- How the storage is shared or replicated to other sites
There are some production-stable and widely used solutions for each one of those questions. Let's study these questions in detail:
- How the connection can be handled
One of the possible answers to this question is just one word—VIP (Virtual IP). Basically, every software component needs to communicate or is interconnected with different logical layers, and those components are often deployed on different servers to divide and equalize the workload. Much of the communication is TCP/IP-based, and here the network protocol gives us a hand.
It is possible to define a VIP that is assigned to the active servers and all the software required to be configured to use that address. So if there is a failover, the IP address will follow the service and all the clients will continue to work. Of course, this solution can't guarantee that there isn't downtime at all, but it will be limited by time and will be for a short period of time. From an administration point of view, apart from checking the failover, the administrator doesn't need to reconfigure anything.
- How the failover can be managed
The answer to this question is: use a resource manager. You need to think of a smart way to move a faulty service to the standby node that is independent of SLA as soon as possible. To achieve the minimum downtime possible, you need to automate the service failover on the standby node and give the business continuity. The fault needs to be found as soon as possible when it happens.
- How the storage is shared or replicated to the other site
This last question can be implemented with different actors, technologies, and methodologies. You can use a shared disk, a replicated Logical Unit Number (LUN) between two storages, or a replicated device with software. Unfortunately, using a replicated LUN between two storages is quite expensive. This software should be closer to the kernel and should be working on the lowest layer possible to be transparent from the operating system's perspective, thereby keeping things easy to manage.
The architecture that you are going to implement needs a component to automate switchover or failover; basically, as said earlier, it requires a resource manager.
One of the resource managers that is widely used and is production mature is Pacemaker. Pacemaker is an open source, high-availability resource manager designed for small and large clusters. Pacemaker is available for download at http://clusterlabs.org/.
Pacemaker provides interesting features that are really useful for your cluster, such as:
- Detecting and recovering server issues at the application level
- Supporting redundant configurations
- Supporting multiple node applications
- Supporting startup/shutdown ordering applications
Practically, Pacemaker replaces you and is automated and fast. Pacemaker does the work that a Unix administrator normally does in the event of a node failure. It checks whether the service is no more available and switches all the configured services on the spare node; plus, it does all this work as quickly as possible. This switchover gives us the time required to do all the forensic analysis while all the services are still available. In another context, the service would be simply unavailable.
There are different solutions that provide cluster management. Red Hat Cluster Suite is a popular alternative. It is not proposed here as it is not really completely tied to Red Hat; however, it is definitely developed with this distribution in mind.
Distributed Replicated Block Device (DRBD) has some features that are the defining points of this solution:
The core functionality of DRBD is implemented on the kernel layer; in particular, DRBD is a driver for a virtual block device, so DRBD works at the bottom of the system I/O stack.
DRBD can be considered equivalent to a networked RAID 1 below the OS's filesystem, at the block level.
This means that DRBD synchronization is synced to the filesystem. The worst-case scenario and more complex to handle is a filesystem replication for a database. In this case, every commit needs to be acknowledged on both nodes before it happens, and all the committed transactions are written on both nodes; DRBD completely supports this case.
Now, what happens when a node is no longer available? It's simple; DRBD will operate exactly as a degraded RAID 1 would. This is a strong point because, if your Disaster Recovery site goes down, you don't need to do anything. Once the node reappears, DRBD will do all the synchronization work for us, that is, rebuilding and resynchronizing the offline node.