Business continuity and continuous application availability are among the most important requirements for many organizations. Advances in virtualization, storage, and networking made enhanced business continuity possible. Information technology solutions can now manage planned and unplanned outages, and provide the flexibility and cost efficiencies that are available from cloud-computing models.

This chapter briefly describes the Stretched Cluster, Enhanced Stretched Cluster, and HyperSwap solutions for IBM Spectrum Virtualize systems. Technical details or implementation guidelines are not presented in this chapter because they are described in separate publications.

This chapter includes the following topics:

IBM SAN Volume Controller systems support three different cluster topologies: Standard, Stretched, and HyperSwap. In this chapter, we describe the Stretched and HyperSwap topologies that provide high availability (HA) functions.

Within standard implementations of IBM Spectrum Virtualize, all controller nodes are physically installed in the same location. To fulfill the different HA needs of customers, the Stretched Cluster configuration was introduced, in which each node from the same I/O Group is physically installed at a different site.

When implemented, this configuration can be used to maintain access to data on the system, even if failures occur at different levels, such as the SAN, back-end storage, IBM Spectrum Virtualize controller, or data center power.

Stretched Cluster is considered a HA solution because both sites work as instances of the production environment (no standby location is used). Combined with application and infrastructure layers of redundancy, Stretched Clusters can provide enough protection for data that requires availability and resiliency.

When IBM Spectrum Virtualize was first introduced, the maximum supported distance between nodes within an I/O Group was 100 meters (328 feet). With the evolution of code and the introduction of new features, stretched cluster configurations were enhanced to support distances up to 300 km (186.4 miles). These geographically dispersed solutions use specific configurations that use Fibre Channel (FC) or Fibre Channel over IP (FC/IP) switch, or Multiprotocol Router (MPR) inter-switch links (ISLs) between different locations.

IBM Spectrum Virtualize V7.2 introduced the Enhanced Stretched Cluster feature that further improved the Stretched Cluster configurations. The Enhanced Stretched Cluster introduced the site awareness concept for nodes and external storage, and the Disaster Recovery (DR) feature that enables you to effectively manage rolling disaster scenarios.

With IBM Spectrum Virtualize V7.5, the site awareness concept was extended to hosts. This extension enables more efficiency for host I/O traffic through the SAN, and easier host path management.

Stretched Cluster and Enhanced Stretched Cluster solutions can also be combined with other replication techniques, such as Metro Mirror or Global Mirror, which allows a three-site configuration for HA and DR purposes.

In a stretched system configuration, each site is defined as an independent failure domain. If one site experiences a failure, the other site can continue to operate without disruption.

You also must configure a third site to host a quorum device that provides an automatic tie-break if a link failure occurs between the two main sites (for more information, see 7.2, “Third site and IP quorum” on page 346). The main site can be in the same room or across rooms in the data center, buildings on the same campus, or buildings in different cities. Different types of sites protect against different types of failures.

If each site is a different power phase within a single location or data center, the system can survive the failure of any single power domain. For example, one node can be placed in one rack installation and the other node can be in another rack. Each rack is considered a separate site with its own power phase. In this case, if power was lost to one of the racks, the partner node in the other rack can be configured to process requests and effectively provide availability to data, even when the other node is offline because of a power disruption.

If each site is a different physical location, the system can survive the failure of any single location. These sites can span shorter distances (for example, two sites in the same city), or they can be spread farther geographically, such as two sites in separate cities. If one site experiences a site-wide disaster, the other site can remain available to process requests.

If configured correctly, the system continues to operate after the loss of one site. The key prerequisite is that each site contains only one node from each I/O group. However, placing one node from each I/O group in different sites for a stretched system configuration does not provide HA. You must also configure the suitable mirroring technology and ensure that all configuration requirements for those technologies are correctly configured.

The HyperSwap HA feature that is available in the IBM Spectrum Virtualize and FlashSystems products enables business continuity during a hardware failure, power outage, connectivity problem, or other disasters, such as fire or flooding.

It provides highly available volumes that are accessible through two sites that are up to 300 km (186.4 miles) apart. A fully independent copy of the data is maintained at each site.

When data is written by hosts at either site, both copies are synchronously updated before the write operation is completed. HyperSwap automatically optimizes itself to minimize data that is transmitted between sites, and to minimize host read and write latency. For more information about the optimization algorithm, see 7.3, “HyperSwap volumes” on page 348.

HyperSwap includes the following key features:

The IBM Spectrum Virtualize HyperSwap configuration requires that at least one control enclosure is implemented in each location. Therefore, a minimum of two control enclosures for each cluster are needed to implement HyperSwap. Configuration with three or four control enclosures is also supported for the HyperSwap.

In stretched cluster or HyperSwap configurations, you can use a third, independent site to house a quorum device to act as the tie-breaker in case of split-brain scenarios. The quorum device can also hold a backup copy of the cluster metadata to be used in specific situations that might require a full cluster recovery.

To use a quorum disk as the quorum device, this third site must have FC or iSCSI connectivity between an external storage system and the IBM Spectrum Virtualize cluster. Sometimes, this third site quorum disk requirement can be expensive in terms of infrastructure and network costs. For this reason, a less demanding solution that is based on a Java application was introduced with the release V7.6, and is known as IP quorum application.

Initially, IP quorum was used only as a tie-breaker solution. However, it was expanded to store cluster configuration metadata with the release V8.2.1, fully serving as an alternative for quorum disk devices. To use an IP quorum application as the quorum device for the third site, no FC connectivity is used. It can be run on any host at the third site, as shown in Figure 7-1.

However, the following strict requirements on the IP network must be met when it is used:

Quorum devices are also required at the sites 1 and 2, and can be disk-based quorum devices or IP quorum applications. The maximum number of IP quorum applications that can be deployed is five.

For more information about IP Quorum requirements and installation, including supported Operating Systems and Java runtime environments (JREs), see this IBM Documentation web page.

For more information about quorum disk devices, see 3.3, “Quorum disks” on page 96.

With the release of IBM Spectrum Virtualize V8.3, a new configuration option was added to the IP Quorum functions, called quorum mode. By default, the IP quorum mode is set to Standard. In Stretched or HyperSwap clusters, this mode can be changed to Preferred or Winner.

This configuration allows you to specify which site resumes I/O after a disruption, based on the applications that run on each site or other factors. For example, you can specify whether a selected site is the preferred for resuming I/O, or if the site automatically “wins” in tie-break scenarios.

If only one site runs critical applications, you can configure this site as Preferred. During a split-brain situation, the system delays processing tie-break operations on other sites that are not specified as “preferred”. That is, the designated preferred site has a timed advantage when a split-brain situation is detected, and starts racing for the quorum device a few seconds before the non-preferred sites. Therefore, the likelihood of reaching the quorum device first is higher. If the preferred site is damaged or cannot reach the quorum device, the other sites can attempt to win the tie-break and continue I/O.

This configuration is recommended for use when no third site is available for a quorum device to be installed. In this case, when a split-brain situation is detected, the site that is configured as the winner always is the site to continue processing I/O, regardless of the failure and its condition. The nodes at the non-winner site always lose the tie-break and stop processing I/O requests until the fault is fixed.

HyperSwap volumes consist of a master volume and a master change volume (CV) in one site, and an auxiliary volume and an auxiliary CV in the other system site. An active-active synchronous mirroring relationship exists between the two sites. As with a regular Metro Mirror relationship, the active-active relationship keeps the master Volume and auxiliary volume synchronized.

The relationship uses the CVs as journaling volumes during any resynchronization process. The master CV must be in the same I/O Group as the master volume. It also is recommended that it is in the same pool as the master volume. A similar practice applies to the auxiliary CV and the auxiliary volume. For more information about the change volume, see “Global Mirror functional overview” on page 267.

The HyperSwap volume always uses the unique identifier (UID) of the master volume. The HyperSwap volume is assigned to the host by mapping only the master volume, even though access to the auxiliary volume is ensured by the HyperSwap function.

Figure 7-2 shows how the HyperSwap volume is implemented.

The active-active synchronous replication workload traverses the SAN by using the node-to-node communication. Master and auxiliary volumes also have a specific role of Primary or Secondary, which is based on the Metro Mirror active-active relationship direction.

Starting with the IBM Spectrum Virtualize 8.3.1 code level, reads are always done in the local copy of the volume. Write operations are always routed to the Primary copy. Therefore, hosts that access the Secondary copy for writes might experience an increased latency in the I/O operations. As a mitigation of this behavior, if sustained workload (that is, more than 75% of I/O operations for at least 20 minutes) is running over Secondary volumes, the HyperSwap function switches the direction of the active-active relationships, which swaps the Secondary volume to Primary and vice versa.

The business continuity solutions that are described in this section feature different characteristics in terms of implementation and features. Table 7-1 provides a comparison of these business continuity solutions that can help to identify the most fitting solution to a specific environment and needs.

A business continuity solution implementation requires special considerations in the infrastructure and network setup. In Stretched and HyperSwap topologies, the communication between the IBM Spectrum Virtualize controllers must be optimal and free of errors for best performance, as the internode messaging and cache mirroring is done across the sites. Have a dedicated private SAN for internode communication so that it is not affected by regular SAN activities.

An important recommendation is to review the site attribute of all the components to ensure they are accurate. With the site awareness algorithm present in the IBM Spectrum Virtualize code, optimizations are done to reduce the cross-site workload. If this attribute is missing or not accurate, increased unnecessary cross-site traffic might occur, which can lead to higher response time to the applications.

To further increase the system’s resiliency, plan to have at least one hot-spare node per site. This configuration helps reduce the time that an I/O group remains with a single controller node when a node fails, or during planned actions, such as system code upgrades or hardware maintenance.

For more information, see the following resources:

For detailed step-by-step configurations, see the following IBM Documentation web pages:

The HyperSwap feature requires implementing the storage network (SAN) to ensure that the inter-node communication on the Fibre Channel ports on the control enclosures between the sites is on dedicated fabrics. No other traffic (hosts, back-end controllers, if any) or traffic that is unrelated to the HyperSwapped cluster can be allowed on this fabric. Two fabrics should be used: one that is private for the inter-node communication, and one that is public for all other data.

A few SAN designs can achieve this separation, and some potential problems can occur with incorrect SAN design and implementation.

For more information about the design options and some common problems, see IBM Spectrum Virtualize HyperSwap SAN Implementation and Design Best Practices, REDP-5597, and this related IBM Support video about designing a SAN for HyperSwap Spectrum Virtualize clusters.

For more information about a detailed step-by-step configuration, see this IBM Documentation web page.