High availability (HA) pairs and failover behavior
High availability (HA) pairs provide hardware redundancy that is required for non-disruptive operations and fault tolerance. They give each node in the pair the software functionality to take over its partner's storage and subsequently give back the storage.
Takeover and giveback are the operations that let you take advantage of the HA configuration to perform non-disruptive operations and avoid service interruptions. Takeover is the process in which a node takes over the storage of its partner. Giveback is the process in which the storage is returned to the partner. You can initiate the processes in different ways.
The following topics are covered:
5.1 What an HA pair is
An HA pair consists of two storage systems (nodes) whose controllers are connected to each other directly. In this configuration, one node can take over its partner's storage to provide continued data service if the partner goes down.
You can configure the HA pair so that each node in the pair shares access to a common set of storage, subnets, and tape drives, or each node can own its own distinct set of storage.
The controllers are connected to each other through an HA interconnect. This allows one node to serve data that resides on the disks of its failed partner node. Each node continually monitors its partner, mirroring the data for each other’s nonvolatile memory (NVRAM or NVMEM). The interconnect is internal and requires no external cabling if both controllers are in the same chassis.
Figure 5-1 shows an HA pair in the cluster.
Figure 5-1 HA pair with dedicated HA interconnect and cluster interconnect
5.2 Preferred practices for HA pairs
Here we list some preferred practices for working with HA pairs:
•To ensure that your HA pair is robust and operational, you need to be familiar with configuration preferred practices.
•Do not use the root aggregate for storing data.
•Do not create new volumes on a node when takeover, giveback, or aggregate relocation operations are in progress or pending.
•Make sure that each power supply unit in the storage system is on a different power grid so that a single power outage does not affect all power supply units.
•Use the logical interface (LIF) with defined failover policies to provide redundancy and improve availability of network communication.
•Maintain consistent configuration between the two nodes.
An inconsistent configuration is often the cause of failover problems.
•Test the failover capability routinely (for example, during planned maintenance) to ensure proper configuration.
•Make sure that each node has sufficient resources to adequately support the workload of both nodes during takeover mode.
•Use the Config Advisor tool to help ensure that failovers are successful.
•If your system supports remote management (through an RLM or Service Processor), make sure that you configure it properly.
•Follow advised limits for FlexVol volumes, dense volumes, Snapshot copies, and LUNs to reduce the takeover or giveback time.
When adding traditional or FlexVol volumes to an HA pair, consider testing the takeover and giveback times to ensure that they fall within your requirements.
•Multipath HA is required on all HA pairs.
•Avoid using the -only-cfo-aggregates parameter with the storage failover giveback command.
5.3 Cluster consisting of a single HA pair
Cluster high availability (HA) is activated automatically when you enable storage failover on clusters that consist of two nodes, and you should be aware that automatic giveback is enabled by default. On clusters that consist of more than two nodes, automatic giveback is disabled by default, and cluster HA is disabled automatically.
A cluster with only two nodes presents unique challenges in maintaining a quorum, the state in which a majority of nodes in the cluster have good connectivity. In a two-node cluster, neither node holds epsilon, the value that designates one of the nodes as the master. Epsilon is required in clusters with more than two nodes. Instead, both nodes are polled continuously to ensure that if takeover occurs, the node that is still up and running has full read-write access to data as well as access to logical interfaces and management functions. This continuous polling function is referred to as cluster high availability (HA).
Cluster HA is different than and separate from the high availability provided by HA pairs and the storage failover commands. While crucial to full functional operation of the cluster after a failover, cluster HA does not provide the failover capability of the storage failover functionality.
|
Note: If you have a two-node switchless configuration that uses direct-cable connections between the nodes instead of a cluster interconnect switch, you must ensure that the switchless-cluster-network option is enabled. This ensures proper cluster communication between the nodes.
|
Steps to enable cluster HA and switchless-cluster in a two-node cluster:
1. Enter the following command to enable cluster HA:
cluster ha modify -configured true
If storage failover is not already enabled, you will be prompted to confirm enabling of both storage failover and auto-giveback.
2. If you have a two-node switchless cluster, enter the following commands to verify that the switchless-cluster option is set:
a. Enter the following command to change to the advanced-privilege level:
set -privilege advanced
Confirm when prompted to continue into advanced mode. The advanced mode prompt appears (*>).
b. Enter the following command:
network options switchless-cluster show
If the output shows that the value is false, you must issue the following command:
network options switchless-cluster modify true
c. Enter the following command to return to the admin privilege level:
set -privilege admin
5.4 Non-disruptive operations and HA pairs support
HA pairs provide fault tolerance and let you perform non-disruptive operations, including hardware and software upgrades, relocation of aggregate ownership, and hardware maintenance:
•Fault tolerance:
When one node fails or becomes impaired and a takeover occurs, the partner node continues to serve the failed node’s data.
•Non-disruptive software upgrades or hardware maintenance:
During hardware maintenance or upgrades, when you halt one node and a takeover occurs (automatically, unless you specify otherwise), the partner node continues to serve data for the halted node while you upgrade or perform maintenance on the node you halted. During non-disruptive upgrades of Data ONTAP, the user manually enters the storage failover takeover command to take over the partner node to allow the software upgrade to occur. The takeover node continues to serve data for both nodes during this operation.
Non-disruptive aggregate ownership relocation can be performed without a takeover and giveback.
The HA pair supplies non-disruptive operation and fault tolerance due to the following aspects of its configuration:
•The controllers in the HA pair are connected to each other either through an HA interconnect consisting of adapters and cables, or, in systems with two controllers in the same chassis, through an internal interconnect.
•The nodes use the interconnect to perform the following tasks:
– Each node continually checks whether the other node is functioning.
– They mirror log data for each other’s NVRAM or NVMEM.
– They use two or more disk shelf loops, or storage arrays, in which the following conditions apply:
• Each node manages its own disks or array LUNs.
• In case of takeover, the surviving node provides read/write access to the partner's disks or array LUNs until the failed node becomes available again.
•They own their spare disks, spare array LUNs, or both, and do not share them with the other node.
•They each have mailbox disks or array LUNs on the root volume that perform the following tasks:
– Maintain consistency between the pair
– Continually check whether the other node is running or whether it has performed a takeover
– Store configuration information
5.5 HA pairs within the cluster
HA pairs are components of the cluster, and both nodes in the HA pair are connected to other nodes in the cluster through the data and cluster networks. But only the nodes in the HA pair can takeover each other's storage.
Although the controllers in an HA pair are connected to other controllers in the cluster through the cluster network, the HA interconnect and disk-shelf connections are found only between the node and its partner and their disk shelves or array LUNs.
The HA interconnect and each node's connections to the partner's storage provide physical support for high-availability functionality. The high-availability storage failover capability does not extend to other nodes in the cluster.
|
Note: Network failover does not rely on the HA interconnect and allows data network interfaces to failover to different nodes in the cluster outside the HA pair. Network failover is different than storage failover since it enables network resiliency across all nodes in the cluster.
|
Non-HA (or stand-alone) nodes are not supported in a cluster containing two or more nodes. Although single node clusters are supported, joining two separate single node clusters to create one cluster is not supported, unless you wipe clean one of the single node clusters and join it to the other to create a two-node cluster that consists of an HA pair.
The diagram in Figure 5-2 shows two HA pairs in the cluster.
Figure 5-2 Two HA pairs in cluster
5.6 When takeovers occur
Takeovers can be initiated manually or occur automatically when a failover event happens, depending on how you configure the HA pair. In some cases, takeovers occur automatically regardless of configuration.
Takeovers can occur under the following conditions:
•A takeover is manually initiated with the storage failover takeover command.
•A node is in an HA pair with the default configuration for immediate takeover on panic, and that node undergoes a software or system failure that leads to a panic.
By default, the node automatically performs a giveback to return the partner to normal operation after the partner has recovered from the panic and booted up.
•A node that is in an HA pair undergoes a system failure (for example, a loss of power) and cannot reboot.
If the storage for a node also loses power at the same time, a standard takeover is not possible.
•A node does not receive heartbeat messages from its partner.
This could happen if the partner experienced a hardware or software failure that did not result in a panic but still prevented it from functioning correctly.
•You halt one of the nodes without using the -f or -inhibit-takeover true parameter.
•You reboot one of the nodes without using the -inhibit-takeover true parameter.
The -onreboot parameter of the storage failover command is enabled by default.
•Hardware-assisted takeover is enabled and triggers a takeover when the remote management device (RLM or Service Processor) detects failure of the partner node.
5.6.1 What happens during takeover
When a node takes over its partner, it continues to serve and update data in the partner's aggregates and volumes. To do this, it takes ownership of the partner's aggregates, and the partner's LIFs migrate according to network interface failover rules. Except for specific SMB 3.0 connections, existing SMB (CIFS) sessions are disconnected when the takeover occurs.
The following steps occur when a node takes over its partner:
1. If the negotiated takeover is user-initiated, aggregate relocation is performed to move data aggregates one at a time from the partner node to the node that is doing the takeover.
The current owner of each aggregate (except for the root aggregate) is changed from the target node to the node that is doing the takeover. There is a brief outage for each aggregate as ownership is changed. This outage is less than that accrued during a takeover that does not use aggregate relocation.
You can monitor the progress using the storage failover show-takeover command.
The aggregate relocation can be avoided during this takeover instance by using the -bypass-optimization parameter with the storage failover takeover command.
To bypass aggregate relocation during all future planned takeovers, set the -bypass-takeover-optimization parameter of the storage failover command to true.
|
Note: Aggregates are relocated serially during planned takeover operations to reduce client outage. If aggregate relocation is bypassed, it will result in longer client outage during planned takeover events.
|
2. If the takeover is user-initiated, the target node gracefully shuts down, followed by takeover of the target node's root aggr and any aggrs that were not relocated in step 1.
3. Data LIFs migrate from the target node to the node doing the takeover, or any other node in the cluster based on LIF failover rules, before the storage takeover begins.
The LIF migration can be avoided by using the -skip-lif-migration parameter with the storage failover takeover command.
4. Existing SMB (CIFS) sessions are disconnected when takeover occurs.
|
Note: Due to the nature of the SMB protocol, all SMB sessions except for SMB 3.0 sessions connected to shares with the Continuous Availability property set will be disruptive. SMB 1.0 and SMB 2.x sessions cannot reconnect after a takeover event. Therefore, takeover is disruptive and some data loss could occur.
|
5. SMB 3.0 sessions established to shares with the Continuous Availability property set can reconnect to the disconnected shares after a takeover event. If your site uses SMB 3.0 connections to Microsoft Hyper-V and the Continuous Availability property is set on the associated shares, takeover will be non-disruptive for those sessions.
5.6.2 What happens during giveback
The local node returns ownership of the aggregates and volumes to the partner node after any issues on the partner node are resolved or maintenance is complete. In addition, the local node returns ownership when the partner node has booted up and giveback is initiated either manually or automatically.
The following process takes place in a normal giveback. In this discussion, node A has taken over node B. Any issues on Node B have been resolved and it is ready to resume operations.
1. Any issues on node B have been resolved and it is displaying the following message:
Waiting for giveback
Waiting for giveback
2. The giveback is initiated by the storage failover giveback command or by automatic giveback if the system is configured for it. This initiates the process of returning ownership of the node B's aggregates and volumes from node A back to node B.
3. Node A returns control of the root aggregate first.
4. Node B proceeds to complete the process of booting up to its normal operating state.
5. As soon as Node B is at the point in the boot process where it can accept the non-root aggregates, Node A returns ownership of the other aggregates one at a time until giveback is complete.
You can monitor the progress of the giveback with the storage failover show-giveback command.
I/O resumes for each aggregate when giveback is complete for that aggregate, therefore reducing the overall outage window of each aggregate.
5.6.3 Displaying the nodes in a cluster
You can display information about the nodes in a cluster and their state. Example 5-1 displays information about all nodes in a four-node cluster.
Example 5-1 Displaying the nodes in a cluster
cluster1::> cluster show
Node Health Eligibility
--------------------- ------- ------------
node0 true true
node1 true true
node2 true true
node3 true true
The command displays the following information:
•Node name
•Whether the node is healthy
•Whether the node is eligible to participate in the cluster
•Whether the node holds epsilon (advanced privilege level or higher only)
5.6.4 HA policy and giveback of the root aggregate and volume
Aggregates are automatically assigned an HA policy of controller failover (CFO) or storage failover (SFO) that determines how the aggregate and its volumes are given back.
Aggregates created on Clustered Data ONTAP systems (except for the root aggregate containing the root volume) have an HA policy of SFO. During the giveback process, they are given back one at a time after the taken-over system boots.
The root aggregate always has an HA policy of CFO and is given back at the start of the giveback operation. This is necessary to allow the taken-over system to boot. The other aggregates are given back one at a time after the taken-over node completes the boot process.
The HA policy of an aggregate cannot be changed from SFO to CFO in normal operation.
5.7 How aggregate relocation works
Aggregate relocation operations take advantage of the HA configuration to move the ownership of storage aggregates within the HA pair. Aggregate relocation occurs automatically during manually initiated takeover to reduce downtime during planned failover events such as non-disruptive software upgrade, and can be initiated manually for load balancing, maintenance, and non-disruptive controller upgrade. Aggregate relocation cannot move ownership of the root aggregate.
Figure 5-3 shows the relocation of the ownership of aggregate aggr_1 from node1 to node2 in the HA pair.
Figure 5-3 Aggr1 - relocation of the ownership
The aggregate relocation operation can relocate the ownership of one or more storage failover (SFO) aggregates if the destination node can support the number of volumes in the aggregates. There is only a short interruption of access to each aggregate. Ownership information is changed one by one for the aggregates.
During takeover, aggregate relocation happens automatically when the takeover is initiated manually. Before the target controller is taken over, ownership of the aggregates belonging to that controller are moved one at a time to the partner controller. When giveback is initiated, the ownership is automatically moved back to the original node. The -bypass-optimization parameter can be used with the storage failover takeover command to suppress aggregate relocation during the takeover.
The aggregate relocation requires additional steps if the aggregate is currently used by an Infinite Volume with SnapDiff enabled.
|
Aggregate relocation and Infinite Volumes with SnapDiff enabled:
The aggregate relocation requires additional steps if the aggregate is currently used by an Infinite Volume with SnapDiff enabled. You must ensure that the destination node has a namespace mirror constituent and make decisions about relocating aggregates that include namespace constituents.
|
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.