Daily management of ECE storage
In this chapter, we introduce the typical management tasks of ECE storage. This process includes pdisk replacement, ECE node replacement, adding ECE nodes, and updating to new versions of IBM Spectrum Scale software.
We also briefly introduce the process of updating physical disks, SAS HBAs, network card firmware, and the operating system of the ECE storage nodes.
This chapter includes the following topics:
6.1 Drive replacement
When a pdisk is ready for replacement, ECE writes a notification to mmfslog and starts the pdReplacePdisk user exit. Pdisks that are ready for replacement also appear in the IBM Spectrum Scale GUI and in the output of the mmvdisk pdisk list -rg all -replace command.
Complete the following steps to replace a drive:
1. Run the mmvdisk pdisk replace -prepare command. This command verifies that the pdisk is in a replaceable state and turns on the replace light on the disk enclosure (if available).
2. Remove the old drive and insert a new drive of the same type and capacity.
3. Run the mmvdisk pdisk replace command to complete the disk replacement. When you finish the replace operation, you can run the mmvdisk pdisk list command and verify that the pdisk is now in OK state. ECE automatically runs a rebalance operation that moves data from the other drives in the declustered array to the new drive to balance the space in the array.
The disk replacement process is shown in the following example:
1. Prepare for replacement:
[root@gpfstest10 ~]# mmvdisk pdisk replace --rg rg1 --pdisk n005p004 --prepare
mmvdisk: Suspending pdisk n005p004 of RG rg1 in location 06VHMG6-8.mmvdisk: Location 06VHMG6-8 is Enclosure 06VHMG6 Drive 8.
mmvdisk: Carrier released.
mmvdisk:
mmvdisk: - Remove carrier.
mmvdisk: - Replace disk in location 06VHMG6-8 with type '90Y8878'.
mmvdisk: - Reinsert carrier.
mmvdisk: - Issue the following command:
mmvdisk:
mmvdisk: mmvdisk pdisk replace --recovery-group rg1 --pdisk 'n005p004'
2. Remove the old drive and insert a new blank drive of the same size and type.
3. Complete the drive replacement:
[root@gpfstest10 ~]# mmvdisk pdisk replace --rg rg1 --pdisk n005p004 -v no
mmvdisk:
mmvdisk: mmchcarrier : [I] Preparing a new pdisk for use may take many minutes.
mmvdisk:
mmvdisk:
mmvdisk: The following pdisks will be formatted on node gpfstest6:
mmvdisk: //gpfstest6/dev/sdd
mmvdisk: Pdisk n005p004 of RG rg1 successfully replaced.
mmvdisk: Resuming pdisk n005p004#0003 of RG rg1.
mmvdisk: Carrier resumed.
6.1.1 Drive replacement cancellation
In some cases, it might be necessary to cancel a drive replacement operation that was started. You can cancel the drive replacement at any point before issuing the mmvdisk pdisk replace command to finalize the operation.
To cancel a disk replacement operation that was started, complete the following steps:
1. If you inserted a new drive into the system, remove the new drive and reinsert the original drive into the disk enclosure.
2. Run mmvdisk pdisk replace -cancel command to cancel the replacement operation.
6.2 Replacing nodes
A broken node or an active node in a recovery group can be replaced with a new node without stopping service. In this section, two scenarios are described: replacing nodes with new drives and replacing nodes with existing drives.
|
Tip: In this release, we support replacing one node at a time only.
|
6.2.1 Node replacement with new drives
If a node must be replaced and the drives on the old node must be replaced with new drives, the following command sequence can be run to make this change. This process might be required if the old node is physically damaged so that old drives no longer work:
1. Prepare a new node with the same topology and operating system level as the node to be replaced. Server memory, the number of drives, drive types and size, and the network configurations of the new node must match the node that is replaced.
2. Configure the node for ssh/scp password-less access for root user to all IBM Spectrum Scale nodes and itself. Also, configure ssh/scp password-less access from all other nodes to this node.
3. Run mmaddnode to add this node into the IBM Spectrum Scale cluster. Configure the server license and open the IBM Spectrum Scale daemon. For more information about adding a node into gpfs cluster, see IBM Knowledge Center.
4. Define the node roles to be the same as the old server, such as quorum and fsmgr.
|
Note: If the maximum quorum nodes exist before nodes are added to the IBM Spectrum Scale cluster, change the old node to be replaced as non-quorum. Then, change the new node as quorum.
|
5. Run the mmvdisk server configure -N newnode command to configure the new node. Then, restart Spectrum Scale daemon on this node.
6. Run the mmvdisk rg replace command to replace the existing node with a new node. This process triggers several internal steps, including adding the new RG server to the current RG server list (node class), adding new pdisks into the current RG, deleting pdisks that are attached to the old node, and removing the old node from the RG server list. This process also triggers data rebalance to balance the space in the affected declustered arrays.
7. Run the mmvdisk rg list command to ensure that the new node is in the node class, all related pdisks are in OK state, and the node that was replaced and its related pdisks are no longer in the recovery group. Check the RG again after some time to ensure all DAs are in scrub state.
8. Run the mmshutdown -N and mmdelnode -N command to delete the node to be replaced from the cluster.
The following example shows node replacement with new pdisks. In this example, node gpfstest10 is replaced with gpfstest11. For readability, only the commands that are specific for ECE are shown. The older IBM Spectrum Scale commands, such as mmaddnode, are not listed:
1. List the servers that are used by the RG before adding a new node:
[root@gpfstest2 mytools]# mmvdisk rg list --rg rg1 --server
node
number server active remarks
------ -------------------------------- ------- -------
5 gpfstest10 yes serving rg1: root, LG005, LG010
2 gpfstest2 yes serving rg1: LG004, LG009
7 gpfstest3 yes serving rg1: LG002, LG007
6 gpfstest4 yes serving rg1: LG001, LG006
8 gpfstest5 yes serving rg1: LG003, LG008
2. Configure the node as an ECE storage node:
[root@gpfstest2 mytools]# mmvdisk server configure -N gpfstest11
mmvdisk: Checking resources for specified nodes.
mmvdisk: Setting configuration for node 'gpfstest11'.
mmvdisk: Node 'gpfstest11' has a scale-out recovery group disk topology.
mmvdisk: Node 'gpfstest11' is now configured to be a recovery group server.
mmvdisk: For configuration changes to take effect, GPFS should be restarted
mmvdisk: on node 'gpfstest11'.
3. Restart the daemon for the new node.
4. Run node replacement:
[root@gpfstest2 mytools]# mmvdisk rg replace --rg rg1 -N gpfstest10 --new-node gpfstest11
mmvdisk: Checking daemon status on node 'gpfstest11'.
mmvdisk: Analyzing disk topology for node 'gpfstest2'.
mmvdisk: Analyzing disk topology for node 'gpfstest4'.
mmvdisk: Analyzing disk topology for node 'gpfstest3'.
mmvdisk: Analyzing disk topology for node 'gpfstest5'.
mmvdisk: Analyzing disk topology for node 'gpfstest11'.
mmvdisk: Adding 'gpfstest11' to node class 'r1'.
mmvdisk: Checking resources for specified nodes.
mmvdisk: Updating server list for recovery group 'rg1'.
mmvdisk: Updating pdisk list for recovery group 'rg1'.
mmvdisk: This could take a long time.
mmvdisk: Adding node 'gpfstest11' pdisks to recovery group 'rg1'.
mmvdisk: The following pdisks will be formatted on node gpfstest10:
mmvdisk: //gpfstest11/dev/sdd
mmvdisk: //gpfstest11/dev/sdb
mmvdisk: //gpfstest11/dev/sdc
mmvdisk: //gpfstest11/dev/sda
mmvdisk: //gpfstest11/dev/sdf
mmvdisk: //gpfstest11/dev/sde
mmvdisk: //gpfstest11/dev/sdg
mmvdisk: Deleting node 'gpfstest10' pdisks from recovery group 'rg1'.
mmvdisk: Removing node 'gpfstest10' from node class 'r1'.
mmvdisk: Updating server list for recovery group 'rg1'.
5. After the replace command completes, rerun the list command. At this point, we see that gpfstest10 is out of the server list while gpfstest11 is in the list:
[root@gpfstest2 mytools]# mmvdisk rg list --rg rg1 --server
node
number server active remarks
------ -------------------------------- ------- -------
4 gpfstest11 yes serving rg1: root, LG004, LG005
2 gpfstest2 yes serving rg1: LG009, LG010
7 gpfstest3 yes serving rg1: LG002, LG007
6 gpfstest4 yes serving rg1: LG001, LG006
8 gpfstest5 yes serving rg1: LG003, LG008
6.2.2 Replacing nodes and preserving drives from the old node
Circumstances exist in which a node must be replaced, but the drives for that node can still be used. For example, the node crashes and cannot return to service, but all of the pdisks that are attached to this node are working and contain some data strips.
Although this situation is not supported by the mmvdisk command at the time of this writing, this situation will be supported in the future. For now, the recommended approach is to “clone” a new node and reuse the pdisks from the old node.
Complete the following steps:
1. Ensure that the old node to be replaced is in “unknown” state. If this node is still active, it must be set to out of service. For example, shut down the node and disable the IBM Spectrum Scale daemon network interface on this node.
2. Prepare one new node, with the potential to have same topology and operating system level as the node to be replaced. (server memory, HBA cards, network configurations, and so on).
3. Install gpfs rpms with the same version as the old node, and build the GPL layer.
4. Remove all the disk drives that are used by ECE from the old node and insert them into the new node.
|
Important: Each disk drive must be placed into the slot in the new node that corresponds to the same slot in the old node. For example, if you removed the drive from slot 1 in the old node, it must be placed into slot 1 in the new node.
|
5. Set the hostname and IP addresses from the old node to the new node and configure the SSH/SCP. Check that password-less ssh/scp is configured correctly from this node to all the other nodes in the cluster and vice versa. This process can be done by using the mmnetverify connectivityc-N all command.
6. Check that all the physical disks that were moved from the old node can be seen on the new node.
7. Run the mmsdrrestore -p nodename command on the new node, where nodename is any node in the cluster that is in active state.
8. Run the mmstartup -N newnode command to start the daemon on this node.
9. Use the same mmvdisk rg list commands to verify that the new node and its pdisks are in the correct state.
|
Tip: After the node replacement process is completed the first scenario, a different node name and IP address exists for the pdisks. The old node name and IP address are removed from this recovery group.
For the second scenario, the same node and IP address is used. The pdisk names also are the same. In this scenario, it appears the old node was recovered instead of being replaced with a new node (although parameters, such as the system serial number, are different).
|
6.3 Adding nodes
With IBM Spectrum Scale Erasure Code Edition, you can scale out storage capacity and storage bandwidth by adding nodes to a recovery group. The current limit is 32 nodes per recovery group, and 128 ECE storage nodes per cluster.
When a node is added to a recovery group, the new node must have the same configuration and operating system level as the existing nodes. Server memory, the number of drives, drive types and size, and network configurations of the new node must match the existing nodes in the recovery group.
Scaling out an ECE RG can be divided into the following steps:
1. Add nodes into IBM Spectrum Scale cluster.
2. Add the nodes into an ECE RG.
To add nodes into IBM Spectrum Scale cluster, follow the procedure for adding nodes for general IBM Spectrum Scale. Here, only the process for how to add the node into RG is described.
Use the mmvdisk rg add command to support scaling out the recovery group. This function first adds the new node into the RG, which adds one server into the recovery group server list (mmvdisk node class) and adds pdisks into every DA. This process leads to all DAs going into rebalance state.
After the rebalance finishes for all DAs, two more user Log Groups are created in this recovery group, which creates VDisks or NSDs for all VDisk sets and adds the NSDS into file systems that are backed by the recovery group. The spare disk settings are also updated for every DA.
Complete the following steps to add one node:
1. Ensure that the new node is a member of the IBM Spectrum Scale cluster and is in active state. For more information about adding a node into a cluster, see IBM Knowledge Center. Also, ensure that the node accepted the IBM Spectrum Scale server license.
2. Run the mmvdisk server list -N newnode --disk-topology command to verify that this node includes the same disk topology as the recovery group.
3. Run the mmvdisk server configure -N newnode -recycle one command to configure it as an ECE server. This command checks that the new server topology matches the servers, and sets the initial tuning configuration values for this node. It also restarts the daemon on the new node to apply these configuration changes.
4. Run the mmvdisk rg add --rg rgname -N newnode command to add this node to a recovery group. This command returns without waiting for the rebalance to finish. All DAs should be in rebalance state. At this point, you must wait for all DAs to enter the scrub state before completing the node add procedure. While waiting, run the mmvdisk recoverygroup list --recovery-group rgname --declustered-array command to check the DA state.
5. After all DAs complete their rebalance work and are in scrub state, run the mmvdisk recoverygroup add --recovery-group rgname --complete-node-add command to finish adding the node. This process creates two log groups, creates VDisks for all vdisk sets, creates NSDs, and add the NSDs to file systems if the VDisk sets belong to some file system.
In the following example, node gpfstest12 is added into the recoverygroup rg1:
1. Before adding nodes to rg1, five nodes are in this recovery group. The file system gpfs1 also was created on it. File system gpfs1 features 10 NSDs:
[root@gpfstest2 mytools]# mmvdisk server list --nc r1
node
number server active remarks
------ -------------------------------- ------- -------
4 gpfstest11 yes serving rg1: root, LG005, LG010
2 gpfstest2 yes serving rg1: LG004, LG009
7 gpfstest3 yes serving rg1: LG002, LG007
6 gpfstest4 yes serving rg1: LG001, LG006
8 gpfstest5 yes serving rg1: LG003, LG008
[root@gpfstest2 mytools]# mmlsdisk gpfs1
disk driver sector failure holds holds storage
name type size group metadata data status availability pool
------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------
RG001LG001VS001 nsd 512 1 yes yes ready up system
RG001LG002VS001 nsd 512 2 yes yes ready up system
RG001LG003VS001 nsd 512 1 yes yes ready up system
RG001LG004VS001 nsd 512 2 yes yes ready up system
RG001LG005VS001 nsd 512 1 yes yes ready up system
RG001LG006VS001 nsd 512 2 yes yes ready up system
RG001LG007VS001 nsd 512 1 yes yes ready up system
RG001LG008VS001 nsd 512 2 yes yes ready up system
RG001LG009VS001 nsd 512 1 yes yes ready up system
RG001LG010VS001 nsd 512 2 yes yes ready up system
[root@gpfstest2 mytools]# df -H /gpfs1
Filesystem Size Used Avail Use% Mounted on
gpfs1 609G 4.0G 605G 1% /gpfs1
2. Configure the node as an ECE storage node:
[root@gpfstest2 mytools]# mmvdisk server configure -N gpfstest12
mmvdisk: Checking resources for specified nodes.
mmvdisk: Setting configuration for node 'gpfstest12'.
mmvdisk: Node 'gpfstest12' has a scale-out recovery group disk topology.
mmvdisk: Node 'gpfstest12' is now configured to be a recovery group server.
mmvdisk: For configuration changes to take effect, GPFS should be restarted
mmvdisk: on node 'gpfstest12'.
3. Restart the daemon and check that the node is in active state. This restart can be skipped if the -recycle one parameter was passed in the previous step:
[root@gpfstest2 mytools]# mmshutdown -N gpfstest12 ;mmstartup -N gpfstest12
Thu Aug 22 11:02:53 EDT 2019: mmshutdown: Starting force unmount of GPFS file systems
Thu Aug 22 11:02:58 EDT 2019: mmshutdown: Shutting down GPFS daemons
Thu Aug 22 11:03:06 EDT 2019: mmshutdown: Finished
Thu Aug 22 11:03:06 EDT 2019: mmstartup: Starting GPFS ...
[root@gpfstest2 mytools]# mmgetstate -N gpfstest12
Node number Node name GPFS state
-------------------------------------------
3 gpfstest12 active
4. Run the mmvdisk rg add command:
[root@gpfstest2 mytools]# mmvdisk rg add --rg rg1 -N gpfstest12
mmvdisk: Checking daemon status on node 'gpfstest12'.
mmvdisk: Checking resources for specified nodes.
mmvdisk: Adding 'gpfstest12' to node class 'r1'.
mmvdisk: Analyzing disk topology for node 'gpfstest2'.
mmvdisk: Analyzing disk topology for node 'gpfstest11'.
mmvdisk: Analyzing disk topology for node 'gpfstest4'.
mmvdisk: Analyzing disk topology for node 'gpfstest3'.
mmvdisk: Analyzing disk topology for node 'gpfstest5'.
mmvdisk: Analyzing disk topology for node 'gpfstest12'.
mmvdisk: Updating server list for recovery group 'rg1'.
mmvdisk: Updating pdisk list for recovery group 'rg1'.
mmvdisk: The following pdisks will be formatted on node gpfstest11:
mmvdisk: //gpfstest12/dev/sdf
mmvdisk: //gpfstest12/dev/sdh
mmvdisk: //gpfstest12/dev/sdg
mmvdisk: //gpfstest12/dev/sdb
mmvdisk: //gpfstest12/dev/sdd
mmvdisk: //gpfstest12/dev/sde
mmvdisk: //gpfstest12/dev/sdc
mmvdisk: Node 'gpfstest12' added to recovery group 'rg1'.
mmvdisk: Log group and vdisk set operations for recovery group 'rg1'
mmvdisk: must be deferred until rebalance completes in all declustered arrays.
mmvdisk: To monitor the progress of rebalance, use the command:
mmvdisk: mmvdisk recoverygroup list --recovery-group rg1 --declustered-array
mmvdisk: When rebalance is completed, issue the command:
mmvdisk: mmvdisk recoverygroup add --recovery-group rg1 --complete-node-add
5. Run the mmvdisk command to monitor the rebalance progress. The rebalance state is highlighted in red:
[root@gpfstest2 mytools]# mmvdisk recoverygroup list --recovery-group rg1 --declustered-array
declustered needs vdisks pdisks replace capacity scrub
array service user log total spare threshold total raw free raw duration background task
----------- ------- ---- --- ----- ----- --------- --------- -------- -------- ---------------
DA1 no 10 11 42 3 2 10 TiB 10 TiB 14 days rebalance (0%)
mmvdisk: Total capacity is the raw space before any vdisk set definitions.
mmvdisk: Free capacity is what remains for additional vdisk set definitions.
mmvdisk: Attention: Recovery group 'rg1' has an incomplete node addition (gpfstest12).
mmvdisk: Complete the node addition with the command:
mmvdisk: mmvdisk recoverygroup add --recovery-group rg1 --complete-node-add
6. After a few minutes, the rebalance finishes. The DA is in scrub state, which is highlighted in green:
[root@gpfstest2 mytools]# mmvdisk recoverygroup list --recovery-group rg1 --declustered-array
declustered needs vdisks pdisks replace capacity scrub
array service user log total spare threshold total raw free raw duration background task
----------- ------- ---- --- ----- ----- --------- --------- -------- -------- ---------------
DA1 no 10 11 42 3 2 10 TiB 10 TiB 14 days scrub (0%)
mmvdisk: Total capacity is the raw space before any vdisk set definitions.
mmvdisk: Free capacity is what remains for additional vdisk set definitions.
mmvdisk: Attention: Recovery group 'rg1' has an incomplete node addition (gpfstest12).
mmvdisk: Complete the node addition with the command:
mmvdisk: mmvdisk recoverygroup add --recovery-group rg1 --complete-node-add
7. Run the mmvdisk rg add -complete-node-add command:
[root@gpfstest2 mytools]# mmvdisk recoverygroup add --recovery-group rg1 --complete-node-add
mmvdisk: Verifying that the DAs in recovery group 'rg1' are idle.
mmvdisk: Updating log vdisks for recovery group 'rg1'.
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG011LOGHOME
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG012LOGHOME
mmvdisk: Updating vdisk sets for recovery group 'rg1.'
mmvdisk: 2 vdisks and 2 NSDs will be created in vdisk set 'vs1'.
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG011VS001
mmvdisk: (mmcrvdisk) [I] Processing vdisk RG001LG012VS001
mmvdisk: Created all vdisks in vdisk set 'vs1'.
mmvdisk: (mmcrnsd) Processing disk RG001LG011VS001
mmvdisk: (mmcrnsd) Processing disk RG001LG012VS001
mmvdisk: Created all NSDs in vdisk set 'vs1'.
mmvdisk: Extending file system 'gpfs1'.
mmvdisk: The following disks of gpfs1 will be formatted on node gpfstest2:
mmvdisk: RG001LG011VS001: size 4972 MB
mmvdisk: RG001LG012VS001: size 4972 MB
mmvdisk: Extending Allocation Map
mmvdisk: Checking Allocation Map for storage pool system
mmvdisk: Completed adding disks to file system gpfs1.
8. Check the server list to verify that one more server and two more LGs were added:
[root@gpfstest2 tangyub]# mmvdisk server list --nc r1
node
number server active remarks
------ -------------------------------- ------- -------
4 gpfstest11 yes serving rg1: root, LG005, LG010
3 gpfstest12 yes serving rg1: LG004, LG011
2 gpfstest2 yes serving rg1: LG009, LG012
7 gpfstest3 yes serving rg1: LG002, LG007
6 gpfstest4 yes serving rg1: LG001, LG006
8 gpfstest5 yes serving rg1: LG003, LG008
[root@gpfstest2 tangyub]# mmlsdisk gpfs1
disk driver sector failure holds holds storage
name type size group metadata data status availability pool
------------ -------- ------ ----------- -------- ----- ------------- ------------ ------------
RG001LG001VS001 nsd 512 1 yes yes ready up system
RG001LG002VS001 nsd 512 2 yes yes ready up system
RG001LG003VS001 nsd 512 1 yes yes ready up system
RG001LG004VS001 nsd 512 2 yes yes ready up system
RG001LG005VS001 nsd 512 1 yes yes ready up system
RG001LG006VS001 nsd 512 2 yes yes ready up system
RG001LG007VS001 nsd 512 1 yes yes ready up system
RG001LG008VS001 nsd 512 2 yes yes ready up system
RG001LG009VS001 nsd 512 1 yes yes ready up system
RG001LG010VS001 nsd 512 2 yes yes ready up system
RG001LG011VS001 nsd 512 1 yes yes ready up system
RG001LG012VS001 nsd 512 2 yes yes ready up system
[root@gpfstest2 tangyub]# df -H /gpfs1
Filesystem Size Used Avail Use% Mounted on
gpfs1 730G 4.2G 726G 1% /gpfs1
6.4 Upgrading to a new IBM Spectrum Scale release
Two upgrade modes are supported for IBM Spectrum Scale release upgrade: offline and online.
For offline upgrade, IBM Spectrum Scale daemon is shut down on all nodes and the IBM Spectrum Scale packages are updated. Next, IBM Spectrum Scale daemon is started on all nodes. For more information about the offline upgrade procedure, see this web page.
For the online upgrade, the IBM Spectrum Scale software must be upgraded sequentially for each of the servers in a recovery group. The Spectrum Scale services must be restarted on each node, and only one node is restarted at a time to minimize the disruption to active workloads.
To run the online upgrade manually, complete the following steps:
1. Check whether the quorum will be lost for IBM Spectrum Scale cluster after this node is shut down. If the quorum will be lost, stop the upgrade and troubleshoot the problem.
2. Use the mmvdisk recovery group list command to check the recovery group’s fault tolerance to ensure that the recovery group has at least one node and one pdisk fault tolerance. If failed nodes or drives exist, they must be repaired before continuing with the upgrade.
3. Use the mmchmgr command to transfer all manager roles on the node to be upgraded to another active node; for example, the file manager and cluster manager roles.
4. If CES protocols are running on the node to be upgraded, use the mmces command to transfer those addresses to another CES node in the same ECE building block.
5. Transfer the log groups that are served by the node to be upgrade to other active nodes by using the tsrecgroupserver command.
6. Unmount any Spectrum Scale file systems that are mounted by using the mmunmount command.
7. Run the mmshutdown command to shut down Spectrum Scale services on the node.
8. Install the new IBM Spectrum Scale packages and build the GPL. This process is the same as the general IBM Spectrum Scale upgrade process. For more information, see the Rolling Upgrade section in IBM Knowledge Center.
9. Run the mmstartup command to open the daemon on this node. Wait for approximately 3 minutes. Then, check that all of the pdisks in this node are in OK state and that all LGs are balanced in the node class.
10. Transfer back any of the all roles that were moved in Step 3 back to the newly upgraded node.
11. Repeat steps 1 - 10 on each of the other nodes in the ECE building block to complete the rolling upgrade.
6.5 Upgrading operating system, firmware, driver, and patch
With ECE (as with most software defined storage products), the management of each server’s operating system, firmware, driver, and patch updates are the responsibility of the customer. Planning is required to minimize any effect to any workloads that are served by the ECE systems because some operations can cause an ECE node to be out of service, which automatically triggers background integrity management operations, such as rebuild and rebalance.
Although other operations might affect an individual drive only, and all nodes stay in active state, performance might be affected in this case. We strongly encourage you to make a detailed plan before running any command, and if possible, verify that plan on a test cluster.
An operating system upgrade and a network adapter driver upgrade leads to a node out of service from an IBM Spectrum Scale perspective. The fault tolerance and performance are affected.
If disk drive firmware is updated online, I/O errors or I/O timeouts can occur during the upgrading process. Example 6-1 shows the IBM Spectrum Scale logs from the recovery group master node while the disk firmware is upgrading, which is listed here for reference. You can see pdisks go into “diagnosing” state for a short time, and then return to “OK” state. You might see write errors on the pdisk during the firmware upgrade steps.
Example 6-1 Log messages in mmfs.log during pdisk firmware updates
2019-08-22_13:16:51.559-0400: [D] Pdisk n003p004 of RG rg1 state changed from ok/0000.000 to diagnosing/0020.000.
2019-08-22_13:16:56.570-0400: [D] Pdisk n003p004 of RG rg1 state changed from diagnosing/0020.000 to ok/0000.000.
2019-08-22_13:19:34.652-0400: [D] Pdisk n003p005 of RG rg1 state changed from ok/0000.000 to diagnosing/0020.000.
2019-08-22_13:19:38.333-0400: [D] Pdisk n003p005 of RG rg1 state changed from diagnosing/0020.000 to ok/0000.000.
2019-08-22_13:20:56.591-0400: [D] Pdisk n003p006 of RG rg1 state changed from ok/0000.000 to diagnosing/0020.000.
2019-08-22_13:21:00.182-0400: [D] Pdisk n003p006 of RG rg1 state changed from diagnosing/0020.000 to ok/0000.000.
2019-08-22_13:22:18.553-0400: [D] Pdisk n003p001 of RG rg1 state changed from ok/0000.000 to diagnosing/0020.000.
2019-08-22_13:22:22.724-0400: [D] Pdisk n003p001 of RG rg1 state changed from diagnosing/0020.000 to ok/0000.000.
2019-08-22_13:23:40.484-0400: [D] Pdisk n003p003 of RG rg1 state changed from ok/0000.000 to diagnosing/0020.000.
2019-08-22_13:23:41.599-0400: [E] Pdisk n003p003 of RG rg1 path //gpfstest12/dev/sdg: pdisk time-out on write: sector 302032880 length 8.
2019-08-22_13:23:41.698-0400: [E] Pdisk n003p003 of RG rg1 path //gpfstest12/dev/sdg: pdisk time-out on write: sector 83750376 length 2056.
2019-08-22_13:23:44.399-0400: [D] Pdisk n003p003 of RG rg1 state changed from diagnosing/0020.000 to ok/0000.000
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.