Maintenance
Among the many benefits that the IBM Spectrum Virtualize software provides is to greatly simplify the storage management tasks that system administrators need to perform. However, as the IT environment grows and gets renewed, so does the storage infrastructure.
This chapter highlights guidance for the day-to-day activities of storage administration by using the IBM Spectrum Virtualize software installed on IBM SAN Volume Controller, the IBM Storwize family, and IBM FlashSystem V9000. This guidance can help you to maintain your storage infrastructure with the levels of availability, reliability, and resiliency demanded by today’s applications, and to keep up with storage growth needs.
This chapter focuses on the most important topics to consider in IBM Spectrum Virtualize administration so that you can use it as a checklist. It also provides tips and guidance.
 
Important: The practices that are described here were effective in many IBM Spectrum Virtualize installations worldwide for organizations in several areas. They all had one common need, which was to easily, effectively, and reliably manage their SAN storage environment. Nevertheless, whenever you have a choice between two possible implementations or configurations, if you look deep enough, you always have both advantages and disadvantages over one another. Do not take these practices as absolute truth, but rather use them as a guide. The choice of which approach to use is ultimately yours.
This chapter includes the following sections:
9.1 Documenting IBM Spectrum Virtualize and SAN environment
This section focuses on the challenge of automating the documentation that is needed for an IBM Spectrum Virtualize solution. Consider the following points:
•Several methods and tools are available to automate the task of creating and updating the documentation. Therefore, the IT infrastructure might handle this task.
•Planning is key to maintaining sustained and organized growth. Accurate documentation of your storage environment is the blueprint with which you plan your approach to short-term and long-term storage growth.
•Your storage documentation must be conveniently available and easy to consult when needed. For example, you might need to determine how to replace your core SAN directors with newer ones, or how to fix the disk path problems of a single server. The relevant documentation might consist of a few spreadsheets and a diagram.
 
Storing documentation: Avoid storing IBM Spectrum Virtualize and SAN environment documentation only in the SAN. If your organization has a disaster recovery plan, include this storage documentation in it. Follow its guidelines about how to update and store this data. If no disaster recovery plan exists and you have the proper security authorization, it might be helpful to store an updated copy offsite.
In theory, this IBM Spectrum Virtualize and SAN environment documentation should be sufficient for any system administrator who has average skills in the products that are included. Make a copy that includes all of your configuration information. Use the copy to create a functionally equivalent copy of the environment by using similar hardware without any configuration, off-the-shelf media, and configuration backup files. You might need the copy if you ever face a disaster recovery scenario, which is also why it is so important to run periodic disaster recovery tests.
Create the first version of this documentation as you install your solution. If you completed forms to help plan the installation of your IBM Spectrum Virtualize solution, use these forms to help you document how your IBM Spectrum Virtualize solution was first configured. Minimum documentation is needed for an IBM Spectrum Virtualize solution. Because you might have more business requirements that require other data to be tracked, remember that the following sections do not address every situation.
9.1.1 Naming conventions
Whether you are creating your IBM Spectrum Virtualize and SAN environment documentation or you are updating what is already in place, first evaluate whether you have a good naming convention in place. With a good naming convention, you can quickly and uniquely identify the components of your IBM Spectrum Virtualize and SAN environment. System administrators can then determine whether a name belongs to a volume, storage pool, MDisk, host, or host bus adapter (HBA) by looking at it. Because error messages often point to the device that generated an error, a good naming convention quickly highlights where to start investigating when an error occurs.
Typical IBM Spectrum Virtualize and SAN component names limit the number and type of characters you can use. For example, IBM Spectrum Virtualize names are limited to 63 characters, which makes creating a naming convention a bit easier than in previous versions of IBM Spectrum Virtualize code.
 
Names: In previous versions of IBM Spectrum Virtualize code, names were limited to 15 characters. Starting with version 7.1, the limit is 63 characters.
Many names in IBM Spectrum Virtualize and SAN environment can be modified online. Therefore, you do not need to worry about planning outages to implement your new naming convention. Server names are the exception, as explained in “Hosts” on page 312.
The naming examples that are used in the following sections are effective in most cases, but might not be fully adequate for your particular environment or needs. The naming convention to use is your choice, but you must implement it in the whole environment.
Storage controllers
IBM Spectrum Virtualize names the storage controllers controllerX, with X being a sequential decimal number. If multiple controllers are attached to your IBM Spectrum Virtualize solution, change the name so that it includes, for example, the vendor name, the model, or its serial number. Therefore, if you receive an error message that points to controllerX, you do not need to log in to IBM Spectrum Virtualize to know which storage controller to check.
 
Note: IBM Spectrum Virtualize detects controllers based on their WWNN. If you have a storage controller that has one WWNN for each worldwide port name (WWPN), this configuration might lead to many controllerX names pointing to the same physical box. In this case, prepare a naming convention to cover this situation.
MDisks and storage pools
When IBM Spectrum Virtualize detects new MDisks, it names them by default as mdiskXX, where XX is a sequential number. Change the XX value to something more meaningful. For example, you can change it to include the following information:
•A reference to the storage controller it belongs to (such as its serial number or last digits)
•The extpool, array, or RAID group that it belongs to in the storage controller
•The LUN number or name it has in the storage controller
Consider the following examples of MDisk names with this convention:
•23K45_A7V10, where 23K45 is the serial number, 7 is the array, and 10 is the volume
•75VXYZ1_02_0206, where 75VXYZ1 is the serial number, 02 is the extpool, and 0206 is the LUN
Storage pools have several different possibilities. One possibility is to include the storage controller, the type of back-end disks, the RAID type, and sequential digits. If you have dedicated pools for specific applications or servers, another possibility is to use them instead. Consider the following examples:
•P05XYZ1_3GR5: Pool 05 from serial 75VXYZ1, LUNs with 300 GB FC DDMs and RAID 5
•P16XYZ1_EX01: Pool 16 from serial 75VXYZ1, pool 01 dedicated to Exchange Mail servers
Volumes (formerly VDisks)
Volume names should include the following information:
•The hosts or cluster to which the volume is mapped
•A single letter that indicates its usage by the host, as shown in the following examples:
 – B: For a boot disk, or R for a rootvg disk (if the server boots from SAN)
 – D: For a regular data disk
 – Q: For a cluster quorum disk (do not confuse with IBM Spectrum Virtualize quorum disks)
 – L: For a database logs disks
 – T: For a database table disk
•A few sequential digits, for uniqueness
For example, ERPNY01_T03 indicates a volume that is mapped to server ERPNY01 and database table disk 03.
Hosts
In today’s environment, administrators deal with large networks, the internet, and Cloud Computing. Use good server naming conventions so that they can quickly identify a server and determine the following information:
•Where it is (to know how to access it)
•What kind it is (to determine the vendor and support group in charge)
•What it does (to engage the proper application support and notify its owner)
•Its importance (to determine the severity if problems occur)
Changing a server’s name in IBM Spectrum Virtualize is as simple as changing any other IBM Spectrum Virtualize object name. However, changing the name on the operating system of a server might have implications for application configuration and require a server reboot. Therefore, you might want to prepare a detailed plan if you decide to rename several servers in your network. The following example is for server name conventions for LLAATRFFNN:
•LL is the location, which might designate a city, data center, building floor, or room.
•AA is a major application, for example, billing, ERP, and Data Warehouse.
•T is the type, for example, UNIX, Windows, and VMware.
•R is the role, for example, Production, Test, Q&A, and Development.
•FF is the function, for example, DB server, application server, web server, and file server.
•NN is numeric.
SAN aliases and zones
SAN aliases often need to reflect only the device and port that is associated to it. Including information about where one particular device port is physically attached on the SAN might lead to inconsistencies if you make a change or perform maintenance and then forget to update the alias. Create one alias for each device port WWPN in your SAN, and use these aliases in your zoning configuration. Consider the following examples:
•NYBIXTDB02_FC2: Interface fcs2 of AIX server NYBIXTDB02 (WWPN)
•SVC02_N2P4: SAN Volume Controller cluster SVC02, port 4 of node 2 (WWPN format 5005076XXXXXXXXX)
Be mindful of the IBM Spectrum Virtualize port aliases. There are mappings between the last digits of the port WWPN and the node FC port, but these mappings vary depending on the SAN Volume Controller model or the Storwize product.
•SVC02_IO2_A: SAN Volume Controller cluster SVC02, ports group A for iogrp 2 (aliases SVC02_N3P1, SVC02_N3P3, SVC02_N4P1, and SVC02_N4P3)
•D8KXYZ1_I0301: DS8000 serial number 75VXYZ1, port I0301(WWPN)
•TL01_TD06: Tape library 01, tape drive 06 (WWPN)
If your SAN does not support aliases, for example, in heterogeneous fabrics with switches in some interoperations modes, use WWPNs in your zones. However, remember to update every zone that uses a WWPN if you ever change it.
Have your SAN zone name reflect the devices in the SAN it includes (normally in a one-to-one relationship) as shown in the following examples:
•servername_svcclustername (from a server to the SAN Volume Controller)
•svcclustername_storagename (from the SAN Volume Controller cluster to its back-end storage)
•svccluster1_svccluster2 (for remote copy services)
9.1.2 SAN fabrics documentation
The most basic piece of SAN documentation is a SAN diagram. It is likely to be one of the first pieces of information you need if you ever seek support from your SAN switches vendor. Also, a good spreadsheet with ports and zoning information eases the task of searching for detailed information, which, if included in the diagram, makes the diagram difficult to use.
Brocade SAN Health
The Brocade SAN Health Diagnostics Capture tool is a no-cost, automated tool that can help you retain this documentation. SAN Health consists of a data collection tool that logs in to the SAN switches that you indicate and collects data by using standard SAN switch commands. The tool then creates a compressed file with the data collection. This file is sent to a Brocade automated machine for processing by secure web or email.
After some time (typically a few hours), the user receives an email with instructions about how to download the report. The report includes a Visio Diagram of your SAN and an organized Microsoft Excel spreadsheet that contains all your SAN information. For more information and to download the tool, see this website:
The first time that you use the SAN Health Diagnostics Capture tool, explore the options provided to learn how to create a well-organized and useful diagram. Figure 9-1 shows an example of a poorly formatted diagram.
Figure 9-1 A poorly formatted SAN diagram
Figure 9-2 shows a tab of the SAN Health Options window in which you can choose the format of SAN diagram that best suits your needs. Depending on the topology and size of your SAN fabrics, you might want to manipulate the options in the Diagram Format or Report Format tabs.
Figure 9-2 Brocade SAN Health Options window
SAN Health supports switches from manufacturers other than Brocade, such as McData and Cisco. Both the data collection tool download and the processing of files are available at no cost. You can download Microsoft Visio and Excel viewers at no cost from the Microsoft website.
Another tool, which is known as SAN Health Professional, is also available for download at no cost. With this tool, you can audit the reports in detail by using advanced search functions and inventory tracking. You can configure the SAN Health Diagnostics Capture tool as a Windows scheduled task.
 
Tip: Regardless of the method that is used, generate a fresh report at least once a month. Keep previous versions so that you can track the evolution of your SAN.
IBM Spectrum Control reporting
If you have IBM Spectrum Control running in your environment, you can use it to generate reports on your SAN. For more information about how to configure and schedule IBM Spectrum Control reports, see the IBM Spectrum Control documentation at:
Ensure that the reports that you generate include all the information that you need. Schedule the reports with a period that you can use to backtrack any changes that you make.
9.1.3 IBM Spectrum Virtualize documentation
You can back up the configuration data for an IBM Spectrum Virtualize system after preliminary tasks are completed. Configuration data for the system provides information about your system and the objects that are defined in it.
Before you back up your configuration data, the following prerequisites must be met:
•No independent operations that change the configuration for the system can be running while the backup command is running.
•No object name can begin with an underscore character (_).
 
Note: The system automatically creates a backup of the configuration data each day at 1 AM. This backup is known as a cron backup and is written on the configuration node to /dumps/svc.config.cron.xml_<serial#>.
Use these instructions to generate a manual backup at any time:
1. Issue the svcconfig backup command to back up your configuration:
The command displays messages similar to the ones in Example 9-1.
Example 9-1 Sample svcconfig backup command output
CMMVC6112W io_grp io_grp1 has a default name
CMMVC6112W io_grp io_grp2 has a default name
CMMVC6112W mdisk mdisk14 ...
CMMVC6112W node node1 ...
CMMVC6112W node node2 ...
....................................................
The svcconfig backup command creates three files that provide information about the backup process and the configuration. These files are created in the /dumps directory of the configuration node. Table 9-1 describes the three files that are created by the backup process.
Table 9-1 Files created by the backup process
File name
Description
svc.config.backup.xml_<serial#>
Contains your configuration data.
svc.config.backup.sh_<serial#>
Contains the names of the commands that were issued to create the backup of the system.
svc.config.backup.log_<serial#>
Contains details about the backup, including any reported errors or warnings.
2. Check that the svcconfig backup command completes successfully, and examine the command output for any warnings or errors. The following output is an example of the message that is displayed when the backup process is successful:
CMMVC6155I SVCCONFIG processing completed successfully
3. If the process fails, resolve the errors and run the command again.
4. Copy the backup file from the configuration node. With MS Windows, use the PuTTY pscp utility. With UNIX or Linux, you can use the standard scp utility.
The configuration backup file is in Extensible Markup Language (XML) format and can be imported into your IBM Spectrum Virtualize documentation spreadsheet.
The configuration backup file might contain too much data, for example it contains information about each internal storage drive that is installed in the system. Importing the file into your IBM Spectrum Virtualize documentation spreadsheet might make it unreadable.
In this case, consider collecting the output of specific commands. At a minimum, you should collect the output of the following commands:
•svcinfo lsfabric
•svcinfo lssystem
•svcinfo lsmdisk
•svcinfo lsmdiskgrp
•svcinfo lsvdisk
•svcinfo lshost
•svcinfo lshostvdiskmap
Import the commands into a spreadsheet, preferably with each command output on a separate sheet.
One way to automate either task is to first create a batch file (Windows) or shell script (UNIX or Linux) that collects and stores this information. For more information, see 9.8, “IBM Spectrum Virtualize scripting” on page 337. Then, use spreadsheet macros to import the collected data into your IBM Spectrum Virtualize documentation spreadsheet.
When you are gathering IBM Spectrum Virtualize information, consider the following preferred practices:
•If you are collecting the output of specific commands, use the -delim option of these commands to make their output delimited by a character other than tab, such as comma, colon, or exclamation mark. You can import the temporary files into your spreadsheet in comma-separated values (CSV) format, specifying the same delimiter.
 
Note: It is important to use a delimiter that is not already part of the output of the command. Commas can be used if the output is a particular type of list. Colons might be used for special fields, such as IPv6 addresses, WWPNs, or iSCSI names.
•If you are collecting the output of specific commands, save the output to temporary files. To make your spreadsheet macros simpler, you might want to preprocess the temporary files and remove any “garbage” or undesired lines or columns. With UNIX or Linux, you can use text edition commands such as grep, sed, and awk. Freeware software is available for Windows with the same commands, or you can use any batch text editor tool.
The objective is to fully automate this procedure so you can schedule it to run automatically on a regular basis. Make the resulting spreadsheet easy to consult and have it contain only the information that you use frequently. The automated collection and storage of configuration and support data (which is typically more extensive and difficult to use) are described in 9.1.7, “Automated support data collection” on page 319.
9.1.4 Storage documentation
Fully allocate all of the available space in the storage controllers that you use as back-end to the IBM Spectrum Virtualize solution. This way, you can perform all your Disk Storage Management tasks by using IBM Spectrum Virtualize.
You must generate only documentation of your back-end storage controllers manually one time after configuration. Then, you can update the documentation when these controllers receive hardware or code updates. As such, there is little point to automating this back-end storage controller documentation. The same applies to the IBM Spectrum Virtualize internal disk drives and enclosures.
However, if you use split controllers, this option might not be the best one. The portion of your storage controllers that is used outside the IBM Spectrum Virtualize solution might have its configuration changed frequently. In this case, see your back-end storage controller documentation for more information about how to gather and store the information that you need.
9.1.5 Technical Support information
If you must open a technical support incident for your storage and SAN components, create and keep available a spreadsheet with all relevant information for all storage administrators. This spreadsheet should include the following information:
•Hardware information:
 – Vendor, machine and model number, serial number (example: IBM 2145-CF8 S/N 75ABCDE)
 – Configuration, if applicable
 – Current code level
•Physical location:
 – Datacenter, including the complete street address and phone number
 – Equipment physical location, including the room number, floor, tile location, and rack number
 – Vendor’s security access information or procedure, if applicable
 – Onsite person’s contact name and phone or page number
•Support contract information:
 – Vendor contact phone numbers and website
 – Customer’s contact name and phone or page number
 – User ID to the support website, if applicable
Do not store the password in the spreadsheet unless the spreadsheet is password-protected.
 – Support contract number and expiration date
By keeping this data on a spreadsheet, storage administrators have all the information that they need to complete a web support request form or to provide to a vendor’s call support representative. Typically, you are asked first for a brief description of the problem and then asked later for a detailed description and support data collection.
9.1.6 Tracking incident and change tickets
If your organization uses an incident and change management and tracking tool (such as IBM Tivoli Service Request Manager®), you or the storage administration team might need to develop proficiency in its use for several reasons:
•If your storage and SAN equipment are not configured to send SNMP traps to this incident management tool, manually open incidents whenever an error is detected.
•Disk storage allocation and deallocation and SAN zoning configuration modifications should be handled under properly submitted and approved change tickets.
•If you are handling a problem yourself, or calling your vendor’s technical support desk, you might need to produce a list of the changes that you recently implemented in your SAN or that occurred since the documentation reports were last produced or updated.
When you use incident and change management tracking tools, adhere to the following guidelines for IBM Spectrum Virtualize and SAN Storage Administration:
•Whenever possible, configure your storage and SAN equipment to send SNMP traps to the incident monitoring tool so that an incident ticket is automatically opened and the proper alert notifications are sent. If you do not use a monitoring tool in your environment, you might want to configure email alerts that are automatically sent to the cell phones or pagers of the storage administrators on duty or on call.
•Discuss within your organization the risk classification that a storage allocation or deallocation change ticket is to have. These activities are typically safe and nondisruptive to other services and applications when properly handled. However, they have the potential to cause collateral damage if a human error or an unexpected failure occurs during implementation.
Your organization might decide to assume more costs with overtime and limit such activities to off-business hours, weekends, or maintenance windows if they assess that the risks to other critical applications are too high.
•Use templates for your most common change tickets, such as storage allocation or SAN zoning modification, to facilitate and speed up their submission.
•Do not open change tickets in advance to replace failed, redundant, hot-pluggable parts, such as disk drive modules (DDMs) in storage controllers with hot spares, or SFPs in SAN switches or servers with path redundancy. Typically, these fixes do not change anything in your SAN storage topology or configuration, and do not cause any more service disruption or degradation than you already had when the part failed.
Handle these fixes within the associated incident ticket because it might take longer to replace the part if you need to submit, schedule, and approve a non-emergency change ticket.
An exception is if you must interrupt more servers or applications to replace the part. In this case, you must schedule the activity and coordinate support groups. Use good judgment and avoid unnecessary exposure and delays.
•Keep handy the procedures to generate reports of the latest incidents and implemented changes in your SAN Storage environment. Typically, you do not need to periodically generate these reports because your organization probably already has a Problem and Change Management group that runs such reports for trend analysis purposes.
9.1.7 Automated support data collection
In addition to the easier-to-use documentation of your IBM Spectrum Virtualize and SAN Storage environment, collect and store for some time the configuration files and technical support data collection for all your SAN equipment.
For IBM Spectrum Virtualize, this information includes snap data. For other equipment, see the related documentation for more information about how to gather and store the support data that you might need.
You can create procedures that automatically create and store this data on scheduled dates, delete old data, or transfer the data to tape.
9.1.8 Subscribing to IBM Spectrum Virtualize support
Subscribing to IBM Spectrum Virtualize support is probably the most overlooked practice in IT administration, and yet it is the most efficient way to stay ahead of problems. With this subscription, you can receive notifications about potential threats before they can reach you and cause severe service outages.
To subscribe to this support and receive support alerts and notifications for your products, see the following IBM Support website:
If you do not have an IBM ID, create an ID.
You can subscribe to receive information from each vendor of storage and SAN equipment from the IBM website. You can often quickly determine whether an alert or notification is applicable to your SAN storage. Therefore, open them when you receive them and keep them in a folder of your mailbox.
9.2 Storage management users
Almost all organizations have IT security policies that enforce the use of password-protected user IDs when their IT assets and tools are used. However, some storage administrators still use generic, shared IDs, such as superuser, admin, or root, in their management consoles to perform their tasks. They might even use a factory-set default password. Their justification might be a lack of time, forgetfulness, or the fact that their SAN equipment does not support the organization’s authentication tool.
SAN storage equipment management consoles often do not provide access to stored data, but one can easily shut down a shared storage controller and any number of critical applications along with it. Moreover, having individual user IDs set for your storage administrators allows much better backtracking of your modifications if you must analyze your logs.
IBM Spectrum Virtualize supports the following authentication methods:
•Local authentication by using password
•Local authentication by using SSH keys
•Remote authentication using LDAP
•Remote authentication using Tivoli
Regardless of the authentication method you choose, complete the following tasks:
•Create individual user IDs for your Storage Administration staff. Choose user IDs that easily identify the user. Use your organization’s security standards.
•Include each individual user ID into the UserGroup with only enough privileges to perform the required tasks.
•If required, create generic user IDs for your batch tasks, such as Copy Services or Monitoring. Include them in a CopyOperator or Monitor UserGroup. Do not use generic user IDs with the SecurityAdmin privilege in batch tasks.
•Create unique SSH public and private keys for each of your administrators.
•Store your superuser password in a safe location in accordance to your organization’s security guidelines and use it only in emergencies.
9.3 Standard operating procedures
To simplify the SAN storage administration tasks that you use most often (such as SAN storage allocation or removal, or adding or removing a host from the SAN), create step-by-step, predefined standard procedures for them. The following sections provide guidance for keeping your IBM Spectrum Virtualize environment working correctly and reliably.
9.3.1 Allocating and deallocating volumes to hosts
When you allocate and deallocate volumes to hosts, consider the following guidelines:
•Before you allocate new volumes to a server with redundant disk paths, verify that these paths are working well and that the multipath software is free of errors. Fix any disk path errors that you find in your server before you proceed.
•When you plan for future growth of space efficient VDisks, determine whether your server’s operating system supports the particular volume to be extended online. Previous AIX releases, for example, do not support online expansion of rootvg LUNs. Test the procedure in a nonproduction server first.
•Always cross-check the host LUN ID information with the vdisk_UID of the SAN Volume Controller. Do not assume that the operating system recognizes, creates, and numbers the disk devices in the same sequence or with the same numbers as you created them in the SAN Volume Controller/Storwize.
•Ensure that you delete any volume or LUN definition in the server before you unmap it in IBM Spectrum Virtualize. For example, in AIX, remove the hdisk from the volume group (reducevg) and delete the associated hdisk device (rmdev).
•From version 7.4 onwards, consider enabling volume protection by using chsystem vdiskprotectionenabled yes -vdiskprotectiontime <value_in_minutes>. Volume protection ensures that some CLI actions (most of those that either explicitly or implicitly remove host-volume mappings or delete volumes) are policed to prevent the removal of mappings to volumes or deletion of volumes that are considered active. Active means that the system has detected IO activity within the specified time in minutes to the volume from any host.
 
Note: Volume protection cannot be overridden by the use of the -force flag in the affected CLI commands. Volume protection must be disabled to carry on an activity that is currently blocked.
•Ensure that you explicitly remove a volume from any volume-to-host mappings and any copy services relationship to which it belongs before you delete it. At all costs, avoid the use of the -force parameter in rmvdisk. If you issue the svctask rmvdisk command and it still has pending mappings, IBM Spectrum Virtualize prompts you to confirm and is a hint that you might have done something incorrectly.
•When you are deallocating volumes, plan for an interval between unmapping them to hosts (rmvdiskhostmap) and deleting them (rmvdisk). The IBM internal Storage Technical Quality Review Process (STQRP) asks for a minimum of a 48-hour interval so that you can perform a quick backout if you later realize you still need some data in that volume.
9.3.2 Adding and removing hosts
When you add and remove host (or hosts) in IBM Spectrum Virtualize, consider the following guidelines:
•Before you map new servers to IBM Spectrum Virtualize, verify that they are all error free. Fix any errors that you find in your server and IBM Spectrum Virtualize before you proceed. In IBM Spectrum Virtualize, pay special attention to anything inactive in the svcinfo lsfabric command.
•Plan for an interval between updating the zoning in each of your redundant SAN fabrics, such as at least 30 minutes. This interval allows for failover to occur and stabilize, and for you to be notified if unexpected errors occur.
•After you perform the SAN zoning from one server’s HBA to IBM Spectrum Virtualize, you should list its WWPN by using the svcinfo lshbaportcandidate command. Use the svcinfo lsfabric command to certify that it was detected by the IBM Spectrum Virtualize nodes and ports that you expected. When you create the host definition in IBM Spectrum Virtualize (svctask mkhost), try to avoid the -force parameter. If you do not see the host’s WWPNs, it might be necessary to scan fabric from the host. For example, use the cfgmgr command in AIX.
9.4 IBM Spectrum Virtualize code update
Because IBM Spectrum Virtualize might be at the core of your disk and SAN storage environment, its update requires planning, preparation, and verification. However, with the appropriate precautions, an update can be conducted easily and transparently to your servers and applications. This section highlights applicable guidelines for IBM Spectrum Virtualize update.
Most of the following sections explain how to prepare for the IBM Spectrum Virtualize update. The last two sections present version-independent guidelines to update the IBM Spectrum Virtualize system and disk drive.
9.4.1 Current and target IBM Spectrum Virtualize code level
First, determine your current and target IBM Spectrum Virtualize code level. Log in to your IBM Spectrum Virtualize web-based GUI and find the current version. The specific tab to use varies depending on the version itself. Alternatively, if you are using the CLI, run the svcinfo lssystem command.
IBM Spectrum Virtualize code levels are specified by four digits in the format:
•V is the major version number
•R is the release level
•M is the modification level
•F is the fix level
As target, use the latest general availability (GA) IBM Spectrum Virtualize release unless you have a specific reason not to update:
•The specific version of an application or other component of your SAN Storage environment has a known problem or limitation.
•The latest IBM Spectrum Virtualize GA release is not yet cross-certified as compatible with another key component of your SAN storage environment.
•Your organization has mitigating internal policies, such as the use of the “latest minus 1” release, or prompting for “seasoning” in the field before implementation.
For more information, see the following websites:
•Storwize V3700 Concurrent Compatibility and Code Cross-Reference:
•Storwize V5000 Concurrent Compatibility and Code Cross-Reference:
•Storwize V7000 Concurrent Compatibility and Code Cross-Reference:
•SAN Volume Controller Concurrent Compatibility and Code Cross-Reference:
9.4.2 IBM Spectrum Virtualize Upgrade Test Utility
Install and run the latest IBM Spectrum Virtualize Upgrade Test Utility before you update the IBM Spectrum Virtualize code. To download the Upgrade Test Utility, see this website:
This tool verifies the health of your IBM Spectrum Virtualize solution for the update process. It also checks for unfixed errors, degraded MDisks, inactive fabric connections, configuration conflicts, hardware compatibility, disk drives firmware, and many other issues that might otherwise require cross-checking a series of command outputs.
 
Note: The Upgrade Test Utility does not log in storage controllers or SAN switches. Instead, it reports the status of the connections of IBM Spectrum Virtualize to these devices. It is the users’ responsibility to check these components for internal errors.
You can use the GUI or the CLI to install and run the Upgrade Test Utility.
Figure 9-3 shows the Storwize version 7.7 GUI window that is used to install and run the Upgrade Test Utility. It is uploaded and installed like any other software update. The Test Only option is only available from version 7.6 onwards.
Figure 9-3 IBM Spectrum Virtualize Upgrade Test Utility installation using the GUI
Example 9-2 shows how to install and run Upgrade Test Utility in CLI. In this case, the Upgrade Test Utility found warnings and errors and indicates recommended actions.
Example 9-2 Upgrade test by using the CLI
IBM_Storwize:Spectrum_Virtualize_Cluster:superuser>svctask applysoftware -file IBM_INSTALL_svcupgradetest_20.9
CMMVC9001I The package installed successfully.
IBM_Storwize:Spectrum_Virtualize_Cluster:superuser>svcupgradetest -v 7.7.1.2 -d
svcupgradetest version 20.9
 
Please wait, the test may take several minutes to complete.
 
******************* Warning found *******************
 
The upgrade utility has detected that email notifications for error
reporting have either not been configured or that the Call Home function
has not been configured to automatically open a problem record. This may be
caused by an invalid or missing email address. Please review the following
technote to understand the benefits of enabling call home and inventory emails.
http://www.ibm.com/support/docview.wss?rs=591&uid=ssg1S1004537
 
******************* Warning found *******************
 
This tool has found the internal disks of this system are
not running the recommended firmware versions.
Details follow:
 
+-------------+-----------+------------+---------------------------------------+
| Model | Latest FW | Current FW | Drive Info |
+-------------+-----------+------------+---------------------------------------+
| ST9300603SS | B53E | B53B | Drive 2 in slot 19 in enclosure 1 |
| | | | Drive 3 in slot 18 in enclosure 1 |
| HK230041S | 2936 | 291E | Drive 0 in slot 24 in enclosure 1 |
| | | | Drive 1 in slot 23 in enclosure 1 |
+-------------+-----------+------------+---------------------------------------+
 
We recommend that you upgrade the drive microcode at an
appropriate time. If you believe you are running the latest
version of microcode, then check for a later version of this tool.
You do not need to upgrade the drive firmware before starting the
software upgrade.
 
******************** Error found ********************
 
The system identified that one or more drives in the system
are running microcode with a known issue.
 
The following flashes are appropriate for your drives:
 
* http://www.ibm.com/support/docview.wss?rs=591&uid=ssg1S1004327
The following drives are affected by this issue:
0, 1
 
Results of running svcupgradetest:
==================================
 
The tool has found 1 errors and 2 warnings.
9.4.3 IBM Spectrum Virtualize hardware considerations
Before you start the update process, always check whether your IBM Spectrum Virtualize hardware and target code level are compatible.
If part or all your current hardware is not supported at the target code level that you want to update to, replace the unsupported hardware with newer models before you update to the target code level.
Conversely, if you plan to add or replace hardware with new models to an existing cluster, you might have to update your IBM Spectrum Virtualize code first.
9.4.4 Attached hosts preparation
If the appropriate precautions are taken, the IBM Spectrum Virtualize update is not apparent to the attached servers and their applications. The automated update procedure updates one IBM Spectrum Virtualize node at a time, while the other node in the I/O group covers for its designated volumes.
However, to ensure that this feature works, the failover capability of your servers’ multipath software must be working properly. This capability can be mitigated by enabling NPIV if your current code level supports this function. For more information about NPIV, see Chapter 6, “Hosts” on page 229.
Before you start IBM Spectrum Virtualize update preparation, check the following items for every server that is attached to IBM Spectrum Virtualize that you update:
•The operating system type, version, and maintenance or fix level
•The make, model, and microcode version of the HBAs
•The multipath software type, version, and error log
For information about troubleshooting, see these websites (require an IBMid):
•The IBM Support page on SAN Volume Controller Flashes and Alerts (Troubleshooting):
•The IBM Support page on Storwize V7000 Flashes and Alerts (Troubleshooting):
•The IBM Support page on Storwize V5000 Flashes and Alerts (Troubleshooting):
•The IBM Support page on Storwize V3700 Flashes and Alerts (Troubleshooting):
Fix every problem or “suspect” that you find with the disk path failover capability. Because a typical IBM Spectrum virtualize environment has several dozens of servers to a few hundred servers attached to it, a spreadsheet might help you with the Attached Hosts Preparation tracking process.
If you have some host virtualization, such as VMware ESX, AIX LPARs, IBM VIOS, or Solaris containers in your environment, verify the redundancy and failover capability in these virtualization layers.
9.4.5 Storage controllers preparation
As critical as with the attached hosts, the attached storage controllers must correctly handle the failover of MDisk paths. Therefore, they must be running supported microcode versions and their own SAN paths to IBM Spectrum Virtualize must be free of errors.
9.4.6 SAN fabrics preparation
If you are using symmetrical, redundant, independent SAN fabrics, preparing these fabrics for an IBM Spectrum Virtualize update can be safer than hosts or storage controllers. This statement is true assuming that you follow the guideline of a 30-minute minimum interval between the modifications that you perform in one fabric to the next. Even if an unexpected error brings down your entire SAN fabric, the IBM Spectrum Virtualize environment must continue working through the other fabric and your applications must remain unaffected.
Because you are updating your IBM Spectrum Virtualize, also update your SAN switches code to the latest supported level. Start with your principal core switch or director, continue by updating the other core switches, and update the edge switches last. Update one entire fabric (all switches) before you move to the next one so that any problem you might encounter affects only the first fabric. Begin your other fabric update only after you verify that the first fabric update has no problems.
If you are not running symmetrical, redundant independent SAN fabrics, fix this problem as a high priority because it represents a single point of failure (SPOF).
9.4.7 SAN components update sequence
Check the compatibility of your target IBM Spectrum Virtualize code level with all components of your SAN storage environment (SAN switches, storage controllers, server HBAs) and its attached servers (operating systems and eventually, applications).
Applications often certify only the operating system that they run under and leave to the operating system provider the task of certifying its compatibility with attached components (such as SAN storage). However, various applications might use special hardware features or raw devices and certify the attached SAN storage. If you have this situation, consult the compatibility matrix for your application to certify that your IBM Spectrum Virtualize target code level is compatible.
The IBM Spectrum Virtualize Supported Hardware List provides the complete information for using your IBM Spectrum Virtualize SAN storage environment components with the current and target code level. For links to the Supported Hardware List, Device Driver, Firmware, and Recommended Software Levels for different products and different code levels, see the following resources:
•Support Information for SAN Volume Controller:
•Support Information for IBM Storwize V7000:
•Support Information for IBM Storwize V5000:
•Support Information for IBM Storwize V3700:
By cross-checking the version of IBM Spectrum Virtualize is compatible with the versions of your SAN environment components, you can determine which one to update first. By checking a component’s update path, you can determine whether that component requires a multistep update.
If you are not making major version or multistep updates in any components, the following update order is less prone to eventual problems:
1. SAN switches or directors
2. Storage controllers
3. Servers HBAs microcodes and multipath software
4. IBM Spectrum Virtualize system
5. IBM Spectrum Virtualize internal disk drives
 
Attention: Do not update two components of your IBM Spectrum Virtualize SAN storage environment simultaneously, such as the IBM Spectrum Virtualize system and one storage controller. This caution is true even if you intend to do it with your system offline. An update of this type can lead to unpredictable results, and an unexpected problem is much more difficult to debug.
9.4.8 IBM Spectrum Virtualize participating in Metro Mirror or Global Mirror
When you update an IBM Spectrum Virtualize system that participates in an intercluster Copy Services relationship, do not update both clusters in the relationship simultaneously. This situation is not verified or monitored by the automatic update process, and might lead to a loss of synchronization and unavailability.
You must successfully finish the update in one cluster before you start the next one. Try to update the next cluster as soon as possible to the same code level as the first one. Avoid running them with different code levels for extended periods.
 
Note: When you are updating from version 7.1 or earlier to version 7.2 or later, you must stop all Global Mirror (GM) relationships that have their secondary volume on the system that is being updated before starting the update process. This requirement is because of performance improvements in GM code in version 7.2. You can restart these relationships after the update process completes. Other remote copy relationships, such as Metro Mirror (MM) or Global Mirror with Change Volumes (GMCV), do not have to be stopped.
9.4.9 IBM Spectrum Virtualize update
Adhere to the following version-independent guidelines for your IBM Spectrum Virtualize code update:
•Schedule the IBM Spectrum Virtualize code update for a low I/O activity time. The update process puts one node at a time offline. It also disables the write cache in the I/O group that node belongs to until both nodes are updated. Therefore, with lower I/O, you are less likely to notice performance degradation during the update.
•Never power off, reboot, or reset an IBM Spectrum Virtualize node during code update unless you are instructed to do so by IBM Support. Typically, if the update process encounters a problem and fails, it backs out.
•Check whether you are running a web browser type and version that are supported by the IBM Spectrum Virtualize target code level on every computer that you intend to use to manage your IBM Spectrum Virtualize.
•If you are planning for a major IBM Spectrum Virtualize version update, update your current version to its latest fix level before you run the major update.
9.4.10 IBM Spectrum Virtualize disk drive update
Update of disk drive firmware is concurrent whether it is HDD or SSD. However, with SSD, the FPGA level can also be updated. Update of FPGA is not concurrent, so all IOs to the SSDs must be stopped before the update. It is not a problem if SSDs are not yet configured. However, if you have any SSD arrays in storage pools, you must remove SSD MDisks from the pools before the update.
This task can be challenging because removing MDisks from storage pool means migrating all extents from these MDisks to the remaining MDisk in the pool. You cannot remove SSD MDisks from the pool if there is no space left on the remaining MDisks. In such a situation, one option is to migrate some volumes to other storage pools to free enough extents so the SSD MDisk can be removed.
 
Important: More precaution must be taken if you are updating the FPGA on SSD in a 2-tiers hybrid storage pool with Easy Tier running. If the Easy Tier setting on the storage pool has value of auto, Easy Tier switches off after SSD MDisks are removed from that pool, which means it loses all its historical data.
After SSD MDisks are added back to this pool, Easy Tier must start its analysis from the beginning. If you want to avoid such a situation, switch the Easy Tier setting on the storage pool to on. This setting ensures that Easy Tier retains its data after SSD removal.
9.5 SAN modifications
When you administer shared storage environments, human error can occur when a failure is fixed or a change is made that affects one or more servers or applications. That error can then affect other servers or applications because appropriate precautions were not taken.
Human error can include the following examples:
•Disrupting or disabling the working disk paths of a server while trying to fix failed ones.
•Disrupting a neighbor SAN switch port while inserting or pulling out an FC cable or SFP.
•Disabling or removing the working part in a redundant set instead of the failed one.
•Making modifications that affect both parts of a redundant set without an interval that allows for automatic failover during unexpected problems.
Adhere to the following guidelines to perform these actions with assurance:
•Uniquely and correctly identify the components of your SAN.
•Use the proper failover commands to disable only the failed parts.
•Understand which modifications are necessarily disruptive, and which can be performed online with little or no performance degradation.
9.5.1 Cross-referencing HBA WWPNs
With the WWPN of an HBA, you can uniquely identify one server in the SAN. If a server’s name is changed at the operating system level and not at the IBM Spectrum Virtualize host definitions, it continues to access its previously mapped volumes exactly because the WWPN of the HBA did not change.
Alternatively, if the HBA of a server is removed and installed in a second server and the first server’s SAN zones and IBM Spectrum Virtualize host definitions are not updated, the second server can access volumes that it probably should not access.
Complete the following steps to cross-reference HBA WWPNs:
1. In your server, verify the WWPNs of the HBAs that are used for disk access. Typically, you can complete this task by using the SAN disk multipath software of your server. If you are using SDDPCM, run the pcmpath query WWPN command to see output similar to what is shown in Example 9-3.
Example 9-3 Output of the pcmpath query WWPN command
[root@nybixtdb02]> pcmpath query wwpn
Adapter Name PortWWN
fscsi0       10000000C925F5B0
fscsi1       10000000C9266FD1
If you are using server virtualization, verify the WWPNs in the server that is attached to the SAN, such as AIX VIO or VMware ESX.
2. Cross-reference with the output of the IBM Spectrum Virtualize lshost <hostname> command, as shown in Example 9-4.
Example 9-4 Output of the lshost <hostname> command
IBM_2145:svccf8:admin>svcinfo lshost NYBIXTDB02
id 0
name NYBIXTDB02
port_count 2
type generic
mask 1111
iogrp_count 1
WWPN 10000000C925F5B0
node_logged_in_count 2
state active
WWPN 10000000C9266FD1
node_logged_in_count 2
state active
IBM_2145:svccf8:admin>
3. If necessary, cross-reference information with your SAN switches, as shown in Example 9-5. (In Brocade, switches use nodefind <WWPN>.)
Example 9-5 Cross-referencing information with SAN switches
blg32sw1_B64:admin> nodefind 10:00:00:00:C9:25:F5:B0
Local:
Type Pid COS PortName NodeName SCR
N 401000; 2,3;10:00:00:00:C9:25:F5:B0;20:00:00:00:C9:25:F5:B0; 3
Fabric Port Name: 20:10:00:05:1e:04:16:a9
Permanent Port Name: 10:00:00:00:C9:25:F5:B0
Device type: Physical Unknown(initiator/target)
Port Index: 16
Share Area: No
Device Shared in Other AD: No
Redirect: No
Partial: No
Aliases: nybixtdb02_fcs0
b32sw1_B64:admin>
For storage allocation requests that are submitted by the server support team or application support team to the storage administration team, always include the server’s HBA WWPNs to which the new LUNs or volumes are supposed to be mapped. For example, a server might use separate HBAs for disk and tape access, or distribute its mapped LUNs across different HBAs for performance. You cannot assume that any new volume is supposed to be mapped to every WWPN that server logged in the SAN.
If your organization uses a change management tracking tool, perform all your SAN storage allocations under approved change tickets with the servers’ WWPNs listed in the Description and Implementation sessions.
9.5.2 Cross-referencing LUN IDs
Always cross-reference the IBM Spectrum Virtualize vdisk_UID with the server LUN ID before you perform any modifications that involve IBM Spectrum Virtualize volumes. Example 9-6 shows an AIX server that is running SDDPCM. The SAN Volume Controller vdisk_name has no relation to the AIX device name. Also, the first SAN LUN mapped to the server (SCSI_id=0) shows up as hdisk4 in the server because it had four internal disks (hdisk0 - hdisk3).
Example 9-6 Results of running the lshostvdiskmap command
IBM_2145:svccf8:admin>lshostvdiskmap NYBIXTDB03
id name SCSI_id vdisk_id vdisk_name vdisk_UID
0 NYBIXTDB03 0 0 NYBIXTDB03_T01 60050768018205E12000000000000000
IBM_2145:svccf8:admin>
 
 
root@nybixtdb03::/> pcmpath query device
Total Dual Active and Active/Asymmetric Devices : 1
DEV#: 4 DEVICE NAME: hdisk4 TYPE: 2145 ALGORITHM: Load Balance
SERIAL: 60050768018205E12000000000000000
==========================================================================
Path# Adapter/Path Name State Mode Select Errors
0* fscsi0/path0 OPEN NORMAL 7 0
1 fscsi0/path1 OPEN NORMAL 5597 0
2* fscsi2/path2 OPEN NORMAL 8 0
3 fscsi2/path3 OPEN NORMAL 5890 0
If your organization uses a change management tracking tool, include the vdisk_UID and LUN ID information in every change ticket that performs SAN storage allocation or reclaim.
 
Note: Because a host can have many volumes with the same scsi_id, always cross-reference the IBM Spectrum Virtualize volume UID with the host volume UID, and record the scsi_id and LUN ID of that volume.
9.5.3 HBA replacement
Replacing a failed HBA is a fairly trivial and safe operation if it is performed correctly. However, more precautions are required if your server has redundant HBAs and its hardware permits you to “hot” replace it (with the server still running).
Complete the following steps to replace a failed HBA and retain the good HBA:
1. In your server, using the multipath software, identify the failed HBA and record its WWPNs. For more information, see 9.5.1, “Cross-referencing HBA WWPNs” on page 328. Then, place this HBA and its associated paths offline, gracefully if possible. This approach is important so that the multipath software stops trying to recover it. Your server might even show a degraded performance while you perform this task.
2. Some HBAs have a label that shows the WWPNs. If you have this type of label, record the WWPNs before you install the new HBA in the server.
3. If your server does not support HBA hot-swap, power off your system, replace the HBA, connect the used FC cable into the new HBA, and power on the system.
If your server does support hot-swap, follow the appropriate procedures to perform a “hot” replace of the HBA. Do not disable or disrupt the good HBA in the process.
4. Verify that the new HBA successfully logged in to the SAN switch. If it logged in successfully, you can see its WWPNs logged in to the SAN switch port.
Otherwise, fix this issue before you continue to the next step.
Cross-check the WWPNs that you see in the SAN switch with the one you noted in step 1, and make sure that you did not get the WWNN mistakenly.
5. In your SAN zoning configuration tool, replace the old HBA WWPNs for the new ones in every alias and zone to which they belong. Do not touch the other SAN fabric (the one with the good HBA) while you perform this task.
Only one alias should use each WWPN, and zones must reference this alias.
If you are using SAN port zoning (though you should not be) and you did not move the new HBA FC cable to another SAN switch port, you do not need to reconfigure zoning.
6. Verify that the new HBA’s WWPNs appear in the IBM Spectrum Virtualize system by using the lsfcportcandidate command.
If the WWPNs of the new HBA do not appear, troubleshoot your SAN connections and zoning.
7. Add the WWPNs of this new HBA in the IBM Spectrum Virtualize host definition by using the addhostport command. Do not remove the old one yet. Run the lshost <servername> command. Then, verify that the good HBA shows as active, while the failed and old HBA should show as inactive or offline.
8. Return to the server, and reconfigure the multipath software to recognize the new HBA and its associated SAN disk paths. Certify that all SAN LUNs have redundant disk paths through the good and the new HBAs.
9. Return to the IBM Spectrum Virtualize system and verify again (by using the lshost <servername> command) that both the good and the new HBA’s WWPNs are active. In this case, you can remove the old HBA WWPNs from the host definition by using the rmhostport command.
Do not remove any HBA WWPNs from the host definition until you ensure that you have at least two active ones that are working correctly.
By following these steps, you avoid removing your only good HBA by mistake.
9.6 Hardware upgrades for IBM Spectrum Virtualize
The IBM Spectrum Virtualize scalability features allow significant flexibility in its configuration. As a consequence, several scenarios are possible for its growth. The following sections describe these processes:
9.6.1 Adding IBM Spectrum Virtualize nodes to an existing cluster
If your existing IBM Spectrum Virtualize cluster is below the maximum I/O groups limit for your specific product and you intend to upgrade it, you might find yourself installing newer SAN Volume Controller nodes or Storwize control enclosures that are more powerful than your existing ones. Therefore, your cluster will have different node models in different I/O groups.
To install these newer nodes, determine whether you need to upgrade your IBM Spectrum Virtualize code level first. For more information, see 9.4.3, “IBM Spectrum Virtualize hardware considerations” on page 324.
After you install the newer nodes, you might need to redistribute your servers across the I/O groups. Consider the following points:
•Moving a server’s volume to different I/O groups can be done online because of a feature called Non-Disruptive Volume Movement (NDVM), which was introduced in version 6.4 of IBM Spectrum Virtualize. Although this process can be done without stopping the host, careful planning and preparation are advised.
 
Note: You cannot move a volume that is in any type of remote copy relationship.
•If each of your servers is zoned to only one I/O group, modify your SAN zoning configuration as you move its volumes to another I/O group. As best you can, balance the distribution of your servers across I/O groups according to I/O workload.
•Use the -iogrp parameter in the mkhost command to define which I/O groups of IBM Spectrum Virtualize that the new servers will use. Otherwise, IBM Spectrum Virtualize maps by default the host to all I/O groups, even if they do not exist and regardless of your zoning configuration. Example 9-7 shows this scenario and how to resolve it by using the rmhostiogrp and addhostiogrp commands.
Example 9-7 Mapping the host to I/O groups
IBM_2145:svccf8:admin>lshost NYBIXTDB02
id 0
name NYBIXTDB02
port_count 2
type generic
mask 1111
iogrp_count 4
WWPN 10000000C9648274
node_logged_in_count 2
state active
WWPN 10000000C96470CE
node_logged_in_count 2
state active
IBM_2145:svccf8:admin>lsiogrp
id name node_count vdisk_count host_count
0 io_grp0 2 32 1
1 io_grp1 0 0 1
2 io_grp2 0 0 1
3 io_grp3 0 0 1
4 recovery_io_grp 0 0 0
IBM_2145:svccf8:admin>lshostiogrp NYBIXTDB02
id name
0 io_grp0
1 io_grp1
2 io_grp2
3 io_grp3
IBM_2145:svccf8:admin>rmhostiogrp -iogrp 1:2:3 NYBIXTDB02
IBM_2145:svccf8:admin>lshostiogrp NYBIXTDB02
id name
0 io_grp0
IBM_2145:svccf8:admin>lsiogrp
id name node_count vdisk_count host_count
0 io_grp0 2 32 1
1 io_grp1 0 0 0
2 io_grp2 0 0 0
3 io_grp3 0 0 0
4 recovery_io_grp 0 0 0
IBM_2145:svccf8:admin>addhostiogrp -iogrp 3 NYBIXTDB02
IBM_2145:svccf8:admin>lshostiogrp NYBIXTDB02
id name
0 io_grp0
3 io_grp3
IBM_2145:svccf8:admin>lsiogrp
id name node_count vdisk_count host_count
0 io_grp0 2 32 1
1 io_grp1 0 0 0
2 io_grp2 0 0 0
3 io_grp3 0 0 1
4 recovery_io_grp 0 0 0
•If possible, avoid setting a server to use volumes from different I/O groups that have different node types for extended periods of time. Otherwise, as this server’s storage capacity grows, you might experience a performance difference between volumes from different I/O groups. This mismatch makes it difficult to identify and resolve eventual performance problems.
9.6.2 Upgrading IBM Spectrum Virtualize nodes in an existing cluster
If you are replacing the nodes of your existing SAN Volume Controller cluster with newer ones, the replacement procedure can be performed nondisruptively. The new node can assume the WWNN of the node you are replacing, which requires no changes in host configuration, SAN zoning, or multipath software. For more information about this procedure, see SAN Volume Controller at IBM Knowledge Center for your current code level:
From version 7.8, IBM also offers the following Storwize node canisters upgrade options:
•From Storwize V5010 to Storwize V5020
•From Storwize V5010 to Storwize V5030
•From Storwize V5020 to Storwize V5030
•From Storwize V7000 Gen2 to Storwize V7000 Gen2+
The new node canister assumes the WWNN of the node you are replacing automatically, which requires no changes in host configuration, SAN zoning, or multipath software. For more information about this procedure, see IBM Knowledge Center for your product and current code level at these websites:
•Storwize V5000
•Storwize V7000
Nondisruptive node replacement uses failover capabilities to replace one node in an I/O group at a time. If a new node has a different version of IBM Spectrum Virtualize code, it installs the cluster version automatically during the node replacement procedure.
9.6.3 Moving to a new IBM Spectrum Virtualize cluster
You might have a highly populated, intensively used IBM Spectrum Virtualize cluster that you want to upgrade. You might also want to use the opportunity to overhaul your IBM Spectrum Virtualize and SAN storage environment.
Complete the following steps to replace your cluster entirely with a newer, bigger, and more powerful one:
1. Install your new IBM Spectrum Virtualize cluster.
2. Create a replica of your data in your new cluster.
3. Migrate your servers to the new IBM Spectrum Virtualize cluster when convenient.
If your servers can tolerate a brief, scheduled outage to switch from one IBM Spectrum Virtualize cluster to another, you can use IBM Spectrum Virtualize’s remote copy services (Metro Mirror or Global Mirror) to create your data replicas, following these steps:
1. Select a host that you want to move to the new IBM Spectrum Virtualize cluster and find all the old volumes you must move.
2. Zone your host to the new IBM Spectrum Virtualize cluster.
3. Create remote copy relationships from the old volumes in the old Spectrum Virtualize cluster to new volumes in the new Spectrum Virtualize cluster.
4. Map the new volumes from the new Spectrum Virtualize cluster to the host.
5. Discover new volumes on the host.
6. Stop all I/O from the host to the old volumes from the old Spectrum Virtualize cluster.
7. Disconnect and remove the old volumes on the host from the old Spectrum Virtualize cluster.
8. Unmap the old volumes from the old Spectrum Virtualize cluster to the host.
9. Make sure remote copy relationships between old and new volumes in the old and new Spectrum Virtualize cluster are synced.
10. Stop and remove remote copy relations between old and new volumes so that the target volumes in the new Spectrum Virtualize cluster receive read/write access.
11. Import data from the new volumes and start your applications on the host.
If you must migrate a server online, instead, you must use host-based mirroring by completing these steps:
1. Select a host that you want to move to the new Spectrum Virtualize cluster and find all the old volumes that you must move.
2. Zone your host to the new Spectrum Virtualize cluster.
3. Create volumes in the new Spectrum Virtualize cluster of the same size as the old volumes in the old Spectrum Virtualize cluster.
4. Map the new volumes from the new Spectrum Virtualize cluster to the host.
5. Discover new volumes on the host.
6. For each old volume, use host-based mirroring (such as AIX mirrorvg) to move your data to the corresponding new volume.
7. For each old volume, after the mirroring is complete, remove the old volume from the mirroring group.
8. Disconnect and remove the old volumes on the host from the old Spectrum Virtualize cluster.
9. Unmap the old volumes from the old Spectrum Virtualize cluster to the host.
This approach uses the server’s computing resources (CPU, memory, and I/O) to replicate the data. It can be done online if properly planned. Before you begin, make sure it has enough spare resources.
The biggest benefit to using either approach is that they easily accommodate (if necessary) the replacement of your SAN switches or your back-end storage controllers. You can upgrade the capacity of your back-end storage controllers or replace them entirely, as you can replace your SAN switches with bigger or faster ones. However, you do need to have spare resources, such as floor space, power, cables, and storage capacity, available during the migration.
9.6.4 Splitting a Spectrum Virtualize cluster
Splitting a Spectrum Virtualize cluster might become a necessity if you have one or more of the following requirements:
•To grow the environment beyond the maximum number of I/O groups that a clustered system can support
•To grow the environment beyond the maximum number of attachable subsystem storage controllers
•To grow the environment beyond any other maximum system limit
•To achieve new levels of data redundancy and availability
By splitting the clustered system, you no longer have one IBM Spectrum Virtualize system that handles all I/O operations, hosts, and subsystem storage attachments. The goal is to create a second IBM Spectrum Virtualize system so that you can equally distribute the workload over the two systems.
After safely removing nodes from the existing cluster and creating a second IBM Spectrum Virtualize system, choose from the following approaches to balance the two systems:
•Attach new storage subsystems and hosts to the new system, and start putting only new workload on the new system.
•Migrate the workload onto the new system by using the approach described in 9.6.3, “Moving to a new IBM Spectrum Virtualize cluster” on page 334.
It is uncommon to reduce the number of I/O groups. It can happen when you replace old nodes with new more powerful ones. It can also occur in a remote partnership when more bandwidth is required on one site and spare bandwidth is on the other site.
9.7 Adding expansion enclosures
If you plan well, you can buy an IBM Spectrum Virtualize product with enough internal storage to run your business for some time. But as time passes and your environment grows, you will need to add more storage to your system.
Depending on the IBM Spectrum Virtualize product the code level that you have installed, you can add different numbers of expansion enclosures to your system. Because all IBM Spectrum Virtualize systems were designed to make managing and maintaining them as simple as possible, adding an expansion enclosure is an easy task. However, here are some guidance and preferred practices you should follow.
At the time of writing, the following IBM Spectrum Virtualize products only support one chain of expansion enclosures:
•Storwize V3500
•Storwize V3700
•Storwize V5010
•Storwize V5020
New expansion enclosures should be added at the bottom of the chain as long as the limit of enclosures for the product has not been reached.
These other IBM Spectrum Virtualize products support two chains of expansion enclosures:
•Storwize V5000
•Storwize V5030
•Storwize V7000 (Gen1, Gen2, Gen2+)
•FlashSystem V9000 (with SAS expansion option)
•SAN Volume Controller (with SAS expansion option)
As a preferred practice, the number of expansion enclosures should be balanced between both chains. This guideline means that the number of expansion enclosures in every chain cannot differ by more than one. For example, having five expansion enclosures in the first chain and only one in the second chain is incorrect.
 
Note: When counting the number of enclosures in a chain, remember that for Storwize V7000 Gen1 and Storwize V5000 Gen1, the control enclosure is part of the second chain of expansions.
Adding expansion enclosures is simplified because Storwize can automatically discover new expansion enclosures after the SAS cables are connected. It is possible to manage and use the new disk drives without managing the new expansion enclosures. However, unmanaged expansion enclosures are not monitored properly. This issue can lead to more difficult troubleshooting and can make problem resolution take longer. To avoid this situation, always manage newly added expansion enclosures.
Because of internal architecture and classical disk latency, it does not matter in which enclosure SAS or NL-SAS drives are placed. However, if you have some SSD drives and you want to use them in the most efficient way, place them in the control enclosure or in the first expansion enclosures in chains. This configuration ensures every I/O to SSD disk drives travel the shortest possible way through the internal SAS fabric.
 
Note: This configuration is even more important on Storwize V7000 Gen2 and Storwize Gen2+ because the drives in the control enclosure have double the bandwidth available compared to expansion enclosures and should be used for SSD drives if there are any in the system.
9.8 IBM Spectrum Virtualize scripting
Although the IBM Spectrum Virtualize GUI is a powerful interface (similar to other GUIs), it is not well-suited to perform large numbers of specific operations. For complex, often-repeated operations, it is more convenient to use the IBM Spectrum Virtualize CLI interface. CLI commands can be scripted by using any program that can pass text commands to the IBM Spectrum Virtualize Secure Shell (SSH) connection.
On UNIX systems, you can use the ssh command to create an SSH connection with an IBM Spectrum Virtualize system. On Windows systems, you can use the plink.exe utility (which is provided with the PuTTY tool) to achieve the same.
Create an IBM Spectrum Virtualize user with the the lowest level privileges required to access and perform batch operations on the system. Do not grant it Administrator privileges unless strictly necessary. Create and configure an SSH key specifically for it.
9.8.1 Connecting to IBM Spectrum Virtualize using predefined PuTTY
The easiest way to create an SSH connection to the SAN Volume Controller is when the plink.exe utility can call a predefined PuTTY session. When you define a session, include the following information:
•The auto-login user name, which you set to the user created specifically to perform batch operations. To set this parameter, click Connection → Data in the left pane of the PuTTY Configuration window, as shown in Figure 9-4.
 
Note: In Figure 9-4, we use the admin user but you should try to contain the privileges of the user dedicated at batch operations to the bare minimum.
Figure 9-4 Configuring the auto-login user name
•The private key for authentication (for example, icat.ppk), which is the private key that you created. To set this parameter, select Connection → SSH → Auth in the left pane of the PuTTY Configuration window, as shown in Figure 9-5.
Figure 9-5 Configuring the SSH private key
•The IP address of the IBM Spectrum Virtualize system. To set this parameter, select Session at the top left of the PuTTY Configuration window, as shown in Figure 9-6.
Figure 9-6 Specifying the IP address
When you are specifying the basic options for your PuTTY session, you need to set a session name, which in this example is redbook_CF8. To use the predefined PuTTY session, use the following syntax:
plink redbook_CF8
If you do not use a predefined PuTTY session, use the following syntax:
plink admin@<your cluster ip address> -i "C:DirectoryPathKeyName.PPK"
9.8.2 Run commands in the IBM Spectrum Virtualize shell
You can run various limited scripts directly in the restricted IBM Spectrum Virtualize shell. Example 9-8 show a script to restart Global Mirror relationships and groups.
Example 9-8 Restarting Global Mirror relationships and groups
svcinfo lsrcconsistgrp -filtervalue state=consistent_stopped -nohdr |
while read id name unused
do
echo "Restarting group: $name ($id)"
    svctask startrcconsistgrp -force $name
done
svcinfo lsrcrelationship -filtervalue state=consistent_stopped -nohdr |
while read id name master_cluster_id master_cluster_name master_vdisk_id
master_vdisk_name aux_cluster_id aux_cluster_name aux_vdisk_id
aux_vdisk_name primary consistency_group_id junk
do
if [ "$consistency_group_id" == "" ]; then
echo "Restarting relationship: $name ($id)"
svctask startrcrelationship -force $name
fi
done
9.8.3 Scripting toolkit
For more elaborate scripts, the restricted IBM Spectrum Virtualize shell might not be powerful enough. In those cases, perform the processing on the system that is connected to the IBM Spectrum Virtualize shell. The shell should only be used to run single commands.
IBM engineers developed a scripting toolkit that helps to automate IBM Spectrum Virtualize operations. This scripting toolkit is based on Perl and is available at no fee from the following IBM alphaWorks® website:
 
 
Attention: The scripting toolkit is available to users through the IBM alphaWorks website. As with all software that is available on the alphaWorks site, this toolkit was not extensively tested and is provided on an as-is basis. Because the toolkit is not supported in any formal way by IBM Product Support, use it at your own risk.
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.218.81.166