Converting a local PowerHA Standard Edition cluster to a PowerHA Enterprise Edition cluster
In this chapter, we convert a local PowerHA Standard Edition (SE) cluster to a two-site disaster recovery cluster utilizing PowerHA Enterprise Edition (EE). It is always recommended to perform these steps in a test environment before ever performing in production.
This chapter covers the following topics:
The details of each step are well documented in previous PowerHA/EE IBM Redbooks publications. However, we provide a very specific example in the following sections.
5.1 Test environment overview
For our example, we start off with a local two-node cluster with a single IP network and shared storage as shown in Figure 5-1.
Though not shown, dual VIOS servers are utilized within each server. In our case, the storage is presented to the clients through VIOS via shared storage pools. There was no specific technical requirement for this but was requested of us since there is no other existing PowerHA documentation examples showing their use.
We also use Logical Volume Manager (LVM) mirroring across two logical sites. Our environment was not prototypical of what we would consider best practices for production environments. This is mainly because of physical resource limitations. The main idea was to demonstrate that potentially an existing cross-site LVM mirror configuration could be expanded to include Geographic Logical Volume Manager (GLVM) for a third copy. Also, GLVM was the most readily available option for us to demonstrate converting a local cluster to a dual site cluster as it does not require specialized storage to do so.
Figure 5-1 Test environment beginning configuration
The details of the cluster topology configuration are shown in Figure 5-2 on page 83. Since this is a PowerHA v7.1.3 cluster, we utilize unicast instead of the previously required multicast as the primary heartbeat mechanism.
|
[root] / # cltopinfo
Cluster Name: PHASEtoEE
Cluster Type: Stretched
Heartbeat Type: Unicast
Repository Disk: hdisk1 (00f6f5d015a4310b)
Cluster Nodes:
Site 1 (Dallas):
Jessica
Site 2 (FortWorth):
Cassidy
There are 2 node(s) and 1 network(s) defined
NODE Cassidy:
Network net_ether_01
dallasserv 10.10.10.51
cassidy 192.168.100.52
NODE Jessica:
Network net_ether_01
dallasserv 10.10.10.51
jessica 192.168.100.51
Resource Group xsiteRG
Startup Policy Online On Home Node Only
Fallover Policy Fallover To Next Priority Node In The List
Fallback Policy Never Fallback
Participating Nodes Jessica Cassidy
Service IP Label dallasserv
|
Figure 5-2 Test environment beginning topology
The cluster is set up as a two-node (Jessica and Cassidy) single resource group (xsiteRG) LVM mirrored hot-standby configuration. There is only one service IP (dallasserv) address and an application server (banner). Though sites are currently defined, the member nodes assigned to each site will change during this test scenario
Ultimately, the cluster will change to include a third node (Shanley), and it solely will reside in the second site fortworth utilizing GLVM for the data replication across sites. The second local node (Cassidy) will logically become part of the dallas site. No additional resource groups will be added into the configuration.
The cluster resources and resource group defined are shown in Figure 5-3 on page 84.
|
[root] / # clshowres
Resource Group Name xsiteRG
Participating Node Name(s) Jessica Cassidy
Startup Policy Online On Home Node Only
Fallover Policy Fallover To Next Priority Node In The List
Fallback Policy Never Fallback
Site Relationship ignore
Dynamic Node Priority
Service IP Label dallasserv
Filesystems ALL
Filesystems Consistency Check fsck
Filesystems Recovery Method sequential
Filesystems/Directories to be exported (NFSv2/NFSv3)
Filesystems/Directories to be exported (NFSv4)
Filesystems to be NFS mounted
Network For NFS Mount
Filesystem/Directory for NFSv4 Stable Storage
Volume Groups amyvg
Concurrent Volume Groups
Use forced varyon for volume groups, if necessary true
Disks
Raw Disks
Disk Error Management? no
GMVG Replicated Resources
GMD Replicated Resources
PPRC Replicated Resources
SVC PPRC Replicated Resources
EMC SRDF▒ Replicated Resources
TRUECOPY Replicated Resources
GENERIC XD Replicated Resources
Connections Services
Fast Connect Services
Shared Tape Resources
Application Servers banner
Highly Available Communication Links
Primary Workload Manager Class
Secondary Workload Manager Class
Delayed Fallback Timer
Miscellaneous Data
Automatically Import Volume Groups false
Inactive Takeover
SSA Disk Fencing false
Filesystems mounted before IP configured true
WPAR Name
|
Figure 5-3 Beginning cluster resource group configuration
5.2 Install PowerHA Enterprise Edition software
In our scenario, only PowerHA Standard Edition is installed on the two local nodes. It is necessary to install PowerHA Enterprise Edition on both local nodes, along with the new remote node.
When installing, you can choose to install all, or only those filesets specific to your environment. The IBM PowerHA SystemMirror Enterprise Edition requires the installation and acceptance of license agreements for both the Standard Edition cluster.license fileset and the Enterprise Edition cluster.xd.license fileset as shown in Table 5-1, in order for the remainder of the filesets to install.
Table 5-1 PowerHA Enterprise Edition required license filesets
|
Required package
|
Filesets to install
|
|
Standard Edition license
|
cluster.license
|
|
Enterprise Edition license
|
cluster.xd.license
|
The base filesets in the Standard Edition are required to install the Enterprise Edition filesets. The Enterprise package levels must match those of the base runtime level (cluster.es.server.rte). Table 5-2 displays the itemized list of filesets for each of the integrated offerings.
Table 5-2 PowerHA Enterprise Edition - integrated offering solution filesets
|
Replication type
|
Fileset to install
|
|
ESS-Direct Management PPRC
|
cluster.es.pprc.rte
cluster.es.pprc.cmds
cluster.msg.en_US.pprc
|
|
ESS/DS6000/DS8000 Metro Mirror (DSCLI PPRC)
|
cluster.es.spprc.cmds
cluster.es.spprc.rte
cluster.es.cgpprc.cmds
cluster.es.cgpprc.rte
cluster.msg.en_US.cgpprc
|
|
SAN Volume Controller (SVC)
Storwize V7000
|
cluster.es.svcpprc.cmds
cluster.es.svcpprc.rte
cluster.msg.en_US.svcpprc
|
|
XIV, DS8800 in-band and IBM HyperSwap®, DS8700/DS8800 Global Mirror
|
cluster.es.genxd.cmds
cluster.es.genxd.rte
cluster.msg.en_US.genxd
|
|
Geographic Logical Volume Mirroring
|
cluster.doc.en_US.glvm.pdf
cluster.msg.en_US.glvm
cluster.xd.glvm
glvm.rpv.client
glvm.rpv.man.en_US
glvm.rpv.msg.en_US
glvm.rpv.server
glvm.rpv.util
|
|
EMC SRDF
|
cluster.es.sr.cmds
cluster.es.sr.rte
cluster.msg.en_US.sr
|
|
Hitachi TrueCopy/Universal Replicator
HP Continuous Access
|
cluster.es.tc.cmds
cluster.es.tc.rte
cluster.msg.en_US.tc
|
In our scenario, we only needed the base, license, and glvm filesets as shown in Figure 5-4.
|
COMMAND STATUS
Command: OK stdout: yes stderr: no
Before command completion, additional instructions may appear below.
[TOP]
geninstall -I "a -cgNqwXY -J" -Z -d . -f File 2>&1
File:
I:cluster.xd.base 7.1.3.0
I:cluster.xd.glvm 7.1.3.0
I:cluster.xd.license 7.1.3.0
I:glvm.rpv.client 7.1.3.0
I:glvm.rpv.msg.en_US 7.1.3.0
I:glvm.rpv.server 7.1.3.0
I:glvm.rpv.util 7.1.3.0
[MORE...152]
F1=Help F2=Refresh F3=Cancel F6=Command
F8=Image F9=Shell F10=Exit /=Find
|
Figure 5-4 PowerHA Enterprise Edition installation
Upon completion, we then applied SP1 and ended up with the levels shown in Figure 5-5 on page 87.
If you are utilizing storage-specific replication, it may be required to install other additional software to accommodate it with PowerHA/EE. It is also possible that you will need additional IP network connectivity to the storage. For more information, consult the Storage-based high availability and disaster recovery for PowerHA SystemMirror Enterprise Edition guide at:
A unique attribute in PowerHA v7, contributed by Cluster Aware AIX (CAA), is the cluster type definition. The types consist of Standard (or Flat) when no sites are used, and Stretched and Linked for when sites are utilized. Unfortunately, there currently is no way to change from a standard or stretched cluster to a linked cluster without deleting and re-creating the cluster. The primary reason for this is the underlying existing CAA cluster does not allow it.
|
bos.cluster.rte 7.1.3.2 C F Cluster Aware AIX
cluster.es.client.clcomd 7.1.3.1 C F Cluster Communication
cluster.es.client.lib 7.1.3.1 C F PowerHA SystemMirror Client
cluster.es.client.rte 7.1.3.1 C F PowerHA SystemMirror Client
cluster.es.client.utils 7.1.3.0 C F PowerHA SystemMirror Client
cluster.es.cspoc.cmds 7.1.3.1 C F CSPOC Commands
cluster.es.cspoc.rte 7.1.3.1 C F CSPOC Runtime Commands
cluster.es.migcheck 7.1.3.0 C F PowerHA SystemMirror Migration
cluster.es.server.diag 7.1.3.1 C F Server Diags
cluster.es.server.events 7.1.3.1 C F Server Events
cluster.es.server.rte 7.1.3.1 C F Base Server Runtime
cluster.es.server.testtool
cluster.es.server.utils 7.1.3.1 C F Server Utilities
cluster.license 7.1.3.0 C F PowerHA SystemMirror
cluster.man.en_US.es.data 7.1.3.1 C F Man Pages - U.S. English
cluster.msg.en_US.es.client
cluster.msg.en_US.es.server
cluster.xd.base 7.1.3.0 C F PowerHA SystemMirror
cluster.xd.glvm 7.1.3.1 C F PowerHA SystemMirror
cluster.xd.license 7.1.3.0 C F PowerHA SystemMirror
glvm.rpv.client 7.1.3.1 C F Remote Physical Volume Client
glvm.rpv.msg.en_US 7.1.3.0 C F RPV Messages - U.S. English
glvm.rpv.server 7.1.3.1 C F Remote Physical Volume Server
glvm.rpv.util 7.1.3.0 C F Geographic LVM Utilities
|
Figure 5-5 PowerHA/EE filesets with SP1 installed
5.3 Delete existing cluster
Deleting the existing cluster is an unfortunate necessary step as CAA does not support dynamically changing between stretched and linked clusters. This obviously cannot be done with an active cluster. We recommend making a snapshot of the cluster before deleting to be referenced later if needed.
To remove the existing cluster this, like most operations, can be accomplished either via clmgr or through SMIT. We will demonstrate both.
We first execute smitty sysmirror → Cluster Nodes and Networks → Manage the Cluster → Remove the Cluster Definition. We choose yes to remove the cluster definition across all nodes as shown in Figure 5-6. This will also remove the underlying CAA cluster.
|
Remove the Cluster Definition
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
Cluster Name PHASEtoEE
* Remove definition from all nodes [Yes] +
NOTE: All user-configured cluster information
WILL BE DELETED by this operation.
|
Figure 5-6 Delete cluster via SMIT
The clmgr equivalent of deleting the cluster is shown in Example 5-1.
Example 5-1 Delete cluster via clmgr
[jessica:root] / # clmgr delete cluster PHASEtoEE
One or more deletion operations have been detected.
Proceed with the deletion(s)? (y|n) y
Ensuring that the following nodes are offline: cassidy, jessica
Warning: cluster services are already offline on node "cassidy" (state is
"ST_INIT"). Removing that node from the shutdown list.
Warning: cluster services are already offline on node "jessica" (state is
"ST_INIT"). Removing that node from the shutdown list.
Attempting to delete node "cassidy" from the cluster...
Attempting to remove the CAA cluster from "jessica"...
Attempting to delete node "jessica" from the cluster...
5.4 Creating a three node two-site GLVM linked cluster
Before creating the cluster, ensure that all nodes meet the following bare minimum requirements:
– IP interfaces must be configured with addresses
– All IPs used for cluster must be defined in /etc/hosts on each node
– Hostname IP entries for all nodes must be in /etc/cluster/rhosts on each node
– clcomd must be active
– A free disk is available, with PVID, at each site for repository disk
For our scenario, we configured an additional interface and network to each of our existing cluster nodes, along with both interfaces and networks to the new remote node. The IP configuration of each system before adding them into the cluster configuration is shown in Example 5-2.
Also, both of our networks are accessible by all nodes in the cluster. This often may not be the case. More typically, the nodes may be attached to a LAN within a site and also another WAN between sites. Of course, it is imperative that site communications work properly, are of adequate bandwidth, and preferably multiple diversely routed physical links.
Example 5-2 IP interface configuration
[jessica:root] / # netstat -in
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en0 1500 link#2 ee.af.1.71.78.2 637436 0 600984 0 0
en0 1500 10.10.8 10.10.10.51 637436 0 600984 0 0
en0 1500 192.168.100 192.168.100.51 637436 0 600984 0 0
en1 1500 link#3 ee.af.1.71.78.3 36 0 11 0 0
en1 1500 192.168.148 192.168.150.51 36 0 11 0 0
[cassidy:root] /# netstat -in
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en0 1500 link#2 7a.40.c8.b3.15.2 663449 0 687192 0 0
en0 1500 192.168.100 192.168.100.52 663449 0 687192 0 0
en1 1500 link#3 7a.40.c8.b3.15.3 27 0 11 0 0
en1 1500 192.168.148 192.168.150.52 27 0 11 0 0
[shanley:root] /# netstat -in
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en0 1500 link#2 6e.8d.d0.21.d0.2 38633 0 21121 0 0
en0 1500 192.168.100 192.168.100.53 38633 0 21121 0 0
en1 1500 link#3 6e.8d.d0.21.d0.3 21 0 10 0 0
en1 1500 192.168.148 192.168.150.53 21 0 10 0 0
The contents of our /etc/hosts and /etc/cluster/rhosts entries are shown in Example 5-3, along with the clcomd status.
Example 5-3 Hosts and rhosts file contents and clcomd status
[jessica:root] / #cat /etc/hosts
192.168.100.21 phat01
192.168.100.40 nimres2
#net_ether_01
192.168.100.51 jessica
192.168.100.52 cassidy
192.168.100.53 shanley
#XD_data
192.168.150.51 jessica_xd
192.168.150.52 cassidy_xd
192.168.150.53 shanley_xd
#Site Service Addresses
10.10.10.51 dallasserv
10.10.10.52 ftwserv
[jessica:root] / # cat /etc/cluster/rhosts
192.168.100.51
192.168.100.52
192.168.100.53
[jessica:root] / #lssrc -s clcomd
Subsystem Group PID Status
clcomd caa 5308598 active
Upon validating that the previous steps were completed, we can begin configuring the cluster.
5.4.1 Define linked cluster with sites
We begin by executing smitty sysmirror → Cluster Nodes and Networks → Multi Site Cluster Deployment → Setup a Cluster, Nodes and Networks. The options were configured as shown in Figure 5-7 on page 90.
|
Setup Cluster, Sites, Nodes and Networks
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Cluster Name [PHASEtoEE]
* Site 1 Name [dallas]
* New Nodes (via selected communication paths) [jessica cassidy] +
* Site 2 Name [fortworth]
* New Nodes (via selected communication paths) [shanley] +
Cluster Type [Linked Cluster] +
|
Figure 5-7 Add a multi-site cluster via SMIT menu
Upon execution, you will see the discovery process for IP interfaces and disks run against the nodes. This will, if possible, automatically add the interfaces into the cluster topology. By default, the cluster software associates the interfaces with PowerHA networks of type ether. In order to change the default names and the type of a network, we used SMIT menus by executing smitty sysmirror → Cluster Nodes and Networks → Manage Networks and Network Interfaces → Networks → Change/Show a Network. The resulting network and interface definitions are shown in Example 5-4.
In all site configurations, it is common to have an XD_ip network defined. However it is mostly the same as type ether with only different default heartbeat parameters and subnet restrictions removed. You can have both ether networks within a site and XD_ip network across. The XD_data network is unique to GLVM and specifies which network is to be utilized for GLVM data replication traffic. Though it is not required to be a dedicated network, it is recommended.
In our scenario, we only have two networks and they do actually both span the sites, which may not be common overall. Because of this, we could have chosen to keep one network as type ether. However, we thought it was best to show the usage of an XD_ip instead.
Example 5-4 Network topology
[shanley:root] / # cllsif
Adapter Type Network Net Type Attribute Node IP Address
cassidy_xd boot GLVMnet XD_data public cassidy 192.168.150.52
cassidy boot prodnet XD_ip public cassidy 192.168.100.52
dallasserv service prodnet XD_ip public cassidy 10.10.10.51
ftwserv service prodnet XD_ip public cassidy 10.10.10.52
jessica_xd boot GLVMnet XD_data public jessica 192.168.150.51
jessica boot prodnet XD_ip public jessica 192.168.100.51
dallasserv service prodnet XD_ip public jessica 10.10.10.51
ftwserv service prodnet XD_ip public jessica 10.10.10.52
shanley_xd boot GLVMnet XD_data public shanley 192.168.150.53
shanley boot prodnet XD_ip public shanley 192.168.100.53
dallasserv service prodnet XD_ip public shanley 10.10.10.51
ftwserv service prodnet XD_ip public shanley 10.10.10.52
In the next step, we defined the repository disks for the sites. We defined both of them in a single SMIT panel by executing smitty sysmirror → Cluster Nodes and Networks → Multi Site Cluster Deployment → Define Repository Disk and Cluster IP Address. We selected the candidate disks for the CAA repository in each site, hdisk1 for dallas, and hdisk2 in fortworth. We used the default heartbeat mechanism of unicast as shown in Figure 5-8.
|
Multi Site with Linked Clusters Configuration
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Cluster Name PHASEtoEE
* Heartbeat Mechanism Unicast +
* Site Name dallas
* Repository Disk [(00f6f5d015a4310b)] +
Site Multicast Address []
(used only for multicast heartbeat)
* Site Name fortworth
* Repository Disk [(00f61ab216646614)] +
Site Multicast Address []
(used only for multicast heartbeat)
|
Figure 5-8 Defining the repository disks to each site
|
Note: When selecting the CAA repository disk, a PVID is automatically assigned for that disk if it has not a PVID defined already. The SMIT panel gets populated with the PVIDs of the candidate repository disks, making the cluster configuration independent of hdisk device numbers or names.
|
In the next step, we perform the cluster verification and synchronization by executing smitty sysmirror → Cluster Nodes and Networks → Verify and Synchronize Cluster Configuration. Though not necessary at this exact stage, it is recommended to do so as it will also create the CAA cluster. That way, if any problems are encountered they can be addressed early on in the configuration process.
The PowerHA cluster configuration that we have defined so far is shown in the cltopinfo command output in Example 5-5. Also, the specifics of the CAA cluster are shown in Example 5-6 on page 92.
Example 5-5 PowerHA cluster topology
[jessica:root] / # cltopinfo
Cluster Name: PHASEtoEE
Cluster Type: Linked
Heartbeat Type: Unicast
Repository Disks:
Site 1 (dallas@jessica): hdisk1
Site 2 (fortworth@shanley): hdisk2
Cluster Nodes:
Site 1 (dallas):
jessica
cassidy
Site 2 (fortworth):
shanley
There are 3 node(s) and 2 network(s) defined
NODE cassidy:
Network GLVMnet
cassidy_xd 192.168.150.52
Network prodnet
cassidy 192.168.100.52
NODE jessica:
Network GLVMnet
jessica_xd 192.168.150.51
Network prodnet
jessica 192.168.100.51
NODE shanley:
Network GLVMnet
shanley_xd 192.168.150.53
Network prodnet
shanley 192.168.100.53
No resource groups defined
Example 5-6 shows the CAA cluster configuration.
Example 5-6 CAA cluster configuration
[jessica:root] / # lscluster -m
Calling node query for all nodes...
Node query number of nodes examined: 3
Node name: jessica
Cluster shorthand id for node: 1
UUID for node: 9837b6c4-e292-11e3-ac3a-eeaf01717802
State of node: UP NODE_LOCAL
Smoothed rtt to node: 0
Mean Deviation in network rtt to node: 0
Number of clusters node is a member in: 1
CLUSTER NAME SHID UUID
PHASEtoEE 0 9846e270-e292-11e3-ac3a-eeaf01717802
SITE NAME SHID UUID
dallas 1 9837b55c-e292-11e3-ac3a-eeaf01717802
Points of contact for node: 0
----------------------------------------------------------------------------
Node name: shanley
Cluster shorthand id for node: 2
UUID for node: b599dea4-e292-11e3-854b-eeaf01717802
State of node: UP
Smoothed rtt to node: 14
Mean Deviation in network rtt to node: 11
Number of clusters node is a member in: 1
CLUSTER NAME SHID UUID
PHASEtoEE 0 9846e270-e292-11e3-ac3a-eeaf01717802
SITE NAME SHID UUID
fortworth 2 b59a5a28-e292-11e3-854b-eeaf01717802
Points of contact for node: 1
-----------------------------------------------------------------------
Interface State Protocol Status SRC_IP->DST_IP
-----------------------------------------------------------------------
tcpsock->02 UP IPv4 none 192.168.150.51->192.168.150.53
----------------------------------------------------------------------------
Node name: cassidy
Cluster shorthand id for node: 5
UUID for node: 9837b778-e292-11e3-ac3a-eeaf01717802
State of node: UP
Smoothed rtt to node: 7
Mean Deviation in network rtt to node: 3
Number of clusters node is a member in: 1
CLUSTER NAME SHID UUID
PHASEtoEE 0 9846e270-e292-11e3-ac3a-eeaf01717802
SITE NAME SHID UUID
dallas 1 9837b55c-e292-11e3-ac3a-eeaf01717802
Points of contact for node: 2
-----------------------------------------------------------------------
Interface State Protocol Status SRC_IP->DST_IP
-----------------------------------------------------------------------
tcpsock->05 UP IPv4 none 192.168.100.51->192.168.100.52
tcpsock->05 UP IPv4 none 192.168.150.51->192.168.150.52
The exact order of most of the remaining steps are a bit flexible. In our case, we already have a data volume group, amyvg, defined with file systems, rachaelfs, and meganfs, so we will skip those steps. We chose to show configuring the GLVM option first especially when it offers an assistant that minimizes the total number of steps required.
5.4.2 Configure GLVM
The topic of configuring GLVM has been covered in numerous sources including the base PowerHA publications and Redbooks publications. This section is not intended to cover all the detailed options that are available but to focus on one specific implementation as covered in our scenario.
There is some flexibility in the order of the following steps but these we felt are the best order to perform them for consistent results. In PowerHA configurations, it is common, and often required, to have each node in the cluster be both an rpvserver and rpvclient. The rpvserver presents the remote disks to the clients. The clients of course get new disk definitions locally as though it is just another disk. However, they are only accessed via the rpvserver over the IP network.
Add rpvservers to dallas site
For our scenario, we need the four local dallas site disks (hdisk2-5) to be presented to the fortworth site. We also need the two disks from the fortworth site (hdisk3-4) to be presented to both nodes at the dallas site. To accomplish this, we must first define each node to an RPV server site. Then, we must make each disk within each site an rpvserver.
|
Note: When configuring GLVM, it is expecting the disks to not be existing volume group members. If so, they will not show up in the menu pick lists. If you want to use an existing volume group as we have, you must export the volume group via exportvg.
|
We start off on node jessica by executing smitty glvm_utils → Remote Physical Volume Servers → Define / Change / Show Remote Physical Volume Server Site Name. We then enter site name dallas as shown in Figure 5-9. This step is then repeated for node cassidy.
|
Define / Change / Show Remote Physical Volume Server Site Name
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Remote Physical Volume Server Site Name [dallas]
|
Figure 5-9 Define RPV server site to dallas nodes
While still on node jessica, we execute smitty glvm_utils → Remote Physical Volume Servers → Add Remote Physical Volume Servers. We then chose our four disks as shown in Figure 5-10 on page 95.
After selecting the disks and pressing Enter, we are presented with the final rpvserver menu as shown in Figure 5-11 on page 95. We manually enter the IP address that corresponds to the XD_data network interface on node shanley in the fortworth site. We also chose the default to not allow the devices to configure automatically on system restart. Much like volume group activation, we want PowerHA to control this action. Upon pressing Enter to complete the rpvserver configuration, we see rpvserver devices present as shown in Figure 5-12 on page 96. We repeat these steps on the second node, cassidy, in the dallas site with one exception. We also chose the option to not start the devices immediately. This will create the rpvservers and put them in the defined state instead of available.
|
Remote Physical Volume Servers
Move cursor to desired item and press Enter.
Remote Physical Volume Server Site Name Configuration
+--------------------------------------------------------------------------+
| Physical Volume Identifiers |
| |
| Move cursor to desired item and press F7. |
| ONE OR MORE items can be selected. |
| Press Enter AFTER making all selections. |
| |
| # Physical Volume Physical Volume Identifier |
| # ---------------------------------------------------- |
| > hdisk2 00f6f5d01660fbd1 |
| > hdisk3 00f6f5d015a44307 |
| > hdisk4 00f6f5d0166106fa |
| > hdisk5 00f6f5d0166114f3 |
| hdisk6 00f6f5d029906df4 |
| |
| F1=Help F2=Refresh F3=Cancel |
| F7=Select F8=Image F10=Exit |
F1| Enter=Do /=Find n=Find Next |
F9+--------------------------------------------------------------------------+
|
Figure 5-10 Rpvserver disk selection on node jessica in dallas
|
Add Remote Physical Volume Servers
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
Physical Volume Identifiers 00f6f5d01660fbd1 00f6>
* Remote Physical Volume Client Internet Address [192.168.150.53] +
Configure Automatically at System Restart? [no] +
Start New Devices Immediately? [yes] +
|
Figure 5-11 Rpvserver final SMIT window on node jessica
|
COMMAND STATUS
Command: OK stdout: yes stderr: no
Before command completion, additional instructions may appear below.
rpvserver0 Available
rpvserver1 Available
rpvserver2 Available
rpvserver3 Available
|
Figure 5-12 Rpvserver devices are available on node jessica
Add rpvservers to fortworth site
On node shanley, we begin by executing smitty glvm_utils → Remote Physical Volume Servers → Define / Change / Show Remote Physical Volume Server Site Name as shown in Figure 5-13. We then enter site name dallas as shown in Figure 5-9 on page 94. This step is then repeated for node cassidy.
|
Define / Change / Show Remote Physical Volume Server Site Name
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Remote Physical Volume Server Site Name [fortworth]
|
Figure 5-13 Define RPV server site to fortworth node
We next execute smitty glvm_utils → Remote Physical Volume Servers → Add Remote Physical Volume Servers. We then chose only the two local disks (hdisk3-4).
After selecting the disks and pressing Enter, we are presented with the final rpvserver menu as shown in Figure 5-14. We manually enter the IP address that corresponds to the XD_data network interface on node jessica in the dallas site. We also chose the default to not allow the devices to configure automatically on system restart. Much like volume group activation, we want PowerHA to control this action.
|
Add Remote Physical Volume Servers
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
Physical Volume Identifiers 00f61ab21664556b 00f6>
* Remote Physical Volume Client Internet Address [192.168.150.51] +
Configure Automatically at System Restart? [no] +
Start New Devices Immediately? [yes]
|
Figure 5-14 Rpvserver final SMIT window on node shanley in fortworth site
Upon pressing Enter to complete the rpvserver configuration, we see rpvserver devices present as shown in Figure 5-15. Since there is only one node in the fortworth site, no additional rpvserver definitions are required at this time. Though later, we change the definition temporarily to present the disks to node cassidy.
|
COMMAND STATUS
Command: OK stdout: yes stderr: no
Before command completion, additional instructions may appear below.
rpvserver0 Available
rpvserver1 Available
|
Figure 5-15 Rpvserver devices are available on node shanley
The current status of the rpvservers on each node can be seen in Example 5-7.
Example 5-7 Current rpvserver states on all nodes
[jessica:root] / # clcmd lsdev -t rpvstype
-------------------------------
NODE cassidy
-------------------------------
rpvserver0 Defined Remote Physical Volume Server
rpvserver1 Defined Remote Physical Volume Server
rpvserver2 Defined Remote Physical Volume Server
rpvserver3 Defined Remote Physical Volume Server
-------------------------------
NODE shanley
-------------------------------
rpvserver0 Available Remote Physical Volume Server
rpvserver1 Available Remote Physical Volume Server
-------------------------------
NODE jessica
-------------------------------
rpvserver0 Available Remote Physical Volume Server
rpvserver1 Available Remote Physical Volume Server
rpvserver2 Available Remote Physical Volume Server
rpvserver3 Available Remote Physical Volume Server
Adding rpvclients to fortworth site
On node shanley, we execute smitty glvm_utils → Remote Physical Volume Clients → Add Remote Physical Volume Clients. In the first menu regarding IPv6, we keep the default of no and press Enter to continue. We then manually enter the IP address of the XD_data network interface on node jessica. We then get another menu to choose the rpv volume local address. In this case, we chose the XD_data IP of node shanley as shown in Figure 5-16 on page 98.
|
Add Remote Physical Volume Clients
Type or select a value for the entry field.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Remote Physical Volume Server Internet Address [192.168.150.51] +
+--------------------------------------------------------------------------+
| Remote Physical Volume Local Internet Address |
| |
| Move cursor to desired item and press Enter. |
| |
| 192.168.100.53 shanley |
| 192.168.150.53 shanley_xd |
| |
| F1=Help F2=Refresh F3=Cancel |
F1| F8=Image F10=Exit Enter=Do |
F5| /=Find n=Find Next |
+--------------------------------------------------------------------------+
|
Figure 5-16 Rpvclient initial SMIT menu on node shanley
We then get a pop-up menu to choose which disks from node jessica we want to set up as clients. In our case, we choose all four of the disks presented. Upon pressing Enter, we are presented with the final rpvclient SMIT menu as shown in Figure 5-17.
|
Add Remote Physical Volume Clients
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
Remote Physical Volume Server Internet Address 192.168.150.51
Remote Physical Volume Local Internet Address 192.168.150.53
Physical Volume Identifiers 00f6f5d015a4430700000>
I/O Timeout Interval (Seconds) [180] #
Start New Devices Immediately? [yes] +
|
Figure 5-17 Rpvclient final SMIT menu on node shanley
This step results in four new hdisk definitions presented locally onto node shanley as seen in Figure 5-18 on page 99.
|
COMMAND STATUS
Command: OK stdout: yes stderr: no
Before command completion, additional instructions may appear below.
hdisk5 Available
hdisk6 Available
hdisk7 Available
hdisk8 Available
|
Figure 5-18 New hdisks on node shanley
Adding rpvclients to dallas site
On node jessica, we repeat the previous rpvclient steps. We execute smitty glvm_utils → Remote Physical Volume Clients → Add Remote Physical Volume Clients. In the first menu regarding IPv6, we keep the default of no and press Enter to continue. We then manually enter the IP address of the XD_data network interface on node shanley. We then get another menu to choose the rpv volume local address. In this case, we chose the XD_data IP of node shanley.
We then get a pop-up menu to choose which disks from node shanley we want to utilize as clients. In our case, we choose both of the disks (hdisk3-4) presented. Upon pressing Enter, we are presented with the final rpvclient SMIT menu as shown in Figure 5-19.
|
Add Remote Physical Volume Clients
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
Remote Physical Volume Server Internet Address 192.168.150.53
Remote Physical Volume Local Internet Address 192.168.150.51
Physical Volume Identifiers 00f61ab21664556b00000>
I/O Timeout Interval (Seconds) [180] #
Start New Devices Immediately? [yes] +
|
Figure 5-19 Rpvclient final SMIT menu on node jessica
This step results in two new hdisk definitions presented locally onto node jessica as seen in Figure 5-20 on page 100.
|
COMMAND STATUS
Command: OK stdout: yes stderr: no
Before command completion, additional instructions may appear below.
hdisk7 Available
hdisk8 Available
|
Figure 5-20 New hdisks on node jessica
Next, we need to repeat these steps on node cassidy. However, before doing so we have to change the rpvserver configuration on node shanley to change the rpvclient IP address to be the XD_data IP network of node cassidy. This is done easily on node shanley by executing smitty glvm_utils → Remote Physical Volume Servers → Change Multiple Remote Physical Volume Servers and manually typing in the IP address as shown Figure 5-21.
|
Change Multiple Remote Physical Volume Servers
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
Remote Physical Volume Servers rpvserver0 rpvserver1
Remote Physical Volume Client Internet Address [192.168.150.52] +
Configure Automatically at System Restart? [Do not change]
|
Figure 5-21 Change Rpvservers on node shanley to point to node cassidy
Now we repeat the rpvclient steps on node cassidy that we previously executed on note jessica. We execute smitty glvm_utils → Remote Physical Volume Clients → Add Remote Physical Volume Clients. In the first menu regarding IPv6, we keep the default of no and press Enter to continue. We then manually enter the IP address of the XD_data network interface on node shanley. We then get another menu to choose the rpv volume local address. In this case, we chose the XD_data IP of node shanley.
We then get a pop-up menu to choose which disks from node shanley we want to set up as clients. In our case, we choose both of the disks (hdisk3-4) presented. Upon pressing Enter, we are presented with the final rpvclient SMIT menu as shown in Figure 5-22 on page 101. This step will result in two new hdisk definitions presented locally onto node cassidy as seen in Figure 5-23 on page 101.
Upon completion, we changed the rpvservers on node shanley back to the XD_data IP address on node jessica as the client. This is because in our PowerHA cluster, jessica will normally be the primary production system. If a fallover occurs, both the rpvserver and rpvclient definitions and status are automatically controlled by PowerHA.
|
Add Remote Physical Volume Clients
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
Remote Physical Volume Server Internet Address 192.168.150.53
Remote Physical Volume Local Internet Address 192.168.150.52
Physical Volume Identifiers 00f61ab21664556b00000>
I/O Timeout Interval (Seconds) [180] #
Start New Devices Immediately? [yes] +
|
Figure 5-22 Rpvclient final SMIT menu on node cassidy
|
COMMAND STATUS
Command: OK stdout: yes stderr: no
Before command completion, additional instructions may appear below.
hdisk6 Available
hdisk7 Available
|
Figure 5-23 New hdisks on node cassidy
All the disk and rpvclient definitions as shown Example 5-8.
Example 5-8 Hdisks and rpvclients
[jessica:root] / # clcmd lsdev -Cc disk
-------------------------------
NODE cassidy
-------------------------------
hdisk0 Available Virtual SCSI Disk Drive
hdisk1 Available Virtual SCSI Disk Drive
hdisk2 Available Virtual SCSI Disk Drive
hdisk3 Available Virtual SCSI Disk Drive
hdisk4 Available Virtual SCSI Disk Drive
hdisk5 Available Virtual SCSI Disk Drive
hdisk6 Available Remote Physical Volume Client
hdisk7 Available Remote Physical Volume Client
-------------------------------
NODE shanley
-------------------------------
hdisk0 Defined Virtual SCSI Disk Drive
hdisk1 Available Virtual SCSI Disk Drive
hdisk2 Available Virtual SCSI Disk Drive
hdisk3 Available Virtual SCSI Disk Drive
hdisk4 Available Virtual SCSI Disk Drive
hdisk5 Available Remote Physical Volume Client
hdisk6 Available Remote Physical Volume Client
hdisk7 Available Remote Physical Volume Client
hdisk8 Available Remote Physical Volume Client
-------------------------------
NODE jessica
-------------------------------
hdisk0 Available Virtual SCSI Disk Drive
hdisk1 Available Virtual SCSI Disk Drive
hdisk2 Available Virtual SCSI Disk Drive
hdisk3 Available Virtual SCSI Disk Drive
hdisk4 Available Virtual SCSI Disk Drive
hdisk5 Available Virtual SCSI Disk Drive
hdisk6 Available Virtual SCSI Disk Drive
hdisk7 Available Remote Physical Volume Client
hdisk8 Available Remote Physical Volume Client
5.4.3 Create GMVG
In our scenario, we already have an existing LVM mirrored scalable volume group, amyvg, with two copies utilizing mirror pools. We previously exported the volume group in order to allow the disks to show up in the GLVM pick lists. We begin by reimporting the volume group and varying it on node jessica in order to create the third copy and third mirror pool. Example 5-9 shows our current LV copy maps and mirror pools.
Example 5-9 Existing mirror pools and LVM copies
[jessica:root] / # lsmp -A amyvg
VOLUME GROUP: amyvg Mirror Pool Super Strict: yes
MIRROR POOL: primp Mirroring Mode: SYNC
MIRROR POOL: secmp Mirroring Mode: SYNC
[jessica:root] / # lsvg -P amyvg
Physical Volume Mirror Pool
hdisk3 primp
hdisk2 primp
hdisk4 secmp
hdisk5 secmp
[jessica:root] / # lslv -m rachaellv
rachaellv:/rachaelfs
LP PP1 PV1 PP2 PV2 PP3 PV3
0001 0017 hdisk3 0097 hdisk5
0002 0097 hdisk2 0097 hdisk4
0003 0018 hdisk3 0098 hdisk5
0004 0098 hdisk2 0098 hdisk4
0005 0019 hdisk3 0099 hdisk5
0006 0099 hdisk2 0099 hdisk4
0007 0020 hdisk3 0100 hdisk5
0008 0100 hdisk2 0100 hdisk4
0009 0021 hdisk3 0101 hdisk5
0010 0101 hdisk2 0101 hdisk4
0011 0032 hdisk3 0113 hdisk4
0012 0113 hdisk2 0112 hdisk5
0013 0033 hdisk3 0114 hdisk4
0014 0114 hdisk2 0113 hdisk5
0015 0034 hdisk3 0115 hdisk4
0016 0115 hdisk2 0114 hdisk5
0017 0035 hdisk3 0116 hdisk4
0018 0116 hdisk2 0115 hdisk5
0019 0036 hdisk3 0117 hdisk4
0020 0117 hdisk2 0116 hdisk5
[jessica:root] / # lslv -m meganlv
meganlv:/meganfs
LP PP1 PV1 PP2 PV2 PP3 PV3
0001 0022 hdisk3 0102 hdisk5
0002 0102 hdisk2 0102 hdisk4
0003 0023 hdisk3 0103 hdisk5
0004 0103 hdisk2 0103 hdisk4
0005 0024 hdisk3 0104 hdisk5
0006 0104 hdisk2 0104 hdisk4
0007 0025 hdisk3 0105 hdisk5
0008 0105 hdisk2 0105 hdisk4
0009 0026 hdisk3 0106 hdisk5
0010 0106 hdisk2 0106 hdisk4
0011 0027 hdisk3 0108 hdisk4
0012 0108 hdisk2 0107 hdisk5
0013 0028 hdisk3 0109 hdisk4
0014 0109 hdisk2 0108 hdisk5
0015 0029 hdisk3 0110 hdisk4
0016 0110 hdisk2 0109 hdisk5
0017 0030 hdisk3 0111 hdisk4
0018 0111 hdisk2 0110 hdisk5
0019 0031 hdisk3 0112 hdisk4
0020 0112 hdisk2 0111 hdisk5
Create GLVM copy
To create the GLVM copy also involves utilizing smitty glvm_utils. Upon execution, we choose Geographically Mirrored Logical Volumes → Add a Remote Site Mirror Copy to a Logical Volume. We are then presented with a pop-up menu with a list of logical volumes to choose from. In our first pass we choose meganlv. We next choose the remote site, fortworth, and then are presented with a remote candidate disk list with which we can create the mirror. In our case, it is hdisk7 and hdisk8. We chose both disks and are presented with the final SMIT menu as shown in Figure 5-24 on page 104.
In our environment, since we already have two copies we are creating a third copy. So it is necessary to set the option of NEW TOTAL number of logical partition copies to “3”. We also chose not to synchronize the copy of data at this time. We will synchronize all at the same time in a later step.
|
Add a Remote Site Mirror Copy to a Logical Volume
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* LOGICAL VOLUME name meganlv
* NEW TOTAL number of logical partition copies 3 +
* REMOTE PHYSICAL VOLUME Name hdisk7 hdisk8
POSITION on physical volume outer_middle +
RANGE of physical volumes maximum +
Allocate each logical partition copy on a SEPARATE superstrict
physical volume?
SYNCHRONIZE the data in the new logical partition no +
copies?
|
Figure 5-24 Create remote mirrored logical volume
We then repeat this step for both remaining logical volumes rachaellv and shfloglv. After completion, the third copy is stale as shown in Example 5-10. To synchronize, we simply execute syncvg -v amyvg.
Example 5-10 Mirrors stale after creation
[jessica:root] / # lsvg -l amyvg
amyvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
rachaellv jfs2 20 60 6 closed/stale /rachaelfs
meganlv jfs2 20 60 6 closed/stale /meganfs
shfloglv jfs2log 1 3 3 closed/stale N/A
Upon completing, the mirrors are now synchronized as shown Example 5-11.
Example 5-11 Mirrors in sync
[jessica:root] / # lsvg -l amyvg
amyvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
rachaellv jfs2 20 60 6 closed/syncd /rachaelfs
meganlv jfs2 20 60 6 closed/syncd /meganfs
shfloglv jfs2log 1 3 3 closed/syncd N/A
|
Note: GMVGs cannot be configured through C-SPOC. So create on one node, then varyoff and import on the next node. We show later after adding a mirror pool how to accomplish it.
|
Add a mirror pool
We need to add the rpvclient disks, hdisk7 and hdisk8, into a new mirror pool. We must first add the drives into amyvg via extendvg. Then, we use the chpv command to create and add them to a mirror pool as shown in Example 5-12.
Example 5-12 Create new mirror pool
[jessica:root] / # chpv -p thirdmp hdisk7 hdisk8
We verify that there are now three mirror pools by executing the lsmp and lsvg commands as shown in Example 5-13.
Example 5-13 All mirror pools
[jessica:root] / # lsmp -A amyvg
VOLUME GROUP: amyvg Mirror Pool Super Strict: yes
MIRROR POOL: primp Mirroring Mode: SYNC
MIRROR POOL: secmp Mirroring Mode: SYNC
MIRROR POOL: thirdmp Mirroring Mode: SYNC
[jessica:root] / # lsvg -P amyvg
Physical Volume Mirror Pool
hdisk3 primp
hdisk2 primp
hdisk4 secmp
hdisk5 secmp
hdisk7 thirdmp
hdisk8 thirdmp
Importing GMVG to each node
To this point, the RPV servers have been available on the remote node, shanley, and the RPV clients available on the local node, jessica. To import the GMVGs to the other local node, cassidy, the RPV clients need to be available to it. The RPV server must also be changed in order to make the RPV clients available on node cassidy.
This is done as follows:
1. Varyoff the volume groups on the local node, jessica.
2. Make the RPV clients defined on the local node, jessica.
3. Change the RPV servers defined on the remote node, shanley, to client cassidy.
4. Make the RPV clients available on the local node, cassidy.
5. Import the volume group to local node cassidy.
On the local node, jessica, execute:
– varyoffvg amyvg
– rmdev -l hdisk7
– rmdev -l hdisk8
This puts both disks in the defined state on node jessica. Next, we need to change the client address on the RPV servers on node shanley. This can be done in SMIT as we did previously in Figure 5-21 on page 100. However, we will perform it from the command line this time.
On the remote node shanley, execute:
– chdev -l rpvserver0 -a client_address=192.168.150.52
– chdev -l rpvserver1 -a client_address=192.168.150.52
On the local cassidy, the disks were still in available state. If they were not we could do so by executing:
– mkdev -l hdisk7
– mkdev -l hdisk8
At this point, we recommend to verify that disk access is indeed possible from node cassidy. This can be done by executing:
– lquerypv -h /dev/hdisk6
– lquerypv -h /dev/hdisk7
Upon positive confirmation of disk access, we can now import the volume group as follows:
– importvg -y amyvg hdisk7
The results of the command execution and the disk listing is shown in Example 5-14.
Example 5-14 GMVG amyvg imported on second local node cassidy
[cassidy:root] / # importvg -y amyvg hdisk7
amyvg
0516-783 importvg: This imported volume group is concurrent capable.
Therefore, the volume group must be varied on manually.
[cassidy:root] / # lspv
hdisk0 00f70c99013e28ca rootvg active
hdisk1 00f6f5d015a4310b caavg_private active
hdisk2 00f6f5d015a44307 amyvg
hdisk3 00f6f5d01660fbd1 amyvg
hdisk4 00f6f5d0166106fa amyvg
hdisk5 00f6f5d0166114f3 amyvg
hdisk6 00f61ab21664556b amyvg
hdisk7 00f61ab21664742e amyvg
We put the RPV clients, hdisk6 and hdisk7, back in the defined state on node cassidy by executing:
– rmdev -l hdisk6
– rmdev -l hdisk7
Next, we need to import the GMVG amyvg to the remote node shanley. Currently, all four RPV servers are available on node jessica and have node shanley set up as their client as shown in Example 5-15.
Example 5-15 RPVserver status on jessica
[jessica:root] / # lsdev -t rpvstype
rpvserver0 Available Remote Physical Volume Server
rpvserver1 Available Remote Physical Volume Server
rpvserver2 Available Remote Physical Volume Server
rpvserver3 Available Remote Physical Volume Server
[jessica:root] / # lsattr -El rpvserver0
auto_online n Configure at System Boot True
client_addr 192.168.150.53 Client IP Address True
rpvs_pvid 00f6f5d01660fbd10000000000000000 Physical Volume Identifier True
[jessica:root] / # lsattr -El rpvserver1
auto_online n Configure at System Boot True
client_addr 192.168.150.53 Client IP Address True
rpvs_pvid 00f6f5d015a443070000000000000000 Physical Volume Identifier True
[jessica:root] / # lsattr -El rpvserver2
auto_online n Configure at System Boot True
client_addr 192.168.150.53 Client IP Address True
rpvs_pvid 00f6f5d0166106fa0000000000000000 Physical Volume Identifier True
[jessica:root] / # lsattr -El rpvserver3
auto_online n Configure at System Boot True
client_addr 192.168.150.53 Client IP Address True
rpvs_pvid 00f6f5d0166114f30000000000000000 Physical Volume Identifier True
The RPV clients are also still available on node shanley as shown in Example 5-16.
Example 5-16 RPV client status on shanley
[shanley:root] /# lsdev -t rpvclient
hdisk5 Available Remote Physical Volume Client
hdisk6 Available Remote Physical Volume Client
hdisk7 Available Remote Physical Volume Client
hdisk8 Available Remote Physical Volume Client
Just like before, it is recommended to verify that disk access is indeed possibly on node cassidy. This can be done by executing:
– lquerypv -h /dev/hdisk5
– lquerypv -h /dev/hdisk6
– lquerypv -h /dev/hdisk7
– lquerypv -h /dev/hdisk8
Upon positive confirmation of disk access, we can now import the volume group as follows:
– importvg -y amyvg hdisk8
The results of the command execution and the disk listing is shown in Example 5-17.
Example 5-17 GMVG amyvg imported on remote node shanley
[shanley:root] / # importvg -y amyvg hdisk8
amyvg
0516-783 importvg: This imported volume group is concurrent capable.
Therefore, the volume group must be varied on manually.
[shanley:root] / # lspv
hdisk1 00f61ab215ad00d4 rootvg active
hdisk2 00f61ab216646614 caavg_private active
hdisk3 00f61ab21664556b amyvg
hdisk4 00f61ab21664742e amyvg
hdisk5 00f6f5d015a44307 amyvg
hdisk6 00f6f5d01660fbd1 amyvg
hdisk7 00f6f5d0166106fa amyvg
hdisk8 00f6f5d0166114f3 amyvg
Next, we put all RPV clients and servers in the defined state via the rmdev -l command as shown previously. The status of all RPV servers and clients is shown in Example 5-18.
Example 5-18 RPV server and client status on all nodes
[jessica:root] / # clcmd lsdev -t rpvstype
-------------------------------
NODE cassidy
-------------------------------
rpvserver0 Defined Remote Physical Volume Server
rpvserver1 Defined Remote Physical Volume Server
rpvserver2 Defined Remote Physical Volume Server
rpvserver3 Defined Remote Physical Volume Server
-------------------------------
NODE shanley
-------------------------------
rpvserver0 Defined Remote Physical Volume Server
rpvserver1 Defined Remote Physical Volume Server
-------------------------------
NODE jessica
-------------------------------
rpvserver0 Defined Remote Physical Volume Server
rpvserver1 Defined Remote Physical Volume Server
rpvserver2 Defined Remote Physical Volume Server
rpvserver3 Defined Remote Physical Volume Server
[jessica:root] / # clcmd lsdev -t rpvclient
-------------------------------
NODE cassidy
-------------------------------
hdisk6 Defined Remote Physical Volume Client
hdisk7 Defined Remote Physical Volume Client
-------------------------------
NODE shanley
-------------------------------
hdisk5 Defined Remote Physical Volume Client
hdisk6 Defined Remote Physical Volume Client
hdisk7 Defined Remote Physical Volume Client
hdisk8 Defined Remote Physical Volume Client
-------------------------------
NODE jessica
-------------------------------
hdisk7 Defined Remote Physical Volume Client
hdisk8 Defined Remote Physical Volume Client
5.4.4 Create resource group
Before creating the resource group, we re-created our application server, banner, along with two service IPs, dallasserv and ftwserv, and assigned them to their respective sites.
To create the resource group, execute smitty sysmirror → Cluster Applications and Resources → Resource Groups → Add a Resource Group. We are presented with, and fill out the fields to, the SMIT menu as shown in Figure 5-25 on page 109.
|
Add a Resource Group (extended)
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* Resource Group Name [xsiteGLVMRG]
Inter-Site Management Policy [Online on Primary Site] +
* Participating Nodes from Primary Site [jessica cassidy] +
Participating Nodes from Secondary Site [shanley] +
Startup Policy Online On Home Node O> +
Fallover Policy Fallover To Next Prio> +
Fallback Policy Never Fallback +
|
Figure 5-25 Creating two site resource group
5.4.5 Add GMVG into resource group
We now add all of our resources, including GMVG of amyvg, into the resource group. To do so, we execute smitty sysmirror → Cluster Applications and Resources → Resource Groups → Change/Show Resources and Attributes for a Resource Group and choose the only resource group, xsiteGLVMRG, and press Enter. For clarity, only the fields we utilized are shown in Figure 5-26. All other fields, we left them at their default values.
|
Resource Group Name xsiteGLVMRG
Inter-Site Management Policy Online On Primary Site Participating Nodes from Primary Site jessica cassidy
Participating Nodes from Secondary Site shanley
Startup Policy Online On Home Node O>
Fallover Policy Fallover To Next Prio>
Fallback Policy Never Fallback
Service IP Labels/Addresses [dallasserv ftwserv] +
Application Controller Name [banner] +
Volume Groups [amyvg] +
Use forced varyon of volume groups, if necessary true
|
Figure 5-26 Adding GMVG resource to resource group
The ending cluster resource group configuration looks very much like a cross-site LVM mirroring configuration. However, it is both an LVM mirrored solution within the dallas site. It is a GLVM configuration across sites to fortworth.
It is recommended next to synchronize the cluster to determine whether errors exist. An overview of our cluster is shown in Figure 5-27 on page 110.
Figure 5-27 Test cluster final configuration
5.5 Defining manual site split and merge policy
Overall, PowerHA is designed for automatic fallover. However, in cases of sites with data replication involved there is some reluctance to allow it. Especially in the case of site isolation resulting in false fallover. This can lead to very undesired results by having both copies of data active. There are also times, especially when only one node at each site, that customers want to evaluate the initial site failure to determine if it is recoverable quickly before deciding to fallover to the disaster recovery site. This is generally because getting back to the primary production site involves at syncing up in deltas in the data. Though it does not have to be process intensive, it can be time intensive.
In previous versions of PowerHA when utilizing storage replication there was, and still is, an option to choose a Manual recovery action. Initially, this only applied to scenarios in which PowerHA detected that the replicated pairs where not in sync at the time of site failure. So it is codependent on the status of the replicated resources. This means that it was still possible for automatic fallover to proceed even with the manual option set.
In PowerHA v7.1.3, there are new manual split and merge policies. It can, and would be, applied globally across the cluster. However, there is an option to specify if it should apply to storage replication recovery or not.
Now when using GLVM specifically, there historically has been no method for manual fallover within PowerHA. If that was warranted, it was simply suggested to implement stand-alone GLVM.
|
Restriction: The manual split/merge option only applies to linked clusters but cannot be used with those replication methods that utilize the genxd filesets. At time of writing, this consisted of DS8000 and XIV.
|
To configure this feature, we execute smitty sysmirror → Custom Cluster Configuration → Cluster Nodes and Networks → Initial Cluster Setup (Custom) → Configure Split and Merge Policy for a Linked Cluster and then are presented with the SMIT menu as shown in Figure 5-28.
|
Configure Split and Merge Policy for a Linked Cluster
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
Split Handling Policy Manual +
Merge Handling Policy Manual +
Split and Merge Action Plan Reboot
Select Tie Breaker +
Manual Choice Options
Notify Method []
Notify Interval (seconds) [20] +#
Maximum Notifications [15] +#
Default Surviving Site [dallas] +
Apply to Storage Replication Recovery No +
|
Figure 5-28 Configure manual split and merge policy
The description and options of each field are shown in Table 5-3 on page 112.
|
Note: The Manual choice option was added in PowerHA 7.1.3. If you specify a number of notifications, you must choose a default surviving site. Inversely, if you do not choose maximum notifications, you cannot choose a default surviving site.
|
Table 5-3 Configure Cluster Split and Merge Policy fields
|
Field
|
Description
|
|
Split handling policy
|
Select None, the default setting, for the partitions to operate independently of each other after the split occurs.
Select Tie breaker to use the disk that is specified in the Select tie breaker field after a split occurs. When the split occurs, one site wins the SCSI reservation on the tie breaker disk. The site that losses the SCSI reservation uses the recovery action that is specified in the policy setting.
Note: If you select the Tie breaker option in the Merge handling policy field, you must select Tie breaker for this field.
Select Manual to wait for manual intervention when a split occurs. PowerHA SystemMirror does not perform any actions on the cluster until you specify how to recover from the split.
Note: If you select the Manual option in the Merge handling policy field, you must select Manual for this field.
|
|
Merge handling policy
|
Select Majority to choose the partition with the highest number of nodes as the primary partition.
Select Tie breaker to use the disk that is specified in the Select tie breaker field after a merge occurs.
Note: If you select the Tie breaker option in the Split handling policy field, you must select Tie breaker for this field.
Select Manual to wait for manual intervention when a merge occurs. PowerHA SystemMirror does not perform any actions on the cluster until you specify how to handle the merge.
|
|
Split and merge action plan
|
Reboot will reboot all nodes in the site that does not win the tie breaker or is not responded to manually when using the manual choice option below. This is not an editable option.
|
|
Select tie breaker
|
Select an iSCSI disk or a SCSI disk that you want to use as the tie breaker disk. It must support either SCSI-2 or SCSI-3 reserves.
|
|
Manual Choice Option
|
Notify Method is a method to be invoked in addition to a message to /dev/console to inform the operator of the need to choose which site will continue after a split or merge. The method is specified as a path name, followed by optional parameters. When invoked, the last parameter will be either “split” or “merge” to indicate the event.
Notify Interval The frequency of the notification time, in seconds, between the message to inform the operator of the need to choose which site will continue after a split or merge. The supported values are 10 - 3600.
Maximum Notifications is the maximum number of times that PowerHA SystemMirror will prompt the operator to choose which site will continue after a split or merge. The default, blank, is infinite. Otherwise, the supported values are 3 - 1000. However, this value cannot be blank when a surviving site is specified.
Default Surviving Site If the operator has not responded to a request for a manual choice of surviving site on a “split” or “merge”, this site will be allowed to continue. The other site will take the action chosen under “Action Plan”. The time the operator has to respond is “Notify Interval” times “Maximum Notifications+1”.
Apply to Storage Replication Recovery determines if the manual response on a split also applies to those storage replication recovery mechanisms that provide an option for “Manual” recovery. If “Yes” is selected, the partition that was selected to continue on a split will proceed with takeover of the storage replication recovery. This cannot be used when DS8k and XIV replication is used.
|
In our scenario, we chose a Manual for both the split and merge policy. We also specified a Notification Interval of 20 seconds and Maximum Notification of 15. This is roughly equivalent to 5 minutes. Though in actuality, it is closer to five minutes and twenty seconds as the time for an operator is actually one additional notification interval above the maximum notifications as explained in the table above.
Synchronizing the cluster is required for these settings to take effect.
5.6 Testing manual split option
The requirement of a split is that all the IP communications between sites are lost. This can be achieved in numerous ways. In our case, we simply pulled the physical Ethernet cables from node shanley. However, since most environments are virtualized this is rare to have the ability to pull cables and it not affect other systems beyond the cluster nodes. In those cases, other options like disabling switch ports or dynamically changing the virtual adapters can also give the wanted results.
To test the cluster, we begin will all nodes active in the cluster. We then perform the following steps:
1. Halt first node, jessica, in dallas site.
2. Sever links between sites.
3. Respond to prompt to continue on remote site.
4. Respond to prompt to recover on primary site.
5. Restart primary site nodes to rejoin cluster.
6. Manually move resource group back to primary site.
5.6.1 First node failure in site dallas
In example Example 5-19, we show the current status of the resource group and resources in the cluster as we begin. Notice that all disks in amyvg are active. This will change after fallover to the remote site and will be highlighted again later.
Example 5-19 All nodes active and beginning resource status
[jessica:root] / # clRGinfo
-----------------------------------------------------------------------------
Group Name State Node
-----------------------------------------------------------------------------
xsiteGLVMRG ONLINE jessica@dallas
OFFLINE cassidy@dallas
ONLINE SECONDARY shanley@fortwo
[jessica:root] / # lspv
hdisk0 00f6f5d00146570c rootvg active
hdisk1 00f6f5d015a4310b caavg_private active
hdisk2 00f6f5d01660fbd1 amyvg active
hdisk3 00f6f5d015a44307 amyvg active
hdisk4 00f6f5d0166106fa amyvg active
hdisk5 00f6f5d0166114f3 amyvg active
hdisk6 00f6f5d029906df4 None
hdisk7 00f61ab21664556b amyvg active
hdisk8 00f61ab21664742e amyvg active
[jessica:root] / # lsvg -p amyvg
amyvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk3 active 78 58 16..00..11..15..16
hdisk2 active 478 457 96..75..95..95..96
hdisk4 active 478 457 96..75..95..95..96
hdisk5 active 478 458 96..76..95..95..96
hdisk7 active 478 457 96..75..95..95..96
hdisk8 active 478 458 96..76..95..95..96
[jessica:root] / # netstat -i
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en0 1500 link#2 ee.af.1.71.78.2 1015673 0 962125 0 0
en0 1500 10.10.8 dallasserv 1015673 0 962125 0 0
en0 1500 192.168.100 jessica 1015673 0 962125 0 0
en1 1500 link#3 ee.af.1.71.78.3 1454861 0 1644616 0 0
en1 1500 192.168.150 jessica_xd 1454861 0 1644616 0 0
There are several ways to fail a node. Generally, either an immediate shutdown via the Hardware Management Console (HMC) or via halt -q. In our case, we execute the halt command. This results in a fallover to second local dallas node, cassidy. The results of the fallover look very similar to what it did on node jessica. It is just now all resources are online to node cassidy, as shown in Example 5-20.
Example 5-20 Resources active on cassidy after local node failure
[cassidy:root] / # clRGinfo
-----------------------------------------------------------------------------
Group Name State Node
-----------------------------------------------------------------------------
xsiteGLVMRG OFFLINE jessica@dallas
ONLINE cassidy@dallas
ONLINE SECONDARY shanley@fortwo
[cassidy:root] / # lspv
hdisk0 00f70c99013e28ca rootvg active
hdisk1 00f6f5d015a4310b caavg_private active
hdisk2 00f6f5d015a44307 amyvg active
hdisk3 00f6f5d01660fbd1 amyvg active
hdisk4 00f6f5d0166106fa amyvg active
hdisk5 00f6f5d0166114f3 amyvg active
hdisk6 00f61ab21664556b amyvg active
hdisk7 00f61ab21664742e amyvg active
[cassidy:root] / # lsvg -p amyvg
amyvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk2 active 78 58 16..00..11..15..16
hdisk3 active 478 457 96..75..95..95..96
hdisk4 active 478 457 96..75..95..95..96
hdisk5 active 478 458 96..76..95..95..96
hdisk6 active 478 457 96..75..95..95..96
hdisk7 active 478 458 96..76..95..95..96
[cassidy:root] / # netstat -i
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en0 1500 link#2 7a.40.c8.b3.15.2 1266099 0 1267951 0 0
en0 1500 10.10.8 dallasserv 1266099 0 1267951 0 0
en0 1500 192.168.100 cassidy 1266099 0 1267951 0 0
en1 1500 link#3 7a.40.c8.b3.15.3 1019856 0 980645 0 0
en1 1500 192.168.150 cassidy_xd 1019856 0 980645 0 0
Notice again that all the disks are active. This is normal because cassidy has access to all the local disks and during fallover RPV server changes to the new client address of cassidy. However, we will see something different after failing cassidy as this results in a site failure.
5.6.2 Site split
This case is a continuation of the previous section. Dallas site node jessica is still down but both local node cassidy and remote shanley are still active in the cluster. We physically pull the Ethernet cables from node shanley. This results in a site split.
|
Note: A demo of split site recovery on this exact cluster can be found at:
|
Upon detection of the site split, and the fact that we chose manual split merge options previously, we are prompted with the following via the console terminal window on the HMC on both nodes, as shown in Figure 5-29 on page 116.
The manual prompt status can also be found on either node without access to a console terminal window by executing smitty sysmirror → Problem Determination Tools → Manual Response to Split or Merge → Display any needed Manual Response.
Figure 5-29 Manual operator response prompt upon site split
In our case, we do want the remote site, fortworth, to continue with takeover. To do as the console message indicates, we simply execute cl_sm_continue. However, this too can be performed within SMIT via smitty sysmirror → Problem Determination Tools → Manual Response to Split or Merge → Provide a Manual Response. Then, choose either Continue or Recover as shown in Figure 5-30.
We also want the local site node, cassidy, to reboot. We execute cl_sm_recover in order for it to be rebooted for us.
|
Provide a Manual Response
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
* This site should Recover +
+--------------------------------------------------------------------------+
| This site should |
| |
| Move cursor to desired item and press Enter. |
| |
| Continue |
| Recover |
| |
| F1=Help F2=Refresh F3=Cancel |
F1| F8=Image F10=Exit Enter=Do |
F5| /=Find n=Find Next |
F9+--------------------------------------------------------------------------+
|
Figure 5-30 Manual response via SMIT
Now since the resources are activated on node shanley there are a couple of key differences. They are the fact that the site-specific service address of ftwserv is active and that since the primary site is down, the four remote disks are now in the missing state as shown in Example 5-21 on page 117.
Example 5-21 Remote takeover completed
[shanley:root] / # cl_sm_continue
Resource Class Action Response for ResolveOpQuorumTie
[shanley:root] / # clRGinfo
-----------------------------------------------------------------------------
Group Name State Node
-----------------------------------------------------------------------------
xsiteGLVMRG OFFLINE jessica@dallas
OFFLINE cassidy@dallas
ONLINE shanley@fortwo
[shanley:root] / # lspv
hdisk1 00f61ab215ad00d4 rootvg active
hdisk2 00f61ab216646614 caavg_private active
hdisk3 00f61ab21664556b amyvg active
hdisk4 00f61ab21664742e amyvg active
hdisk5 00f6f5d015a44307 amyvg active
hdisk6 00f6f5d01660fbd1 amyvg active
hdisk7 00f6f5d0166106fa amyvg active
hdisk8 00f6f5d0166114f3 amyvg active
[shanley:root] / # lsvg -p amyvg
amyvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk5 missing 78 58 16..00..11..15..16
hdisk6 missing 478 457 96..75..95..95..96
hdisk7 missing 478 457 96..75..95..95..96
hdisk8 missing 478 458 96..76..95..95..96
hdisk3 active 478 457 96..75..95..95..96
hdisk4 active 478 458 96..76..95..95..96
[shanley:root] / # lsvg amyvg |grep STALE
STALE PVs: 4 STALE PPs: 14
[shanley:root] / # netstat -i
Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll
en0 1500 link#2 6e.8d.d0.21.d0.2 1532614 0 1530640 0 0
en0 1500 10.10.8 ftwserv 1532614 0 1530640 0 0
en0 1500 192.168.100 shanley 1532614 0 1530640 0 0
en1 1500 link#3 6e.8d.d0.21.d0.3 1613987 0 1183422 0 0
5.6.3 Restart primary site nodes
In this case we perform two steps:
1. Restart both LPARs via the HMC
2. Restart cluster services on both nodes
Upon successful reintegration of the primary site, PowerHA/EE will automatically perform the following actions:
– Activate RPV servers on dallas site node jessica
– Activate RPV clients on node shanley
– Reactive volume group to change disks from missing to active
– Automatically resync the stale partitions in the volume group
The end result of these actions is shown in Example 5-22.
Example 5-22 Auto resync after site reintegration
[shanley:root] / # clRGinfo
-----------------------------------------------------------------------------
Group Name State Node
-----------------------------------------------------------------------------
xsiteGLVMRG ONLINE SECONDARY jessica@dallas
OFFLINE cassidy@dallas
ONLINE shanley@fortwo
[shanley:root] / # lsvg -p amyvg
amyvg:
PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION
hdisk5 active 78 58 16..00..11..15..16
hdisk6 active 478 457 96..75..95..95..96
hdisk7 active 478 457 96..75..95..95..96
hdisk8 active 478 458 96..76..95..95..96
hdisk3 active 478 457 96..75..95..95..96
hdisk4 active 478 458 96..76..95..95..96
[shanley:root] / # lsvg amyvg |grep STALE
STALE PVs: 0 STALE PPs: 0
5.6.4 Move resource group back to primary site
To move the resource group back to the primary site, like most tasks, it can be done both via SMIT or the clmgr command line. It also can be executed on any active node in the cluster. In our case, we used the clmgr command on node shanley as shown in Example 5-23.
Example 5-23 Resource group move via clmgr
[shanley:root] / # clmgr move rg xsiteGLVMRG site=dallas
Attempting to move group xsiteGLVMRG to site dallas.
Waiting for the cluster to process the resource group movement request....
Waiting for the cluster to stabilize............
Resource group xsiteGLVMRG is online on site dallas.
Cluster Name: PHASEtoEE
Resource Group Name: xsiteGLVMRG
Node Primary State Secondary State
---------------------------- ---------------
jessica@dallas ONLINE OFFLINE
cassidy@dallas OFFLINE OFFLINE
shanley@fortworth OFFLINE ONLINE SECONDARY
To perform the same operation via SMIT, execute smitty cspoc → Resource Group and Applications → Move Resource Groups to Another Site and choose the online resource group xsiteGLVMRG as shown in Figure 5-31 on page 119.
|
Resource Group and Applications
Move cursor to desired item and press Enter.
+--------------------------------------------------------------------------+
| Select a Resource Group |
| |
| Move cursor to desired item and press Enter. Use arrow keys to scroll. |
| |
| # |
| # Resource Group State Node(s) / Site |
| # |
| xsiteGLVMRG ONLINE SECONDARY jessica / dalla |
| xsiteGLVMRG ONLINE shanley / fortw |
| |
| # |
| # Resource groups in node or site collocation configuration: |
| # Resource Group(s) State Node / Site |
| # |
| |
| F1=Help F2=Refresh F3=Cancel |
| F8=Image F10=Exit Enter=Do |
F1| /=Find n=Find Next |
F9+--------------------------------------------------------------------------+
|
Figure 5-31 Select resource group to move
We then choose site dallas from the pop-up picklist as shown in Figure 5-32 on page 120 as it is the only other site possible to move it to. We are then presented with the final SMIT menu as shown in Figure 5-33 on page 120.
Upon successful execution, we get identical feedback as we did from using the clmgr command as shown in Example 5-23 on page 118.
|
Resource Group and Applications
Move cursor to desired item and press Enter.
Show the Current State of Applications and Resource Groups
Bring a Resource Group Online
Bring a Resource Group Offline
Move Resource Groups to Another Node
Move Resource Groups to Another Site
Suspend/Resume Application Monitoring
Application Availability Analysis
+--------------------------------------------------------------------------+
| Select a Destination Site |
| |
| Move cursor to desired item and press Enter. |
| |
| # *Denotes Originally Configured Primary Site |
| dallas |
| |
| F1=Help F2=Refresh F3=Cancel |
| F8=Image F10=Exit Enter=Do |
F1| /=Find n=Find Next |
F9+--------------------------------------------------------------------------+
|
Figure 5-32 Select site to move resource group to
|
Move Resource Group(s) to Another Site
Type or select values in entry fields.
Press Enter AFTER making all desired changes.
[Entry Fields]
Resource Group(s) to be Moved xsiteGLVMRG
Destination Site dallas
Name of site containing data to be preserved +
by data divergence recovery processing
(Asynchronous GLVM Mirroring only)
|
Figure 5-33 Move resource group final SMIT window
5.7 Testing manual merge option
We start off the same way as in the manual split test. We cause a split to occur; however, we simply do not answer the prompt. We are strictly trying to create a merge scenario and all event processing is held up when the prompt occurs. So we have to let each one think they are the only one in the cluster.
We then restore communications between the sites. The prompt notification changes on both nodes as shown in Figure 5-29 on page 116. But this time, the message indicates a merge was detected.
In our case, we told the local site to continue and the remote site to recover. This caused the remote site to reboot. We then restarted cluster services on the remote node and it joined the cluster and stabilized.
Now if we would have told the remote site to recover when the split was detected instead, and communications restored before rejoining it into the cluster, a merge condition would not have occurred.
In either case of a split or merge, the options must be very carefully considered. Better yet, they also must be fully tested to ensure the wanted results.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.