Remote mirroring with IBM Spectrum Accelerate
The remote mirroring function provides a real-time copy between two or more IBM Spectrum Accelerate or XIV storage systems over iSCSI links. This feature provides a method to protect data from site failures and, when used with mirrored snapshots, can provide point-in-time copies of critical data.
This chapter includes the following topics:
7.1 Remote mirroring overview
Mirroring creates a set of consistent data that can be used when there are problems with the production volumes. It also can be used for other purposes, such as testing and backup on the remote site by using snapshots of consistent data.
Remote mirroring is independent of application and operating system and does not require the use of server-processor cycle.
Remote mirroring can be a synchronous copy solution in which a write operation is completed on local and remote site copies before an acknowledgment is returned to the host that issued the write. This type of remote mirroring typically is used for geographically close sites to minimize the effect of I/O delays, which are proportional to the distance between the sites.
Remote mirroring also can be an asynchronous solution in which consistent sets of data are copied to the remote location at predefined intervals while the host I/O operations are acknowledged directly after they are written on the primary site alone. This configuration is typically used for longer distances between sites.
|
Important: The remote mirroring function between two or more IBM Spectrum Accelerate or XIV storage systems is exclusively over iSCSI links.
A reliable, dedicated network is preferred for mirroring. Links can be shared, but require available and consistent network bandwidth. The specified minimum bandwidth (50 Mbps for iSCSI) is a functional minimum and might not necessarily meet the replication speed that is required for a customer environment and workload.
Also, minimum bandwidths are not time-averaged (as typically reported by network monitoring packages) but are instantaneous, constant requirements that are typically achievable through network quality of service (QoS) only.
|
Unless otherwise noted, this chapter describes the basic concepts, functions, and terms that are common to synchronous and asynchronous mirroring that is available with the IBM XIV Storage System and IBM Spectrum Accelerate.
For more information, see IBM XIV Storage System Business Continuity Functions, SG24-7759.
7.1.1 Remote mirror terminology
The following mirroring-related terms are used in this book:
•Local site: This site consists of the primary storage system and the servers that are running applications that are stored on that storage system.
•Remote site: This site holds the mirror copy of the data and often also has standby servers. A remote site can become the active production site by using a consistent data copy.
•Primary: This term denotes the XIV or IBM Spectrum Accelerate that is used for production during typical business routines to serve hosts and have its data replicated to a secondary XIV or IBM Spectrum Accelerate. You might also refer to it as the source system.
•Secondary. This term denotes the system that is used during normal circumstances to act as the mirror (backup) for the primary. You might also refer to it as the destination system.
•Consistency group (CG): This group is a set of related volumes on a single system that are treated as one logical unit. Thus, all CG data reflects correctly ordered writes across all respective volumes within the CGs. Consistency groups are supported within remote mirroring.
•Coupling: This term refers to the pairing of volumes or CGs to form a mirror relationship between the source of a replication and its destination (target).
•Peer: This term refers to one side of a coupling. It can be a volume or a consistency group. However, peers must be of the same type (that is, both volumes or both CGs). Whenever a coupling is defined, a role is specified for each peer. One peer is designated as the source and the other peer is designated as the destination.
•Role: This term denotes the following role that the peer is fulfilling:
– Source: A role that indicates that the peer serves host requests and acts as the source for replication.
– Destination: A role that indicates that the peer does not serve host write requests (it can be used in read-only mode) and acts as the target for replication.
Changing a peer’s role might be warranted after the peer is recovered from a site or system, or because of a link failure or disruption.
•Sync job: This term applies to asynchronous mirroring only. It denotes a synchronization procedure that is run by the source at user-configured intervals that correspond to the asynchronous mirroring definition, or upon manually running the XCLI command mirror_create_snapshot (which is also used for synchronous mirroring, but not as part of a scheduled job).
The resulting job is referred to as snapshot mirror sync job, ad hoc sync job, or manual sync job in contrast with a scheduled sync job. The sync job entails synchronization of data updates that are recorded on the source since the creation time of the most-recent snapshot that was successfully synchronized.
•Offline initialization (offline init): A mechanism whereby XIV or IBM Spectrum Accelerate (by using HASH values) compares respective source and target 64 KB data blocks and copies over only the parts that have different data. Offline initialization aims at expediting the initialization of mirror pairs that are known to be inherently similar (for example, when an asynchronous pair is changed to a synchronous pair).
This unique feature of XIV and IBM Spectrum Accelerate is evident when the data links do not have adequate speed or capacity to transmit the entire volume in a timely fashion. In that case, the pair is first created when the systems are at close proximity and can use fast links. Then, when the storage system that hosts the remote mirror is placed at its final physical destination, only the changed data because those volumes were identical must be copied over the wire.
•Asynchronous schedule interval: This term applies only to asynchronous mirroring. It represents (per a coupling) how often the source automatically runs a new sync job. The default interval and the minimum possible is 20 seconds.
•Recovery point objective (RPO): This setting is a setting that is applicable to asynchronous mirroring only. It represents an objective that is set by the user, which indicates the maximal currency difference that is considered acceptable between the mirror peers (the actual difference between mirror peers can be shorter or longer than the RPO set).
An RPO of zero indicates that no difference between the mirror peers can be tolerated, and that sync mirroring is required. An RPO that is greater than zero indicates that the replicated volume is less current or lags somewhat behind the source volume. In this case, there is a potential for certain transactions that are run against the production volume to be rerun when applications start to use the replicated volume.
For XIV asynchronous mirroring, the required RPO is user-specified. The storage system then reports effective RPO and compares it to the required RPO.
Connectivity, bandwidth, and distance between the systems directly affect RPO. More connectivity, greater bandwidth, and less distance typically enable a lower RPO.
7.1.2 Remote mirroring modes
IBM Spectrum Accelerate supports synchronous mirroring and asynchronous mirroring.
Synchronous mirroring
Synchronous mirroring accommodates a requirement for zero RPO.
To ensure that data is also written to the secondary system (destination role), an acknowledgment of the write operation to the host is issued only after the data is written to both storage systems (IBM Spectrum Accelerate or XIV). This configuration ensures that mirroring peers always contain the same data. A write acknowledgment is returned to the host only after the write data is cached by two separate IBM Spectrum Accelerate or XIV modules at each site, as shown in Figure 7-1.
Figure 7-1 Synchronous mirroring
Host read operations are provisioned by the primary (source role); writing is run at the primary (source role) and replicated to the secondary systems.
Asynchronous mirroring
Asynchronous mirroring provides a consistent replica of data on a target peer through timely replication of data changes that are recorded on a source peer.
Asynchronous mirroring uses the snapshot function, which creates a point-in-time image. In asynchronous mirroring, successive snapshots (point-in-time images) are made and used to create consistent data on the destination peers. The system sync job copies the data that corresponds to the differences between two designated snapshots on the source (most-recent and last-replicated).
For asynchronous mirroring, acknowledgment of write complete is returned to the application as soon as the write data is received at the local storage system, as shown in Figure 7-2.
Figure 7-2 Asynchronous mirroring
7.2 Mirroring schemes
Whether synchronous or asynchronous, mirroring requires two or more IBM Spectrum Accelerate or XIV systems. The source and target of the asynchronous mirroring can be synchronous and form a local mirroring, or they can be at different sites and facilitate a disaster recovery plan. Figure 7-3 on page 180 shows how peers can be spread across multiple storage systems and sites. Any system that is depicted can be an XIV or IBM Spectrum Accelerate storage system.
Figure 7-3 Mirroring replication schemes
A system can be connected to up to 10 other XIV or IBM Spectrum Accelerate targets for mirroring purposes. Any system can be used simultaneously as replication source and replication target.
In a bidirectional configuration, a system concurrently functions as the replication source (source) for one or more couplings, and as the replication target (destination) for other couplings. Figure 7-3 shows possible schemes for how mirroring can be configured.
Figure 7-4 shows connectivity among systems and groups of systems, as shown in the XIV GUI.
Figure 7-4 XIV GUI connectivity among systems and group of systems
7.2.1 Peer designations and roles
A peer (volume or consistency group) is assigned a source or a destination role when the mirror is defined. By default, the location of the source denotes the primary system in a new mirror definition, and the destination denotes the secondary system. An active mirror must have exactly one primary and exactly one secondary.
|
Important: A single system can contain source volumes and CGs (mirroring to another system) and destination volumes and CGs (mirroring from another system). Peers in a source role and peers in a destination role on the same system must belong to different mirror couplings.
|
The following mirroring role status options are available:
•Designations:
– Primary: The designation of the source peer, which is initially assigned the source role.
– Secondary: The designation of the target peer, which initially plays the destination role.
•Role status:
– Source: Denotes the peer with the source data in a mirror coupling. Such peers serve host requests and are the source for synchronization updates to the destination peer. Destination and source roles can be switched by using the mirror_switch_roles command if the status is synchronized for synchronous mirror and it is in an RPO_OK state for an asynchronous mirror.
For synchronous and asynchronous mirroring, the source can be changed (by using the mirror_change_role command) to a destination if the status is inactive.
– Destination: Denotes the target peer in a mirror. Such peers do not serve host write requests and accept synchronization updates from a corresponding source. A destination LUN can be accessed in read-only mode by a host.
Consistency group within an XIV or IBM Spectrum Accelerate
With mirroring (synchronous or asynchronous), the major reason for consistency groups is to handle many mirror pairs as a group (mirrored volumes are consistent). Instead of dealing with many mirror pairs individually, consistency groups simplify the handling of those related pairs.
|
Important: If your mirrored volumes are in a mirrored consistency group, you cannot perform mirroring operations, such as deactivate or change_role on a single volume basis. If you want to do perform this task, you must remove the volume from the consistency group.
For more information, see IBM XIV Storage System Business Continuity Functions, SG24-7759.
|
Consistency groups also play an important role in the recovery process. If mirroring was suspended (for example, because of complete link failure), data on different destination volumes at the remote system is consistent. However, when the links are up again and resynchronization process is started, data that is spread across several destination volumes is not consistent until the source state reaches the synchronized state.
To preserve the consistent state of the destination volumes, the system automatically creates a snapshot of each destination volume and keeps it until the remote mirror volume pair is synchronized. That is, the snapshot is kept until all pairs are synchronized to enable restoration to the same consistent point in time. If the remote mirror pairs are in a consistency group, the snapshot is taken for the whole group of destination volumes and the snapshots are preserved until all pairs are synchronized. Then, the snapshot is deleted automatically.
There are cases where storage requirements go beyond a single system. IBM Hyper-Scale Consistency extends the individual system crash consistency by allowing you to establish a volume consistency group that spans multiple systems. This configuration is called a Cross Consistency Group (XCG). For more information, see IBM Hyper-Scale in XIV Storage, REDP-5053.
7.2.2 Operational procedures
Mirroring operations feature the following activities:
•Configuration
Local and remote replication peers are defined by an administrator who specifies the source and destination peers roles. These peers can be volumes or consistency groups. The secondary peer provides a backup of the primary.
•Initialization
Mirroring operations begin with a source volume that contains data and a formatted destination volume. The first step is to copy the data from the source volume (or CG) to the destination volume (or CG). This process is called initialization. Initialization is performed once in the lifetime of a mirror. After it is performed, both volumes or CGs are considered to be synchronized to a specific point-in-time.
The completion of initialization marks the first point-in-time that a consistent source replica on the destination is available. Details of the process differ depending on the mirroring mode (synchronous or asynchronous). For more information, see IBM XIV Storage System Business Continuity Functions, SG24-7759.
•Offline initialization
Offline initialization operation begins with a source volume that contains data and a destination volume, which also contains data and is related to this same source. At this step, only different chunks are copied from the source to its destination. Offline initialization can be run whenever a mirror pair was suspended or when the mirror type changes from asynchronous to synchronous.
•Mirror mode switching
The toggling between synchronous and asynchronous modes implies the deactivation of the incumbent mirror mode, the deletion of the mirror pair and the respective snapshots on both ends, and unlocking of the destination mirror. Then, the new mode is selected and a new mirror relationship is created between the peers. By using the offline initialization, only the new data that was written to the primary since the deletion of the original mirror is copied over. Therefore, the toggling between the two operational modes does not require a redundant full copy.
•Ongoing operation
After the initialization process is complete, normal mirroring operations begin.
In synchronous mirroring, normal ongoing operation means that all data that is written to the primary volume or CG is first mirrored to the destination volume or CG. At any time, the source and destination volumes or CGs are identical except for any unacknowledged (pending) writes.
In asynchronous mirroring, ongoing operation means that data is written to the source volume or CG and is replicated to the destination volume or CG at specified intervals.
•Monitoring
The storage system effectively monitors the mirror activity and places events in the event log for error conditions. Alerts can be set up to notify the administrator of such conditions. You must set up SNMP trap monitoring tools or email notification to be informed about abnormal mirroring situations.
•Handling communication failures
There are instances when the communication between the sites might break down. The source continues to serve host requests as the synchronous mirroring is based on best effort in attempt to minimize the effect on the hosts. Upon recovery from a link down incident, the changed data is copied over and mirroring is resumed. Events are generated for link failures.
•Role switching
If required, mirror peer roles of the destination and source can be switched. Role switching is always started at the source site. This process is often done for maintenance operations or because of a test drill that verifies the disaster recovery (DR) procedures. Use role switching cautiously, especially with asynchronous mirroring. When roles are switched for an asynchronous mirror, data can be lost for an interval up to the RPO time because the remote site is typically lagging (in time) for a specific asynchronous pair.
Role switching in the case of synchronous mirror is designed so that no data loss can occur. Role switching should be used only for cases, such as a catastrophic host failure at the source site when the pairing is intact, but there were no write operations to the source since the last sync job was completed.
•Role changing
In a disaster at the primary site, the source peer might fail. To allow read/write access to the volumes at the remote site, the volume’s role must be changed from destination to source. A role change changes only the role of the system volumes or CGs to which the command was addressed. Remote mirror peer volumes or CGs are not changed automatically. Therefore, changing roles on both mirror sides if mirroring is to be restored is important (if possible).
7.2.3 Mirroring status
The mirroring status is affected by several factors, such as the links between the XIV or IBM Spectrum Accelerate systems or the initialization state.
Link status
The link status reflects the connection from the source to the destination volume or CG. A link has a direction (from local site to remote or vice versa). A failed link or a failed secondary system both result in a link error status. The link state is one of the factors that determines the mirror operational status. Links can be in one of the following states:
•OK: Link is up and is functioning
•Error: Link is down
Figure 7-5 shows how the link status is reflected in the main XIV GUI.
Figure 7-5 Red indicates link down, green indicates link up
Figure 7-6 shows how the link status is reflected in the XIV GUI when the link is selected.
Figure 7-6 Connectivity details
If several links (at least two) are in one direction and one link fails, this state often does not affect mirroring if the bandwidth of the remaining link is high enough to keep up with the data traffic.
Monitoring the link utilization
The mirroring bandwidth of the links must be high enough to cope with the data traffic that is generated by the changes on the source volumes. Before mirroring is set up during the planning phase, monitor the write activity to the local volumes. The bandwidth of the links for mirroring must be as large as the peak write workload.
After mirroring is implemented, periodically monitor the links usage. By using the statistics windows, you can select targets to show the data traffic to remote systems, as shown in Figure 7-7.
Figure 7-7 Monitoring link usage, with pop-up fly-over shown
Mirror operational status
Mirror operational status is defined as operational or non_operational and includes the following features:
•Mirroring is operational in the following situations:
– The activation state is active.
– The link is UP.
– Both peers have different roles (source or destination).
– The mirror is active.
•Mirroring is non_operational in the following situations:
– The mirror is inactive.
– The link is in an error state or deactivated (link down).
Synchronous mirroring states
|
Note: This section applies to synchronous mirroring only.
|
The synchronization status reflects whether the data of the destination’s volume is identical to the source’s volume. Because the purpose of the remote mirroring feature is to ensure that the destination’s volume is an exact copy of the source’s volume, this status indicates whether this objective is being achieved.
The following states or statuses are possible:
•Initializing
The first step in remote mirroring is to create a copy of all the data from the source volume or CG to the destination volume or CG. During this initial copy phase, the status remains initializing.
•Synchronized (source volume or CG)/consistent (destination volume or CG)
This status indicates that all data that was written to the source volume or CG also was written to the destination volume or CG. Ideally, the source and destination volumes or CGs must always be synchronized. However, this synchronization does not always indicate that the two volumes are identical in a disaster. There are situations when there might be a limited amount of data that was written to one volume, but that was not yet written to its peer volume. This situation means that the write operations are not yet acknowledged to the respective hosts. Such writes are known as pending writes or data in flight.
•Unsynchronized (source volume)/inconsistent (destination volume)
After a volume or CG completes the initializing stage and achieves the synchronized status, it can become unsynchronized (source) and inconsistent (destination). This status occurs when it is not known whether all the data that was written to the source volume also was written to the destination volume. This status can occur in the following cases:
– The communications link is down and as a result certain data might be written to the source volume, but was not yet written to the destination volume.
– Secondary system is down. This issue is similar to communication link errors because in this state, the primary system is updated, whereas the secondary is not.
– Remote mirroring is deactivated. As a result, certain data might be written to the source volume and not to the secondary volume.
The system tracks the partitions that were modified on the source volumes. When the link is operational again or the remote mirroring is reactivated, these changed partitions are sent to the remote system and applied to the respective destination’s volumes.
Asynchronous mirroring states
|
Note: This section applies to asynchronous mirroring only.
|
The mirror state can be in one of the following states:
•Inactive
The synchronization process is disabled. A mirror can be deleted in this state.
•Initializing
The initial copy is not yet complete. Synchronization does not start until the initialization completes. The mirror cannot be deleted during this state.
|
Important: When there is an unstable data link, the initialization can restart. In this case, the progress bar returns to the left side of the display. However, this state does not mean that the initialization is starting again from the beginning. When a mirror initialization is restarted, the initialization resumes where it left off and the progress bar displays the percent complete of the remaining data to copy, not the percentage of the full volume.
|
When initialization is complete, the synchronization process is enabled. Then, sync jobs can be run and data can be copied between source and destination. The following synchronization states are possible:
•RPO_OK: Synchronization completed within the specified sync job interval time (RPO).
•RPO_Lagging: Synchronization completed, but took longer that the specified interval time (RPO).
7.3 Remote mirroring usage
Remote mirroring solutions can be used to address many types of failures and planned outages. The failure scenarios vary and can be a result of events that affect a single system. The failure can stem from events that affect an entire data center or campus. Worse, the failures also can be caused by events that affect a whole geographical region.
Figure 7-8 shows recovery plan solutions that provide protection from single-system failures, local disasters, and regional disasters.
Figure 7-8 Disaster recovery protection levels
The following configurations are possible:
•Single-site high-availability remote mirroring configuration
Protection for the event of a failure or planned outage of a system (single-system failure) can be provided by a zero-distance high-availability (HA) solution, including another storage system in the same location (zero distance). The typical use of this configuration is a synchronous mirroring solution that is part of a HA clustering solution that includes servers and storage systems.
Figure 7-9 shows a single-site HA configuration in which both systems are in the same data center.
Figure 7-9 Single site HA configuration
•Metro region remote mirroring configuration
Protection during a failure or planned outage of an entire location (local disaster) can be provided by a metro distance disaster recovery solution, including another system in a different location within a metro region. The two systems might be in different buildings on a corporate campus or in different buildings within the same city (typically up to approximately 100 km apart). The typical use of this configuration is a synchronous mirroring solution. Figure 7-10 shows a metro region disaster recovery configuration.
Figure 7-10 Metro region disaster recovery configuration
•Out-of-region remote mirroring configuration
Protection during a failure or planned outage of an entire geographic region (regional disaster) can be provided by a global distance disaster recovery solution, including another system in a different location outside the metro region. The two locations might be separated by up to a global distance. The typical use of this configuration is an asynchronous mirroring solution. Figure 7-11 shows an out-of-region disaster recovery configuration.
Figure 7-11 Out-of-region disaster recovery configuration
7.3.1 Use of snapshots
Snapshots can be used with remote mirroring to provide copies of production data for business or IT purposes. Moreover, when used with remote mirroring, snapshots provide protection against data corruption. As with any continuous or near-continuous remote mirroring solution, remote mirroring cannot protect against software data corruption because the corrupted data is copied as part of the remote mirroring solution. However, the snapshot function provides a point-in-time image that can be used for a rapid recovery if software data corruption occurs after the snapshot was taken. The snapshot can be used in combination with remote mirroring, as shown in Figure 7-12.
Figure 7-12 Combining snapshots with remote mirroring
Recovery by using a snapshot requires deletion and re-creation of the mirror. Snapshots can consist of the following components:
•Snapshot (within a single system)
Protection for the event of software data corruption can be provided by restoring the volume to a healthy point-in-time snapshot. The snapshot can be backed up, if needed.
•Local snapshot plus remote mirroring configuration
A snapshot of the production (local) volume can be used in addition to remote mirroring of the production volume when protection from logical data corruption is required in addition to protection against failures and disasters. The extra snapshot of the production volume provides a quick restoration to recover from data corruption. An extra snapshot of the production (local) volume can also be used for other business or IT purposes, such as reporting, data mining, development, and testing.
Figure 7-13 shows a local snapshot plus remote mirroring configuration.
Figure 7-13 Local snapshot plus remote mirroring configuration
•Remote snapshot plus remote mirroring configuration
A snapshot of the consistent replicated data at the remote site can be used in addition to remote mirroring to provide an extra consistent copy of data. This copy can be used for business purposes, such as data mining, reporting. The copy also can be used for IT purposes, such as remote backup to tape or development, test, and quality assurance. Figure 7-14 shows a remote snapshot plus remote mirroring configuration.
Figure 7-14 Remote snapshot plus remote mirroring configuration
7.4 Remote mirroring configurations
The remote mirroring configurations that are described in this section are the fundamental building blocks of remote mirroring solutions and usage scenarios.
7.4.1 Mirroring target systems
Remote mirroring copies data from a peer on one system to a peer on another system (the mirroring target system). The basic underlying mirroring relationship is a one-to-one relationship between two peers, but systems can be connected in the following ways:
•Target configuration: One-to-one
The most typical remote mirroring configuration is a one-to-one relationship between a local system (production system) and a remote system (DR system). This configuration is typical where there is a single production site and a single DR site.
During normal remote mirroring operation, one system (at the DR site) is active as a mirroring target. The other system (at the local production site) is active as a mirroring target only when it becomes available again after an outage and a change of roles between the production and the DR site. Data changes that are made while production is running on the remote (DR) site are copied back to the original production site.
In a configuration with two identically provisioned sites, production might be periodically switched from one site to another as part of normal operations. The system that is the active mirroring target is switched at the same time. The mirror_switch_roles command allows for switching roles in synchronous and asynchronous mirroring. There are special requirements for doing so with asynchronous mirroring.
•Target configuration: Synchronous and asynchronous one-to-one
XIV and IBM Spectrum Accelerate support synchronous and asynchronous mirroring (for different mirror couplings) on the same system; therefore, a single local system can have certain volumes that are synchronously mirrored to a remote system. Other volumes are asynchronously mirrored to the same remote system, as shown in Figure 7-15. Highly response-time-sensitive volumes can be asynchronously mirrored and less response-time-sensitive volumes can be synchronously mirrored to a single remote system.
Figure 7-15 Synchronous and asynchronous peers
•Target configuration: Fan-out
A single local (production) system can be connected to two remote (DR) systems in a fan-out configuration. Both remote systems can be at the same location or each of the target systems can be at a different location. Certain volumes on the local system are copied to one remote system and other volumes on the same local system are copied to a different remote system. This configuration can be used when each system at the DR site has less available capacity than the system at the local site.
• Target configuration: Synchronous and asynchronous fan-out
XIV and IBM Spectrum Accelerate support synchronous and asynchronous mirroring (for different mirror couplings) on the same system; therefore, a single local system can have certain volumes that are synchronously mirrored to a remote system at a metro distance. Other volumes are asynchronously mirrored to a remote system at a global distance, as shown in Figure 7-16. This configuration can be used when higher priority data is synchronously mirrored to another system within the metro area, and lower priority data is asynchronously mirrored to a system within or outside the metro area.
Figure 7-16 Synchronous and asynchronous fan-out
•Target configuration: Fan-in
Two (or more) local systems can have peers that are mirrored to a single remote system in a fan-in configuration. This configuration must be evaluated carefully and used with caution because there is a risk of overloading the single remote system. The performance capability of the single remote system must be carefully reviewed before a fan-in configuration is implemented.
This configuration can be used in situations where there is a single disaster recovery data center that is supporting multiple production data centers, or when multiple systems are mirrored to a single system at a service provider.
•Target configuration: Bidirectional
Two different systems can have different volumes that are mirrored in a bidirectional configuration. This configuration can be used for situations where there are two active production sites and each site provides a DR solution for the other. Each system is active as a production system for certain peers and as a mirroring target for other peers.
7.4.2 Setting the maximum initialization and synchronization rates
IBM Spectrum Accelerate and XIV systems allow a user-specifiable maximum rate (in MBps) for remote mirroring coupling initialization, a different user-specifiable maximum rate for normal sync jobs, and another rate for resynchronization. The initialization, sync job, and resynchronization rates are specified for each mirroring target by using the target_config_sync_rates XCLI command.
The actual effective initialization or synchronization rate also depends on the number and speed of connections between the systems. The maximum initialization rate must be less than or equal to the maximum sync job rate (asynchronous mirroring only), which must be less than or equal to the maximum resynchronization rate.
|
Important: In normal mirror operations, the rates are cumulative. For example, if initialization, synchronous, and asynchronous operations are all active, the amount of data the system attempts to send is the sum of those three values.
|
The rates use the following default values:
•Maximum initialization: 100 MBps
•Maximum sync job: 300 MBps
•Maximum resync: 300 MBps
7.5 IBM Spectrum Accelerate remote mirroring scenarios
The following sections describe the basic steps that are used to set up a remote mirroring environment. For more information, see IBM XIV Storage System Business Continuity Functions, SG24-7759. For similar information about administration of remote mirroring with Hyper-Scale Manager, see IBM FlashSystem A9000 and A9000R Replication Solutions, REDP-5401.
This section includes the following topics:
•Setting up a new remote mirroring solution after migrating data
•Use of remote mirroring to migrate data and reversing the direction of remote mirroring to provide a backup of the migrated volumes
•Recovering from a site failure
•Temporary local site outage
•Deactivating mirror coupling
7.5.1 Basic remote mirroring setup scenario
This example assumes that there are volumes that are defined on an IBM Spectrum Accelerate system that are created as volumes or that data was migrated to the system by using the Data Migration function. These volumes can now be replicated to another system by using the following process:
1. Define the target system.
2. Create the connectivity by connecting mirroring ports.
3. Define mirror coupling peer volumes.
4. Add volumes to a mirrored Consistency Group.
Defining the mirroring target systems
To connect two systems (XIV or IBM Spectrum Accelerate) for remote mirroring, each system must be defined to be a mirroring target of the other. A mirroring target is a system with volumes that receive data copied through remote mirroring.
Verify that the source and target systems have iSCSI ports defined, as shown in Figure 7-17.
Figure 7-17 Verifying ISCSI connectivity
Defining a mirroring target for a system involves selecting the type of mirroring, naming the target, and specifying the target iSCSI initiator name, as shown in Figure 7-18.
Figure 7-18 Creating the target definition
Connecting mirroring ports
After defining remote mirroring targets, one-to-one connections must be made between ports on each system.
|
Important: If the IP network includes firewalls between the mirrored systems, TCP port 3260 (iSCSI) must be open within firewalls so that iSCSI replication can work.
|
Use a minimum of two connections (with each of these ports in a different module). As shown in Figure 7-19, the data flow starts from the production system (on the left) and goes towards mirroring target system (on the right) during normal operation.
Figure 7-19 Defining iSCSI port connections
The data flow is reversed when production is running at the disaster recovery site (on the right) and changes are being copied back to the original production site (mirroring target is on the left).
|
Note: For asynchronous mirroring over iSCSI links, a reliable, dedicated network must be available. It requires consistent network bandwidth and a non-shared link.
|
Defining mirror coupling and peers
After the remote mirroring targets are defined, a coupling or mirror can be defined, which creates a mirroring relationship between two peers.
The two peers in the mirror coupling can be two volumes (volume peers) or two consistency groups (CG peers). Select the source volume, target system, pool, and the type of mirroring. The target volume can be created automatically in the target pool and the mirror relationship can be activated, as shown in Figure 7-20.
Figure 7-20 Defining mirror volume coupling
Defining a Mirrored Consistency Group with the same characteristics as a group of volumes allows for easily controlling remote mirroring operations simultaneously with all volumes in the CG, as shown in Figure 7-21.
Figure 7-21 Defining a consistency group
Figure 7-22 shows creating a mirrored CG.
Figure 7-22 Creating a mirrored consistency group
After a mirrored CG is created, volumes can be added by dragging them within the XIV GUI.
Each of the two peers in the mirroring relationship is given a designation and a role. The designation indicates the original or normal function of each of the two peers: primary or secondary. The peer designation does not change with operational actions or commands. If necessary, the peer designation can be changed by the user.
The role of a peer indicates its current (perhaps temporary) operational function (source or destination). The operational role of a peer can change as the result of user commands or actions. Peer roles typically change during DR testing or a true disaster recovery and production site switch.
When a mirror coupling is created, the first specified peer (for example, the volumes or CG at site 1, as shown in Figure 7-23 on page 200) is the source for data to be replicated to the target system; therefore, it is given the primary designation and the source role.
|
Important: A consistency group that is to be mirrored must not contain any volumes when the CG coupling is defined or the coupling does not allow it to be defined.
|
The second specified peer (or automatically created by the storage system) when the mirroring coupling is created is the target of data replication; therefore, it is given the secondary designation and the destination role.
When a mirror coupling relationship is first created, no data movement occurs.
Activating a mirror coupling
When a mirror coupling is first activated, all data on the source is copied to the destination (normal initialization) or verified to be on the destination, and only changed data is copied (offline initialization). This process is referred to as initialization. Remote mirroring copies volume identification information (that is, physical volume ID/PVID) and any actual data on the volumes. Space that is not used is not copied.
Initialization might take a significant amount of time if a large amount of data is on the source when a mirror coupling is activated. The rate for this initial copy of data can be specified by the user. The speed of this initial copy of data is also affected by the connectivity and bandwidth (number of links and link speed) between the primary and secondary systems.
As an option to remove the effect of distance on initialization, mirroring can be initialized with the target system installed locally. The target system can then be disconnected after initialization, shipped to the remote site, reconnected, and the mirroring reactivated.
A second option to avoid the effect of distance on initialization is by using an offline initialization. The peers can be synchronized locally and the DR system moved to its remote site, or if the target system is already at a remote location with limited WAN capabilities, apply an image backup of the source volume onto the destination, and then activate the offline mirroring initialization.
If a backup tape is physically transported to the remote site, it must be an image backup. File-level backups do not function as expected, and result in retransmission of 100% of the data in the pairing. The mirror pairing is defined normally, with the addition of specifying the offline init option when making the definition of the pairing.
If a remote mirroring configuration is set up when a volume is first created (that is, before any application data is written to the volume), initialization is fast.
When a consistency group mirror coupling is created, the CG must be empty so there is no data movement and the initialization process is fast.
The mirror coupling status at the end of initialization differs for synchronous mirroring and asynchronous mirroring; however, in either case, a consistent set of data is at the remote site when initialization is complete.
Figure 7-23 shows a diagram of active mirror coupling.
Figure 7-23 Active mirror coupling
Adding volume mirror coupling to consistency group mirror coupling
After a volume mirror coupling completes initialization, the source volume can be added to a mirrored CG in the same storage pool. Each mirroring type has extra constraints, such as same role, target, and schedule. The destination volume is automatically added to the consistency group on the remote system.
As shown in Figure 7-24, three active volume couplings that completed initialization are moved into the active mirrored consistency group. This move is done by dragging the couplings within the GUI interface.
Figure 7-24 Consistency group mirror coupling
One or more extra mirrored volumes can be added to a mirrored consistency group later in the same way.
It is also important to understand that in a CG, all volumes feature the same role. Also, consistency groups are handled as a single entity. In asynchronous mirroring, a delay in replicating a single volume affects the status of the entire CG.
Adding a mirrored volume to a mirrored consistency group
Before the process is started, ensure that the following prerequisites are met:
•Volume and CG must be associated with the same pool.
•Volume is not already part of a CG.
•Command must be issued on the source CG only.
•Command must not be run during initialization of volume or CG.
•The following volume mirroring settings must be identical to the CG settings:
– Mirroring type
– Mirroring role
– Mirroring status
– Mirroring target
– Target pool
•The volume synchronization status and mirrored CG synchronization status are RPO OK for asynchronous mirroring or Synchronized for synchronous mirroring.
To add a volume mirror to a mirrored consistency group (for instance, when an application needs extra capacity), complete the following steps:
1. Define volume mirror coupling from the extra source volume at XIV 1 to the destination volume at XIV 2.
2. Activate remote mirroring from the extra source volume at XIV 1 to the destination volume at XIV 2.
Monitor the initialization until it is complete. Volume coupling initialization must complete before the coupling can be moved to a mirrored CG.
3. Add the source volume at XIV 1 to the source consistency group at XIV 1. The extra destination volume at XIV 2 is automatically added to the destination consistency group at XIV 2.
7.5.2 Migrating data by using remote mirroring
Remote mirroring can be used to migrate data to a newly installed IBM Spectrum Accelerate system if the source data is on an XIV system. This process is essentially the same as described in the 7.5.1, “Basic remote mirroring setup scenario” on page 194 and includes the following steps:
1. Define the target system and XIV connectivity.
2. Define mirror coupling peer volumes.
3. Create a mirrored consistency group.
4. Add volumes to the mirrored consistency group.
Adding the volumes to a consistency group is optional; however, doing so is convenient because the set of volumes can be treated as a group, which simplifies the following processes.
5. Wait for synchronization to complete.
6. Switch the roles of the volumes.
The migrated volumes are then available for host activity with a backup copy at the original site.
Defining the target system and system connectivity is shown in Figure 7-25.
Figure 7-25 Defining the target system and system connectivity
Defining the mirror coupling peer volumes is shown in Figure 7-26.
Figure 7-26 Defining mirror coupling peer volumes
Figure 7-27 shows creating a mirror CG.
Figure 7-27 Creating a mirror Consistency Group
Figure 7-28 shows adding mirrored volumes to the CG.
Figure 7-28 Adding mirrored volumes to the consistency group
When mirroring is active and synchronized (consistent), the source and destination roles of mirrored volumes or consistency groups can be switched simultaneously. Role switching is typical for returning mirroring to the normal direction after changes are mirrored in the reverse direction after a production site switch. Role switching is also typical for any planned production site switch.
Host server write activity and remote mirroring activity must be paused briefly before and during the role switch. Also, at least one sync job must complete before the switch to ensure the expected point in time copy of the data exists in the case of asynchronous mirroring. Figure 7-29 shows the switch mirroring roles.
Figure 7-29 Switching mirroring roles
The volumes are now migrated to the new IBM Spectrum Accelerate system and mirroring is active to the XIV system. Hosts can now access the migrated volumes and any changes are mirrored to the original source volumes.
7.5.3 Recovering from a site failure
One of the reasons for remote mirroring is to keep a copy of data in a different location if something happens to the source site. If there is a real disaster, the systems at the local site might need to be replaced or the IBM Spectrum Accelerate system might need to be redeployed. In such circumstances, the production activity moves to the remote site. When the local site is restored, data can be migrated back, as described in 7.5.2, “Migrating data by using remote mirroring” on page 201.
System failure at the local site
Figure 7-30 shows that the source system is no longer operational. The mirror definitions are still on the target system and the role of those volumes can be changed to source.
Figure 7-30 Local site system failure
Figure 7-31 shows the details of the connectivity when a connection is selected.
Figure 7-31 Connecting to the local site failed because the system is no longer available
Changing role of destination volume or CG
The system at the recovery site must be prepared to become the production system. To complete this preparation, the role of the target volumes are changed to source volumes and then host systems can access the last consistent copy of the production data.
When mirroring is active, the destination volume or CG is locked and write access is prohibited. To allow write access to a destination peer if there is failure or the source is unavailable, the destination volume role must be changed to the source role, as shown in Figure 7-32.
Figure 7-32 Changing role of destination volume or CG
Changing the role of a volume from destination to source allows the volume to be accessed. In synchronous mirroring, changing the role also starts metadata recording for any changes that are made to the volume. This metadata can be used for resynchronization (if the new source volume remains the source when remote mirroring is reactivated). In asynchronous mirroring, changing a peer’s role automatically reverts the peer to its last-replicated snapshot.
Deleting mirror coupling definitions
When replacing the source system because of failure, the original mirror definitions must be deleted and can be re-created with the new system during the process of migrating data by using remote mirroring.
When a mirror coupling is deleted, all metadata and mirroring definitions are deleted, and the peers do not have any relationship at all (see Figure 7-33). However, any volumes and consistency groups mirroring snapshots remain on the local and remote systems. To restart mirroring, you can use offline initialization instead of a full copy of data. For more information, see IBM XIV Storage System Business Continuity Functions, SG24-7759.
Figure 7-33 Deleting mirror coupling definitions
The target definition and XIV connectivity can also be deleted, as shown in Figure 7-34.
Figure 7-34 Deleting the target definition and XIV connectivity
Typical usage of mirror deletion is also a one-time data migration that uses remote mirroring. This process includes deleting the mirror couplings after the migration is complete.
7.5.4 Recovering from a temporary site outage
A temporary site outage might require moving production host activity to the remote location. This move can be accomplished by using the procedure that is described in 7.5.3, “Recovering from a site failure” on page 205 for recovering from a site failure. In this case, the local system is restored and mirroring can be resumed.
Because the most current data is at the remote site, the changes must be copied to the original site. The remote site had its role changed to the source role. Therefore, the local system must be changed to a target role. When the destination to a source is changed, the former source first must be changed to the destination role (upon recovery of the primary site) before the secondary role is changed back from source to destination.
In synchronous mirroring, changing a peer role from source to destination allows the destination to accept mirrored data from the source. It also causes deletion of metadata that was used to record any changes while the peer had the source role.
In asynchronous mirroring, changing a peer's role automatically reverts the peer to its last-replicated snapshot.
Removing a mirrored volume from a mirrored consistency group
If a volume in a mirrored consistency group is no longer being used by an application or if actions must be taken against the individual volume, it can be dynamically removed from the consistency group.
In this scenario, you must change the role of the target volumes and because there is no connectivity to the local site, the swap roles command cannot be used to switch both roles. Therefore, the volumes must be removed from the consistency group and then changed from a target role to a source role.
The same process is followed at the local site after it is recovered to change the role from source to target to send the most recent data from the remote site back to the local site. The role of the consistency groups also must be changed by using the same process that is used with the volumes.
The volumes can be added to the consistency group when they become synchronized. Then, the roles are swapped and the production host activity is restored to the local site.
As shown in Figure 7-35, one volume is removed from the example mirrored XIV consistency group with three volumes.
Figure 7-35 Removing a mirrored volume from a mirrored CG
Both peers might temporarily have the source role when a failure at site 1 results in a true disaster recovery production site switch from site 1 to site 2. When site 1 becomes available again and you must switch production back to site 1, the production changes that were made to the volumes at site 2 must be resynchronized to the volumes at site 1.
To complete this process, the peers at site 1 must change their role from source to destination, as shown in Figure 7-36.
Figure 7-36 Changing role to destination volume and CG
Reactivation, resynchronization, and reverse direction
When mirroring is reactivated in the reverse direction, changes that are recorded at the secondary peers are copied to the primary peers, as shown in Figure 7-37.
Figure 7-37 Reactivation and resynchronization
Changing role of a consistency group
The volumes are removed from the consistency group and their roles are reversed. The role of the consistency group must match the roles of the volumes before they can be added to the consistency group. The same process that is used for changing the volume roles can be used with the consistency group roles. Confirm the change to Source, as shown in Figure 7-38.
Figure 7-38 Changing role of the remote consistency group
Confirm changing the role to Destination, as shown in Figure 7-39.
Figure 7-39 Changing role of the local consistency group
Adding volumes to the consistency group
When the volumes and the consistency group are in the Synchronized state, the volumes can be added to the consistency group, as shown in Figure 7-40. Then, the consistency group can swap roles when production host activity is ready to return to the local site.
Figure 7-40 Activating the consistency group and add volumes
Figure 7-41 shows the Switch Roles confirmation option.
Figure 7-41 Swapping roles to restore then normal direction of remote mirroring
7.5.5 Temporarily deactivating mirroring
Mirror deactivation and reactivation in the same direction is described in the following examples:
•Remote mirroring is temporarily deactivated because of communication failure and then automatically reactivated by the system when communication is restored.
•Remote mirroring is temporarily deactivated to create an extra copy of consistent data at the secondary.
•Remote mirroring is temporarily deactivated by user action during peak load in an environment with constrained network bandwidth.
•Planned deactivation of XIV remote mirroring can be done to suspend remote mirroring during a planned network outage or DR test.
Deactivating mirror coupling: Change recording
A mirror coupling can be deactivated by a user command. In this case, the mirror changes to standby mode, as shown in Figure 7-42.
Figure 7-42 Deactivating mirror coupling: Change recording
During standby mode, a consistent set of data is available at the remote site. The currency of the consistent data ages in comparison to the source volumes, and the gap increases while mirroring is in standby mode.
In synchronous mirroring during standby mode, the storage system (XIV or IBM Spectrum Accelerate) metadata is used to note which parts of a source volume are changed but are not yet replicated to the destination volume (because mirroring is not currently active). The actual changed data is not retained in cache; therefore, there is no danger of exhausting cache while mirroring is in standby mode.
When synchronous mirroring is reactivated by a user command or communication is restored, the metadata is used to resynchronize changes from the source volumes to the destination volumes. Mirroring records changes for source volumes only. If you want to record changes to both peer volumes while mirroring is in standby mode, change the destination volume to a source volume.
In asynchronous mirroring, metadata is not used and the comparison between the most-recent and last-replicated snapshots indicates the data that must be replicated.
Mirror reactivation and resynchronization: Normal direction
When mirroring is in standby mode in synchronous mirroring, any changes to volumes with the source role are recorded in metadata. When mirroring is reactivated, changes that are recorded in metadata for the current source volumes are resynchronized to the current destination volumes, as shown in Figure 7-43.
Figure 7-43 Mirror reactivation and resynchronization: Normal direction
The rate for this resynchronization of changes can be specified by the user in MBps by using the XCLI target_config_sync_rates command.
When mirroring is reactivated in the normal direction, changes that are recorded at the primary peers are copied to the secondary peers.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.