Performance and monitoring
This chapter describes the factors that determine and influence the performance of the IBM TS7700. It also describes what actions to take, when necessary, to improve TS7700 performance. In addition, this chapter also covers the possible settings, alerts, and messages that should be considered for exception handling and automation.
 
Note: At the time of updating this book the performance measurements were not available for TS7700 V4.1.2. The IBM TS7700 R4 (TS7760) Performance White Paper contains the most recent updates:
This chapter includes the following sections:
11.1 Overview
R4.0 introduces the TS7760. The TS7760 models (TS7760D and TS7760T) replace all previous models, including the TS7740.
With R3.2 the TS7700 Tape-Attach was introduced. In general, the tape attach models are based on the same physical hardware than the disk-only models, and the basic performance numbers are identical. In addition to the TS770D, the TS7700T needs to write data from the cache to the backend drives, needs to process recalls, and needs additional resources for reclaims. It is necessary to understand how these actions affect the overall performance of the TS7700T.
R3.1 introduced the next generation 8-gigabit (Gb) Fibre Channel connection (FICON) adapters. The TS7700 4-port 8 Gb FICON adapter is the same type as the IBM System Storage DS8700 family 8-port 8 Gb FICON adapter. This adapter provides the same cyclic redundancy check (CRC) protection and inline compression as previous TS7700 FICON adapters provided.
R3.1 supports two ports per 8-Gb FICON adapter only. In a fully populated TS7700, with four 8-Gb adapters, there are eight ports available for TS7700 host connections.
Each port on the 8-Gb FICON adapter supports 512 logical paths, which are twice the number of logical paths that are supported by the 4-Gb FICON adapters. When fully configured with 8 8-Gb FICON channels, the TS7700 supports 4096 logical paths.
This means that you have more flexibility when connecting large numbers of logical partitions (LPARs).
This chapter includes the newest overall performance information, especially for the TS7760 models:
•An overview of the shared tasks that are running in the TS7700 server
•A description of a TS7700 monitoring and performance evaluation methodology
•Understanding the speciality for the TS7700T regarding the different throttling impacts of CP0 and CPx
•Performance monitoring with the TS7700 GUI
•Additional information about performance alerting and thresholds
•Information about the handling of Sync-Deferred and Immediate-deferred copy handling
•A review of bulk volume information retrieval (BVIR) and VEHSTATS reporting
•VEHAUDIT and BVIRAUDIT usage
•A detailed description of the device allocation possibilities regarding the TS7700 capabilities
A brief overview of the tasks in the TS7700 is provided so that you can understand the effect that contention for these resources has on the performance of the TS7700.
The monitoring section can help you understand the performance-related data that is recorded in the TS7700. It discusses the performance issues that might arise with the TS7700. This chapter can also help you recognize the symptoms that indicate that the TS7700 configuration is at or near its maximum performance capability. The information that is provided can help you evaluate the options available to improve the throughput and performance of the TS7700.
Information about extra threshold alerting is provided to help you to implement automation-based monitoring in IBM z/OS. Scenarios are described to show the effect of various algorithms on z/OS and the TS7700 device allocation. These scenarios help you to understand how settings and definitions affect device allocation.
11.2 TS7700 performance characteristics
The TS7700 can provide significant benefits to a tape processing environment. In general, performance depends on such factors, such as total system configuration, Tape Volume Cache (TVC) capacity, the number of physical tape drives available to the TS7700, the number of channels, the read/write ratio, and data characteristics, such as blocksize and mount pattern.
You might experience deviations from the presented figures in your environment. The measurements are based on a theoretical workload profile, and cannot be fully compared with a varying workload. The performance factors and numbers for configurations are shown in the following pages.
Based on initial modeling and measurements, and assuming a 2.66:1 compression ratio, Figure 11-1 on page 616 shows the evolution in the write performance with the TS7700 family, which is also described in more detail in IBM TS7700 R4 (TS7760) Performance, WP102652. The following charts are for illustrative purposes only. Always use the most recently published performance white papers available on the Techdocs website at the following address:
Figure 11-1 shows write performance history.
Figure 11-1 VTS and TS7700 maximum host write throughput
Figure 11-1 shows the evolution of performance in the TS7700 IBM family compared with the previous member of the IBM Tape Virtualization family, the IBM Virtual Tape Server (VTS). All runs were made with 128 concurrent jobs, using 32 kibibyte (KiB) blocks, and queued sequential access method (QSAM) BUFNO = 20. Before R 3.2, volume size is 800 mebibytes (MiB), made up of 300 MiB volumes @ 2.66:1 compression. In R 3.2, the volume size is 2659 MiB (1000 MiB volumes @ 2.66:1 compression).
Figure 11-2 shows the read hit performance numbers in a similar fashion.
Figure 11-2 VTS and TS7700 maximum host read hit throughput
The numbers that are shown in Figure 11-2 were obtained with 128 concurrent jobs in all runs, each job using 32 KiB blocks, and QSAM BUFNO = 20. Before R 3.2, volume size is 800 MiB (300 MiB volumes @ 2.66:1 compression). Since R 3.2, the volume size is 2659 MiB (1000 MiB volumes @ 2.66:1 compression).
From a performance aspect, the architecture offers these important characteristics:
•With the selection of IBM DB2 as the central repository, the TS7700 provides a standard Structured Query Language (SQL) interface to the data, and all data is stored and managed in one place. DB2 also allows for more control over performance.
•The cluster design with virtualization node (vNode) and hierarchical data storage management node (hnode) provides increased configuration flexibility over the monolithic design of the VTS.
•The use of Transmission Control Protocol/Internet Protocol (TCP/IP) instead of FICON for site-to-site communication eliminates the requirement to use channel extenders.
11.3 Basic performance overview
Performance of TS7700 has several characteristics, and is influenced by many aspects. The following section gives a brief overview of the TS7700 performance aspects, and describes some of the dependencies. For more information about TS7700 performance, see the current version of the IBM Virtualization Engine TS7700 Series Best Practices - Understanding, Monitoring and Tuning the TS7700 Performance white paper:
These are the five major aspects that influence the overall performance:
•TS7700 components and task distribution
•Replication modes and grid link considerations
•Workload profile from your hosts
•Lifecycle Management of your data
•Parameters and customization of the TS7700
11.3.1 TS7700 components and task distribution
While writing scratch volumes, or premigrating and recalling virtual volumes from physical stacked volumes, hardware components are shared by tasks running on the TS7700. Some of these tasks represent users’ work, such as scratch mounts, and other tasks are associated with the internal operations of the TS7700, such as reclamation in a TS7700T and TS7740.
See Figure 11-3 for an overview of all of the tasks. The tasks that TS7700 runs, the correlation of the tasks to the components that are involved, and tuning points that can be used to favor certain tasks over others are all described.
Figure 11-3 Tasks that are performed by the TS7700
The tasks are in general the same for all models of the TS7700. For the TS770D, the backend tasks are not applicable.
All of these tasks share resources, especially the TS7700 Server processor, the TVC, and the physical tape drives attached to a TS7700T or TS7740. Contention might occur for these resources when high workload demands are placed on the TS7700. To manage the use of shared resources, the TS7700 uses various resource management algorithms, which can have a significant impact on the level of performance that is achieved for a specific workload.
In general, the administrative tasks (except premigration) have lower priority than host-related operations. In certain situations, the TS7700 grants higher priority to activities to solve a problem state, including the following scenarios:
•Panic reclamation: The TS7740 or TS7700T detects that the number of empty physical volumes has dropped below the minimum value, and reclaims need to be done immediately to increase the count.
•Cache fills with copy data: To protect from uncopied volumes being removed from cache, the TS7700T and TS7740 throttle data coming into the cache. For the TS7700T, this type of throttling occurs only to Host I/O related to the CPx partitions. Data that is written to CP0 is not throttled in this situation.
A complete description of the tasks processing can be found in the IBM Virtualization Engine TS7700 Series Best Practices - Understanding, Monitoring and Tuning the TS7700 Performance white paper:
The TS7700T is not yet reflected in that white paper version.
11.3.2 Grid considerations and replication modes
Data can be copied between the clusters by using the Synchronous, RUN (also known as Immediate), Deferred Copy, or Time Delayed Copy policy settings. Every one of these copy modes has specific characteristics, and influences your RPO for specific application workload.
In addition, the chosen copy mode might also have a direct influence on the performance of the TS7700 grid. Although some of the modes enable the copies to be run at a non-peak time (all types of deferred copies), other copy modes are enforced to run before job end (Synchronous, RUN).
This implies that resources from the TS7700 (cache bandwidth, CPU, and grid link bandwidth) need to be available at this point in time.
The following provide a quick recap of replication modes:
•Synchronous mode means that the data is copied instantly to a second cluster. Synchronous mode copies occur during the writes from the host, and are synchronized when a tape sync event occurs. Due to this behavior, synchronous mode copy has some performance benefits over RUN or deferred copy modes.
Typically, when the Rewind Unload (RUN) is sent, the Synchronous copy has already completed. With an immediate copy (RUN), the second copy is not started until the RUN command is received. In RUN copy mode, the rewind-unload at job end is held up until the received data is copied to the other cluster (or clusters).
•In Deferred Copy mode, data is queued for copying, but the copy does not have to occur before job end. Deferred copy mode allows for a temporarily higher host data rate than Synchronous or RUN copy mode, and can be useful for meeting peak workload demands. Consistently exceeding the capacity of your configuration can result in a copy queue, which cannot be depleted before the next workload peak. Deferred copies are controlled during heavy host I/O with Deferred Copy Throttling (DCT). For more information, see 11.8.5, “Tuning possibilities for copies: Deferred Copy Throttling” on page 661.
The number of concurrent copy tasks for deferred and RUN copies can be altered by an authorized user by using a host console Library Request command. When altering the copy tasks, consider the Grid network configuration and throughput capabilities to make the best use of the available resources and not over-commit the network or source cluster for copy activity.
The customer can influence the number of copies for concurrent deferred and RUN copies, but not for the number of concurrent Synchronous mode copies. In addition, mechanisms exist to control whether deferred copies are running during peak Host I/O. Be aware, that delaying deferred copies might have an impact on your RPO.
•The Time Delayed Replication mode is useful to produce copies to multiple TS7700 clusters only after a predefined timeline has expired. This might reduce the number of needed copies, but (if you specify hours after create/access) the copies can be produced in a different timeline. You cannot specify a specific start.
In certain situations, requested Synchronous mode copy or RUN copies cannot be processed. In this case (depending on your customization), the jobs will either not run, or they produce Synchronous-Deferred or Immediate-Deferred copies. These copies are processed later, when the situation is relieved. For more information about the Synchronous deferred and Immediate Deferred copies, see 11.15.2, “Handling Replication Exceptions” on page 694.
The copies are processed in the following order:
1. Synchronous-Deferred PG0
2. Synchronous-Deferred PG1
3. Immediate PG1
4. Immediate-Deferred PG0
5. Immediate-Deferred PG1
6. Deferred PG0
7. Deferred PG1
In the grid, extra tasks can be performed:
•Remote or Cross-cluster mounts:
 – Using another cluster’s cache
 – Another cluster that uses this cluster’s cache
•Cluster coordination traffic:
 – Ownership takeover
 – Volume attribute changes
 – Logical volume insert
 – Configuration
 
Clarification: Cross-cluster mounts to other clusters do not move data through local cache. Also, reclaim data does not move through the cache.
Cluster families and cooperative replication
Considering a composite library with two or more clusters at a local site and two or more members at a remote site. If more than one cluster needs a copy of a volume at the remote site, cluster families make it possible to send only one copy of the data across a long-distance grid link network. When deciding where to source a volume, a cluster gives higher priority to a cluster in its family over a cluster in another family.
Family members are given higher weight when deciding which cluster to prefer for TVC selection.
Members of a family source their copies within the family when possible. In this manner, data does not have to travel across the long link between sites, optimizing the use of the data link and shortening the copy time. Also, the family members cooperate among themselves, each pulling a copy of separate volume and exchanging them later among family members.
With cooperative replication, a family prefers retrieving a new volume that the family does not have a copy of yet, over copying a volume within a family. When fewer than 20 new copies are to be made from other families, the family clusters copy among themselves. Therefore, second copies of volumes within a family are deferred in preference to new volume copies into the family.
When a copy within a family is queued for 12 hours or more, it is given equal priority with copies from other families. This prevents family copies from stagnating in the copy queue.
For details about cluster families, see IBM Virtualization Engine TS7700 Series Best Practices -TS7700 Hybrid Grid Usage at:
11.3.3 Workload profile from your hosts
In general, there are two types of workloads:
•Planned workload: This type of workload is driven by predictable action, mostly batch jobs. These planned actions can be influenced by the operation team regarding the execution time of the jobs.
•Unplanned workload: The other type of workload is the user driven workload, for example, hierarchical storage management (HSM) or object access method (OAM) processing requests, and event-driven workload, for example, database log archiving or System Management Facilities (SMF) processing exists.
Unplanned read workload might have peaks that can affect on the one hand the response times of these actions (read / recall times). However, these actions can also influence the deferred copy times and, in a TS7740, the reclamation execution. Changes in the workload profile might affect the replication time of deferred copies and can lead to throttling situations. Therefore, review the performance charts of the TS7700 to identify workload profile changes, and to take appropriate performance tuning measurements if necessary.
11.3.4 Lifecycle Management of your data
This specific aspect is important for a TS7740 or TS7700T. Depending on your amount of data (and logical volumes) with a short expiration date, the TS7740/TS7700T needs to run more reclamation. This has an impact to your back-end drives and TS7740 and TS7700T processor cycles.
In a hybrid grid, this data can be placed in the TS7700D, and can be replicated with the Time Delayed copy mode. This can reduce the backend activities in the TS7740 and TS7700T. Using the delay premigration queue on a TS7700T can also reduce the back-end activities for migrate and reclaim.
11.3.5 Parameters and customization of the TS7700
The TS7700 offers you a broad variety of tuning possibilities, especially for the cache management, the replication activities, and the backend activities.
The following list describes some examples of TS7700 tuning activities:
•Preference group of the data (data is preferably in cache or not)
•Number of the tape cache partitions in a TS7700T
•Using the premigration delay feature in the TS7700T
•Premigration threshold and control of premigration tasks for TS7700T and TS7740
•Deferred Copy Throttling (to prioritize the host workload)
•Number of concurrent copy tasks
•Schedule for reclamation
•Number of physical volume pools
Consider that some of these activities have dependencies.
If you change the preconfigured values, review your adjustment with the performance monitoring tools.
For more information, see the IBM Virtualization Engine TS7700 Series Best Practices - Understanding, Monitoring and Tuning the TS7700 Performance white paper:
11.3.6 Terminology of throughput
A TS7700 disk-cache-only cluster has a fairly consistent workload throughout.
Because the TS7740 and TS7700T contain physical tapes to which the cache data is periodically written, recalls from tape to cache occur, and Copy Export and reclaim activities occur, the TS7740 and TS7700T exhibits four basic throughput rates:
•Peak write
•Sustained write
•Read-hit
•Recall
These four rates are described in the following sections.
Peak and sustained write throughput
For the TS7740 and TS7700T, a measurement is not begun until all previously written data has been copied, or premigrated, from the disk cache to physical tape. Starting with this initial condition, data from the host is first written into the TS7740 and TS7700T disk cache with little if any premigration activity taking place. This approach allows for a higher initial data rate, and is termed the peak data rate.
After a pre-established threshold of non-premigrated data is reached, premigration starts, which can reduce the host write data rate. This threshold is called the premigration priority threshold, and has a default value of 1600 GB. When a further threshold of non-premigrated data is reached, the incoming host activity is actively throttled to allow for increased premigration activity.
This throttling mechanism operates to achieve a balance between the amount of data coming in from the host and the amount of data being copied to physical tape. The resulting data rate for this mode of behavior is called the sustained data rate, and can theoretically continue on forever, given a constant supply of logical and physical scratch tapes.
This second threshold is called the premigration throttling threshold, and has a default value of 2000 GB. These two thresholds can be used with the peak data rate to project the duration of the peak period. Both the priority and throttling thresholds can be increased through a host command-line request, which is described later in this chapter.
Read-hit and recall throughput
Similar to write activity, there are two types of TS7740 and TS7700T read performance:
•Read-hit (also referred to as peak) occurs when the data that is requested by the host is in the disk cache.
•Recall (also referred to as read-miss) occurs when the data requested is no longer in the disk cache, and must be first read in from physical tape.
Read-hit data rates are typically higher than recall data rates.
Summary
The two read performance metrics, along with peak and sustained write performance, are sometimes referred to as the four corners of virtual tape performance. Performance depends on several factors that can vary greatly from installation to installation, such as number of physical tape drives, spread of requested logical volumes over physical volumes, location of the logical volumes on the physical volumes, and length of the physical media.
11.3.7 Throttling in the TS7700
Throttling is the mechanism adopted to control and balance several tasks that run at the same time within the TS7700, prioritizing certain tasks over others. These mechanisms are called upon only when the system reaches higher levels of utilization, where the components are used almost to their maximum capacity and bottlenecks start to show. The criteria balances the user needs with the vital resources that are needed for the TS7700 to function.
This control is accomplished by delaying the launch of new tasks and prioritizing more important tasks over the other tasks. After the tasks are dispatched and running, control over the execution is accomplished by slowing down a specific functional area by introducing calculated amounts of delay in the operations. This alleviates stress on an overloaded component, or leaves extra central processor unit (CPU) cycles to another needed function, or waits for a slower operation to finish.
The subsystem has a series of self-regulatory mechanisms that try to optimize the shared resources usage. Subsystem resources, such as CPU, cache bandwidth, cache size, host channel bandwidth, grid network bandwidth, and physical drives, are limited, and they must be shared by all tasks moving data throughout the subsystem.
The resources implicitly throttle by themselves when reaching their limits. The TS7700 introduces various explicit throttling methods to give higher priority tasks more of the
shared resources:
•Incoming traffic from the host (host throttling)
•RUN copy processing from other cluster (copy throttling)
•Copy processing of deferred copies to other cluster (deferred copy throttling)
TS7700T specific throttling behaviors
From a throttling perspective, the TS7700T is different than a TS7700 disk only model or a TS7740. Resident partition and Tape partitions have two independent throttling measurements, and are treated differently. Reaching a throttling limit on the Tape partitions, for example PMTHLVL, has no impact on the workload directed to the resident partition and vice versa.
The following rules apply:
•Throttling initiated by reaching the maximum Host Throughput applies to all partitions (Resident and Tape partitions).
•Throttling initiated by reaching any premigration limit (PMTHLVL or maximum size of the premigration feature queue) impacts only tape partitions, but not the workload directed
to CP0.
•Copy Throttling and Deferred copy throttling have a common measurement regardless of whether the workload is created in CP0 or CP1.
 
Important: Resident partition (CP0) and Tape Partitions (CP1 - CP7) are monitored and handled separately in a TS7700T.
However, even if the PMTHLVL throttling does not apply to the CP0 of a TS7700T, there is still an indirect influence because of the shared cache modules.
Consider the following items when you configure or monitor a TS7700T resource usage:
•Workloads that create data (either host I/O, remote writes, or copy processes from other clusters) in the CP0 uses resources of the cache bandwidth (write to cache).
•After PMTHLVL is crossed for the CPx, the Host I/O creating data in the CPx is throttled, but there will be no throttling to the jobs creating data in CP0.
•In small configurations (for example, up to four drawers), this can lead to the situation where the jobs running to CP0 use the cache bandwidth resources, and resources for premigration might be limited. This is especially true for a TS7720T due to the used cache.
•If the unpremigrated amount of data still increases, the throttling of the workload into the CPx increases too. This might reach a number of delays to the jobs where the jobs creating data in CPx are seriously impacted.
TS7740/TS7700T tape drives usage considerations
The physical tape drives are managed by the TS7740/TS7700T internal management software, and cannot be accessed from any other attached host. These drives are used exclusively by the TS7740/TS7700T for the mounts that are required for copying virtual volumes to stacked volumes, recalling virtual volumes into the cache, and reclaiming stacked volume space.
The availability of TS7740/TS7700T physical tape drives for certain functions can significantly affect TS7740/TS7700T performance.
The two major maintenance or “housekeeping” tasks at work are the premigration of data from cache to tape, and deferred copies to and from other clusters. The TS7740 and the TS7700T delay these housekeeping tasks to preference host I/O while no thresholds are reached (PMTHLVL).
The TS7740/TS7700T manages the internal allocation of these drives as required for various functions, but it usually reserves at least one physical drive for recall and one drive for premigration.
TVC management algorithms also influence the allocation of physical tape drives, as described in the following examples:
•Cache freespace low: The TS7740/TS7700T increases the number of drives available to the premigration function, and reduces the number of drives available for recalls.
•Premigration threshold crossed: The TS7740/TS7700T reduces the number of drives available for recall down to a minimum of one drive to make drives available for the premigration function.
The number of drives available for recall or copy is also reduced during reclamation.
The number of drives for premigration can be restricted on a physical pool base. If the number of drives available for premigration is restricted, or the physical drives are already used by other processes, it can lead to limiting the number of virtual volumes in the cache to be premigrated. This might lead to premigration throttling (host I/O is throttled), and later it can lead to free space or copy queue throttling.
If no physical drive is available when a recall is requested, elongated virtual mount times for logical volumes that are being recalled can be the result.
Recall performance is highly dependent on both the placement of the recalled logical volumes on stacked volumes, and the order in which the logical volumes are recalled. To minimize the effects of volume pooling on sustained write performance, volumes are premigrated by using a different distribution algorithm.
This algorithm chains several volumes together on the same stacked volume for the same pool. This can change recall performance, sometimes making it better, sometimes making it worse. Other than variations in performance because of differences in distribution over the stacked volumes, recall performance must be constant.
Reclaim policies must be set in the Management Interface (MI) for each volume pool. Reclamation occupies drives and can affect performance. Using multiple physical pools can cause a higher usage of physical drives for premigration and reclaim.
In general, the more pools are used, the more drives are needed. If all drives are busy, and a recall is requested, the reclaim process is interrupted. That can take some seconds to minutes, because the actual moving logical volume needs to be finished, and then the cartridge needs to be exchanged.
The Inhibit Reclaim schedule is also set from the MI, and it can prevent reclamation from running during specified time frames during the week. If Secure Data Erase is used, fewer physical tape drives might be available even during times when you use inhibited reclamation. If used, limit it to a specific group of data. Inhibit Reclaim specifications only partially apply to Secure Data Erase.
 
Note: Secure Data Erase does not acknowledge your settings and can run erasure operations if there are physical volumes to be erased.
The use of Copy Export and Selective Dual Copy also increases the use of physical tape drives. Both are used to create two copies of a logical volume in a TS7740/TS7700T.
11.4 Monitoring TS7700 performance
The IBM TS7700 series is part of a line of tape virtualization products that has revolutionized the way that mainframes use their tape resources. As the capability of tape virtualization has grown, so has the need to efficiently manage the large number of logical volumes that the system supports. Internal to the TS7700, a large amount of information is captured and maintained about the state and operational aspects of the resources within the TS7700.
The TS7700 provides a MI based on open standards through which a storage management application can request specific information that the TS7700 maintains. The open standards are not currently supported for applications running under IBM z/OS, so an alternative method is needed to provide the information to mainframe applications.
You can use the following interfaces, tools, and methods to monitor the TS7700:
•IBM TS4500 and TS3500 tape library Specialist (TS7740/TS7700T only)
•TS7700 M)
•Bulk Volume Information Retrieval function (BVIR)
•IBM Tape Tools: VEHSTATS, VEHAUDIT, and VEHGRXCL
•Host Console Request Commands
The specialist and MI are web-based. With the BVIR function, various types of monitoring and performance-related information can be requested through a host logical volume in the TS7700. Finally, the VEHSTATS tools can be used to format the BVIR responses, which are in a binary format, to create usable statistical reports.
With the VEHSTATS data, there are now performance evaluation tools available on Techdocs that quickly create performance-related charts. Performance tools are provided to analyze 24 hours worth of 15-minute data, seven days worth of one-hour interval data, and 90 days worth of daily summary data. For spreadsheets, data collection requirements, and trending evaluation guides, see the following Techdocs website:
All interfaces, tools, and methods to monitor the TS7700 are explained in detail next. An overview of these interfaces, tools, and methods is shown in Figure 11-4.
Figure 11-4 Interfaces, tools, and methods to monitor the TS7700
11.4.1 Base information: Types of statistical records
All of the mentioned interfaces and tools process the two types of statistics that are provided by the TS7700:
Point-in-time statistics
These statistics are performance-related. The point-in-time information is intended to supply information about what the system is doing the instant that the request is made to the system. This information is not persistent on the system. The TS7700 updates these statistics every 15 seconds, but it does not retain them.
This information focuses on the individual components of the system and their current activity. These statistics report operations over the last full 15-second interval. You can retrieve the point-in-time statistics from the TS7700 at any time by using the BVIR facility. A subset of point-in-time statistics is also available on the TS7700 MI.
The response records are written in binary undefined (U) format of maximum 24,000 bytes.
 
Tips:
•If a cluster or node is not available at the time that the point-in-time statistics are recorded, except for the headers, all the data fields for that cluster or node are zeros.
•The request records are written in fixed-block architecture (FB) format. To read the response records, use the Undefined (U) format with a maximum blocksize of 24,000. The response records are variable in length.
Historical statistics
Historical (HIS) statistics encompass a wide selection of performance and capacity planning information. They are intended to help with capacity planning and tracking system use over an extended period. The information focuses more on the system as a whole, and the movement of data through the system. These statistics are collected by the TS7700 every 15 minutes, and are stored for 90 days in a TS7700 database.
The user can also retrieve these records by using BVIR. A subset of the historical statistics is also available on the TS7700 MI. More information is available in 11.5, “Cache capacity” on page 643.
The historical statistics for all clusters are returned in the response records. In a TS7700 grid configuration, this way means that the request volume can be written to any cluster to obtain the information for the entire configuration. The response records are written in a binary undefined (U) format of a maximum of 24,000 bytes.
 
Tips:
•If a cluster or node is not available at the time that the historical statistics are recorded, except for the headers, all the data fields for that cluster or node are zeros.
•The TS7700 retains 90 days worth of historical statistics. If you want to keep statistics for a longer period, be sure that you retain the logical volumes that are used to obtain the statistics.
•The request records are written in FB format. To read the response records, use the undefined (U) format with a maximum blocksize of 24,000. The response records are variable in length.
Both point-in-time statistics and historical statistics are recorded. The point-in-time records present data from the most recent interval, providing speedometer-like statistics. The historical statistics provide data that can be used to observe historical trends.
These statistical records are available to a host through the BVIR facility. For more information about how to format and analyze these records, see 11.15, “Alerts and exception and message handling” on page 692.
Each cluster in a grid has its own set of point-in-time and historical statistics for both the vNode and hnode.
For a complete description of the records, see IBM TS7700 Series Statistical Data Format White Paper and TS7700 VEHSTATS Decoder reference, which are available on the following Techdocs web pages:
11.4.2 Using the TS4500 Management GUI
The TS45000 Management GUI has a new design, and have now the same look and feel as the TS7700 series. To gain an overview of the system, hover with the mouse over the TS4500 icon. Depending on your position, the actual health status is provided, but configuration information is also shown. Figure 11-5 shows an example.
Figure 11-5 TS4500 Overview example
In the TS4500 Logical Library view, you can find the information regarding the number of cartridges, drives, and maximum cartridges. Use the FILTER option for selecting the columns.
Figure 11-6 shows this window, and the selected properties.
Figure 11-6 TS4500 logical library display with properties
11.4.3 Using the TS3500 Tape Library Specialist for monitoring
The Tape Library Specialist (TS3500 Tape Library Specialist), only available with the TS7740/TS7700T, allows users to manage and monitor items that are related to the TS3500 tape library. Initially, the web user interface to the TS3500 tape library only supported a single user at any time. Now, each Ethernet-capable frame on the TS3500 tape library allows five simultaneous users of the web user interface so that multiple users can access the TS3500 Tape Library Specialist interface at the same time.
Figure 11-7 shows the TS3500 tape library System Summary window.
Figure 11-7 TS3500 Tape Library Specialist System Summary window
The TS3500 Tape Library Specialist session times out after a default setting of 10 minutes. This is different from the TS7700 MI. You can change the default values through the TS3500 Tape Library Specialist by selecting Manage Access → Operator Panel Security, which opens the window that is shown in Figure 11-8.
Figure 11-8 TS3500 Tape Library Specialist Operator Panel Security window
Some information that is provided by the TS3500 Tape Library Specialist is in a display-only format and there is no option to download data. Other windows provide a link for data that is available only when downloaded to a workstation. The data, in comma-separated value (CSV) format, can be downloaded directly to a computer and then used as input for snapshot analysis for the TS3500. This information refers to the TS3500 and its physical drive usage statistics from a TS3500 standpoint only.
For more information, including how to request and use this data, see IBM TS3500 Tape Library with System z Attachment A Practical Guide to Enterprise Tape Drives and TS3500 Tape Automation, SG24-6789.
The following information is available:
•Accessor Usage: Display only
 – Activity of each Accessor and gripper
 – Travel meters of Accessors
•Drive Status and Activity: Display only
•Drive Statistics: Download only
 – Last VOLSER on this drive
 – Write and Read megabytes (MB) per drive
 – Write and Read errors corrected per drive
 – Write and Read errors uncorrected per drive
•Mount History for cartridges: Download only
 – Last Tape Alert
 – Number of Mounts of a specific cartridge
 – Number of Write and Read retries of a specific cartridge in the lifecycle
 – Number of Write and Read permanent errors of a specific cartridge in the lifecycle
•Fibre Port statistics: Download only
The Fibre Port statistics include fiber errors, aborts, resets, and recoveries between the TS7700 and the physical tape drives in the TS3500 tape library.
 
Consideration: This statistic does not provide information from the host to the TS7700 or from the host to the controller.
•Library statistics, on an hourly basis: Download only
 – Total Mounts
 – Total Ejects
 – Total Inserts
 – Average and Maximum amount of time that a drive was mounted on a drive (residency)
 – Average and Maximum amount of time that was needed to run a single mount
 – Average and Maximum amount of time that was needed to run an eject
These statistics can be downloaded to a workstation for more analysis. These statistics are not included in the BVIR records processed by the TS7700.
11.4.4 Using the TS7700 Management Interface to monitor IBM storage
The TS7700 MI belongs to the family of tools that are used for reporting and monitoring IBM storage products. These tools do not provide reports, but they can be used for online queries about the status of the TS7700, its components, and the distributed libraries. They also provide information about the copies that have not completed yet and the amount of data to be copied.
The TS7700 MI is based on a web server that is installed in each TS7700. You can access this interface with any standard web browser.
The TS7700 MI is a Storage Management Initiative - Specification (SMI-S) - compliant interface that provides you with a single access point to remotely manage resources through a standard web browser. The MI is required for implementation and operational purposes. In a TS7700 configuration, two possible web interfaces are available:
•The TS4500 Management GUI or the TS3500 Tape Library Specialist
•The TS7700 MI
A link is available to the physical tape library management interface from the TS7700 MI, as shown at the lower left corner of Figure 11-9 on page 634. This link might not be available if not configured during TS7740/TS7700T installation, or for a TS7700D.
The Performance and Statistics windows of the TS7700 MI are described.
Performance and statistics
Information that relates to viewing performance information and statistics for the TS7700 for single and multi-cluster grid configurations is described. The graphical views display snapshots of the processing activities from the last 15 minutes if nothing else is stated when describing the windows. You can access the following selections by going to the Performance and Statistics section in the TS7700 MI. The examples are taken from different cluster configurations.
The navigation pane is available on the left side of the MI, as shown in the Grid Summary window shown in Figure 11-9.
Figure 11-9 TS7700 MI Performance
Historical Summary
This window (Figure 11-10 on page 635) shows the various performance statistics over a period of 24 hours. Data is retrieved from the Historical Statistic Records. It presents data in averages over 15-minute periods.
Multiple views can be selected, also dependent on the installed cluster type. Such as displaying for the maximum of a whole day:
•Throughput performance
•Throttling information
•Copy Queue
The value that is shown in the performance graph, Figure 11-10, can be changed by selecting the different metrics. To do so, click Select Metric and choose the requested values. The maximum number of values you can view in one picture is ten. You can also save the graph by pressing the download button (disc).
Figure 11-10 shows the Historical Summary “Throughput” window for a TS7760T
Figure 11-10 TS7700 MI Historical Summary of TS7760T
The precustomized “Throughput” enables you to see all cache bandwidth relevant information
•Primary cache device write
•Primary cache device read
•Remote read
•Remote write
•Link copy in
•Link copy out
•Write to tape
•Recall from tape
A pre-customized “Throttling” graph is shown in Figure 11-11. It shows which type of throttling happened during the selected interval, and the throttling impact in milliseconds.
Figure 11-11 TS7700 MI Historical Summary Throttling overview
A precustomized “Copy Queue” is shown in Figure 11-12.
Figure 11-12 TS7700 MI Historical Summary Copy Queue graph
This copy queue view shows how many MiB are sitting in the queue for a specific copy consistency policy.
All of these metrics can be changed by selecting different metrics. Remember, that you can select only 10 items for one graph, as shown in Figure 11-13.
Figure 11-13 Historical summary overview
Although this is a snapshot of one day or less, the performance evaluation tools on TECHDOCS provide you a 24-hour or 90-day overview of these numbers. Review the numbers to help you with these tasks:
•Identify your workload peaks and possible bottlenecks.
•See trends to identify increasing workload.
•Identify schedule times for reclaim.
For more information about using the tool, see 11.6.1, “Interpreting Cache throughput: Performance graph” on page 647. The Performance evaluation tool does not support new content regarding the TS7700T yet.
Virtual Mounts window
Use this window (Figure 11-14) to view virtual mount statistics for the TS7700. The virtual mount statistics for each cluster are displayed in two bar graphs and tables: One for the number of mounts and one for average mount time. This example is from a TS7700 Cluster that is part of a six-cluster grid configuration.
Figure 11-14 TS7700 MI Virtual Mounts window
The “Number of logical mounts during last 15 minutes” table has the following information:
Cluster The cluster name
Fast Ready Number of logical mounts that are completed by using the scratch (Fast Ready) method
Cache Hits Number of logical mounts that are completed from cache
Cache Misses Number of mount requests that were not fulfilled from cache
Total Total number of logical mounts
The “Average mount times (ms) during last 15 minutes” table has the following information:
Cluster The cluster name
Fast Ready Average mount time for scratch (Fast Ready) logical mounts
Cache Hits Average mount time for logical mounts that are completed from cache
Cache Misses Average mount time for requests that are not fulfilled from cache
This view gives you an overview only if you run out of virtual drives in a cluster. Depending on your environment, it does not show you, if in a specific LPAR or sysplex, there might be a shortage of virtual drives. Especially if you define virtual drives in a static way to an LPAR (without an allocation manager), a certain LPAR might not have enough drives. To ensure that a specific LPAR has enough virtual drives, analyze your environment with Tapetools MOUNTMON.
Physical Mounts window
Use this window (Figure 11-15) to view physical mount statistics for the TS7740. The physical mount statistics for each cluster are displayed in two bar graphs and tables: One for the number of mounts by category and one for average mount time per cluster. The example in Figure 11-15 is from a TS7740 cluster that is part of a multi-cluster grid configuration (four-cluster grid).
Figure 11-15 TS7740 MI Physical Mounts window
The table cells show the following items:
Cluster The cluster name
Pre-migrate Number of premigrate mounts
Reclaim Number of reclaim mounts
Recall Number of recall mounts
Secure Data Erase Number of Secure Data Erase mounts
Total Total physical mounts
Mount Time Average mount time for physical mounts
Review the used numbers of physical drives to help you with the following tasks:
•Identify upcoming bottlenecks.
•Determine whether it is appropriate to add or reduce physical pools. Using a larger number of pools requires more physical drives to handle the premigration, recall, and reclaim activity.
•Determine possible timelines for Copy Export operations.
Host Throughput window
You can use this window (Figure 11-16 on page 639) to view host throughput statistics for the TS7700. The information is provided in 15-second intervals, unlike the 15-minute intervals of other performance data.
Use this window to view statistics for each cluster, vNode, host adapter, and host adapter port in the grid. At the top of the window is a collapsible tree where you view statistics for a specific level of the grid and cluster. Click the grid to view information for each cluster. Click the cluster link to view information for each vNode. Click the vNode link to view information for each host adapter. Click a host adapter link to view information for each port.
The example in Figure 11-16 is from a TS7700 cluster that is part of a multi-cluster grid configuration (four-cluster grid).
Figure 11-16 TS7700 MI Host Throughput window
The host throughput data is displayed in two bar graphs and one table. The bar graphs are for raw data that is coming from the host to the host bus adapter (HBA), and for compressed data that is going from the HBA to the virtual drive on the vNode.
The letter next to the table heading corresponds with the letter in the diagram above the table. Data is available for a cluster, vNode, host adapter, and host adapter port. The table cells include the following items:
Cluster The cluster or cluster component for which data is being displayed (vNode, host adapter, or host adapter port)
Compressed Read (A) Amount of data that is read between the virtual drive and the HBA
Raw Read (B) Amount of data that is read between the HBA and host
Read Compression Ratio Ratio of compressed read data to raw read data
Compressed Write (D) Amount of data that is written from the HBA to the virtual drive
Raw Write (C) Amount of data that is written from the host to the HBA
Write Compression Ratio Ratio of compressed written data to raw written data
Although this is a snapshot, the performance evaluation tools on Techdocs provide you with a 24-hour, 7-day, or 90-day overview about these numbers.
Review these numbers to help you to identify the following conditions:
•Identify the compression ratio in your environment for cache and stacked volume planning.
•Identify any bottlenecks on the host throughput (FC enablement).
Cache Throttling window
You can use this window (Figure 11-17) to view cache throttling statistics for the TS7700. This example is from a TS7700 cluster that is part of a multi-cluster grid configuration (four-cluster grid).
Figure 11-17 TS7700 MI Cache Throttling window
Cache throttling is a time interval that is applied to TS7700 internal functions to improve throughput performance to the host. The cache throttling statistics for each cluster that relate to copy and write are displayed both in a bar graph form and in a table. The table shows the following items:
Cluster The cluster name
Copy The amount of time that is inserted between internal copy operations
Write The amount of time that is inserted between host write operations
Cache Utilization window
You can use this window (Figure 11-18) to view cache utilization statistics for the TS7700D or the TS7740. If a TS7700T is installed this selection is called “Cache Partitions” and is explained in the next section.
This example is from a TS7740 cluster that is part of a multi-cluster grid configuration (four-cluster grid).
Figure 11-18 TS7740 MI Cache Utilization window
The cache utilization statistics can be selected for each cluster. Various aspects of cache performance are displayed for each cluster. Select them from the Select cache utilizations statistics menu. The data is displayed in both bar graph and table form, and can be displayed also by preference groups 0 and 1.
The following cache utilization statistics are available:
•Cache Preference Group possible values:
 – 0: Volumes in this group have a preference for removal from cache over other volumes.
 – 1: Volumes in this group have a preference to be retained in cache over other volumes.
•Number of logical volumes currently in cache: The number of logical volumes present in the cache preference group.
•Total amount of data currently in cache: Total amount of data present in volumes that are assigned to the cache preference group.
•Median duration that volumes have remained in cache: Rolling average of the cache age of volumes that are migrated out of this cache preference group for the specified amount of time (last 4 hours, 48 hours, and 35 days).
•Number of logical volumes migrated: Rolling average of the number of volumes that are migrated to this cache preference group (4 hours, 48 hours, and 35 days). Bar graph is not used.
 
Clarification: Median Duration in Cache and Number of Logical Volumes Migrated statistics have a table column for each of the time periods that are mentioned in parentheses.
Review this data with the performance evaluation tool from Techdocs to identify the following conditions:
•Cache shortages, especially in your TS7700D
•Improvement capabilities for your cache usage through the adjustment of copy policies
Cache Partitions
You can use this window (Figure 11-19) to view Tape cache partitions and their utilization. Depending on your filter list, you might see the following output:
Figure 11-19 Tape Partition view in a TS7700T
For more information, see “Cache Partition” on page 387.
Grid Network Throughput window
Use this window (Figure 11-20 on page 643) to view network path utilization (Grid Network Throughput) statistics for the TS7700 Cluster.
 
Consideration: The Grid Network Throughput option is not available in a stand-alone cluster.
This window presents information about cross-cluster data transfer rates. This selection is present only in a multi-cluster grid configuration. If the TS7700 grid has only one cluster, there is no cross-cluster data transfer through the Ethernet adapters.
The example in Figure 11-20 is from a TS7700 Cluster that is part of a multi-cluster grid configuration (four-cluster grid).
Figure 11-20 TS7700 MI grid network throughput in a six-cluster grid
The table displays data for cross-cluster data transfer performance (MBps) during the last 15 minutes. The table cells show the following items:
Cluster The cluster name
Outbound Access Data transfer rate for host operations that move data from the specified cluster into one or more remote clusters
Inbound Access Data transfer rate for host operations that move data into the specified cluster from one or more remote clusters
Copy Outbound Data transfer rate for copy operations that pull data out of the specified cluster into one or more remote clusters
Copy Inbound Data transfer rate for copy operations that pull data into the specified cluster from one or more remote clusters
Review this data with the performance evaluation tools on Techdocs to identify the following conditions:
•Identify upcoming performance problems due to grid link usage.
•Identify the amount of transferred data to review your settings, such as DAA, SAA, override policies, and Copy Consistency Points.
11.5 Cache capacity
The amount of used cache is depending on the installed configuration and the TS7700 models. In general, the TS7740 and TS7700T are configured with smaller cache sizes, because the main portion of the data will be destaged to the backend environment.
For TS7700D or the TS7700T CP0 the aim is that all data is kept in cache. However it is possible to use the function of autoremoval, to allow data in a TS7700D or TS7700T CP0 to be removed, if otherwise a short on storage condition would occur.
In the TS7740 and TS7700T CPx the Storage Class with the storage preference determine if a logical volume is kept in cache (PG1) or is migrated as soon as possible (PG1).
In the TS7700D or TS7700T CP0, you can control if autoremoval is allowed or not. If autoremoval is enabled, it is possible to define, if a single logical volume is not eligible for autoremoval (Pinned), is eligible but should be kept if possible (prefer keep) or should be removed first if storage is needed (prefer remove).
Other than this, additional cache control exists:
•Tape partitions in the TS7700T to restrict the amount of data that can be used by a specific environment (LPAR/ Application/ HLQ), depending on the ACS routines
•How much space is protected in the CP0 that cannot be used for overspill
•Delay premigration in CPx, to keep data in cache for a specific time
•How copies are treated in the clusters (LI REQ SETTING: COPYFSC)
•How recalled data is treated in the cluster (LI REQ SETTING: RECLPG)
•How unwanted copies are treated (LI REQ SETTING: EXISTDEL)
11.5.1 Interpreting Cache Usage: MI
In the MI, you have several possibilities to view the cache usage. In the Cluster summary, you can determine how much space is consumed. In a TS7700T, you can use the Tape Partition window to get more detailed information. In the TS7740 and TS7700D, you find similar information in the Performance Section “Cache Utilization.”
Remember that this information is only a snapshot view.
11.5.2 Interpreting Cache Usage: VEHSTATS
In the H30TVCx reports, you find information about the cache usage. In a TS7700D and TS7740 the whole cache is reported in the H30TVC.
In a TS7700T, the CP0 usage is reported in the H30TVC1, and CP1 is reported in H30TVC2 and so on.
There is no explicit usage of the cache capacity for each partition reported. The total TVC usage is reported in TOTAL TVC_GB USED. Also, you find information regarding the following data:
•How many logical volumes are kept in the different preference classes, depending on the models
•How long logical volumes are kept in the different storage preferences (4 hours, 48 hours and 35 days for TS7740 or TS7700T CPx)
•How many logical volumes have been removed with autoremoval
11.5.3 Interpreting Cache Usage: LI REQ,distlib,CACHE
The LI REQ command with the subcommand CACHE shows you the actual usage of the cache of the cluster that is defined in the command. Figure 11-21 on page 645 shows an example output.
In the example that is shown in Figure 11-21, multiple tape partitions exists. In the CP1, 1287 GB are waiting for premigration. This TS7700 has only 1 FC 5274 installed, so throttling was applied (PMT and CPYT).
TAPE VOLUME CACHE STATE V4.0
TS7700 VIRTUALIZATION ENGINE MODEL: TS7720 TAPE ATTACH
TOTAL INSTALLED/ENABLED GBS: 23859 / 23859
TOTAL ADJUSTED CACHE USED GBS: 2196
CACHE ENCRYPTION STATUS: CAPABLE
OVERCOMMITTED CACHE PARTITIONS: NONE
PRIMARY CACHE RESIDENT ONLY INFORMATION
PRIVATE CACHE USED GBS: 3848
SCRATCH CACHE USED GBS: 0
CP ALLOC USED PIN PKP PRM COPY CPYT
0 8859 2012 0 2012 0 0 0
PRIMARY TAPE MANAGED PARTITIONS
CP ALLOC USED PG0 PG1 PMIGR D_PMIGR COPY PMT CPYT
1 8000 1756 1287 469 1287 0 0 41 41
2 4000 79 0 79 0 0 0 41 41
3 2000 0 0 0 0 0 0 41 41
4 0 0 0 0 0 0 0 41 41
5 1000 0 0 0 0 0 0 41 41
6 0 0 0 0 0 0 0 41 41
7 0 0 0 0 0 0 0 41 41
Figure 11-21 Example of LI REQ,distlib,CACHE
11.5.4 Tuning cache usage - Making your cache deeper
A deeper cache improves the likelihood of a volume being in cache for a recall. A cache-hit for a recall improves performance when compared to a cache-miss that requires a recall from physical tape. The TS7700 statistics provide a cache hit ratio for read mounts that can be monitored to ensure that the cache-hit rate is not too low. Generally, you want to keep the cache-hit ratio above 80%. Your cache can be made deeper in several ways:
•Add more cache.
•For TS7740 or tape attached partitions in the TS7700T, use the Storage Class (SC) construct to use Preference Level 0 (PG0). PG0 volumes are removed from cache soon after they are premigrated to physical tape. PG0 volumes are actively removed from cache and do not wait for the cache to fill before being removed.
This approach leaves more room for the PG1 volumes, which remain in cache when possible, to be available for recalls. Many clients have effectively made their cache deeper by examining their jobs and identifying which of them are most likely not to be recalled. Use SC to assign these jobs to PG0.
•For TS7700D or a TS7700T CP0, set the SC constructs to use prefer remove for volumes that you do not expect to be mounted. Use pinned for those that you know you will be mounting and need fast access times, and prefer keep for the others.
•With the SC construct PG1, the volume on the selected TVC for I/O operations is preferred to be in the cache of that cluster. The copy that is made on the other clusters is preferred to be removed from cache. If the TS7700 is used for the copy, ensure that this default setting is not overridden by the Host Console command. The behavior can be set by using SETTING, CACHE, COPYFSC:
 – When disabled, logical volumes that are copied into cache from a Peer TS7700 are managed as PG0 (prefer to be removed from cache).
 – When the ENABLE keyword is specified, the logical volumes that are copied into the cache from a peer TS7700 are managed by using the actions that are defined for the SC construct associated with the volume as defined at the TS7700.
This setting works on a distributed library level. It needs to be specified on each cluster. For a deep cache, DISABLE is the preferred keyword.
•By default, logical volumes that are recalled into cache are managed as though they were newly created or modified. You can modify cache behavior by using the SETTING Host Console command: SETTING, CACHE, RECLPG0:
 – When enabled, logical volumes that are recalled into cache are managed as PG0 (prefer to be removed from cache). This overrides the actions that are defined for the SC that is associated with the recalled volume.
 – When the DISABLE keyword is specified, logical volumes that are recalled into cache are managed by using the actions that are defined for the SC construct associated with the volume as defined at the TS7700.
This setting works on a distributed library level. It needs to be specified on each cluster. The preferred keyword depends on your requirements. ENABLE is the best setting if it is likely that the recalled logical volumes are used only once. With the setting DISABLE, the logical volume stays in cache for further retrieval if the SC is defined as PG1 in the cluster that is used for the I/O TVC.
11.5.5 Tuning cache usage - Management of unwanted copies
In rare cases, a copy of a logical volume on a cluster exists that should not exist on this cluster according to the Management Class (MC) definition. This situation can occur after a read, without the “retain copy mode option” or after migration scenarios. This copy was flagged as “E” for exist.
In previous releases, these copies were deleted at mount/demount time, if the copy was inconsistent. If the copy was consistent, it was kept in the cluster.
With R3.2, this command was enhanced, and now enables you to determine not only how unwanted copies are treated, but also when this command is run.
With R3.1, a new HCR command was introduced, with the following options:
LI REQ,distributed library,SETTING,EXISTDEL,CRITERIA,[STALE|ALL NONE]
•STALE: The “E” copy is deleted if this copy is inconsistent. This is the default.
•ALL: The “E” copy is deleted from the cluster, if all other non-“E” copy mode sites are consistent.
•NONE: The “E” copy will never be deleted.
Before Release 3.2, these settings were acknowledged only if the logical volume is mounted or unmounted. With Release 3.2, a new command has been introduced to determine when these settings are acknowledged:
LI REQ,distributed library,SETTING,EXISTDEL,WHEN,[ACTCLOSE|AUTO]
•ACTCLOSE: The “E” copy is deleted by mount /dismount processing. This is the same behavior before R3.2. This is the default.
•AUTO: The “E” copy is deleted by periodic checks. The check runs once per 24 hours, and deletes 100 “E” copies during that process.
The deletion of an “E” copy can be processed only if all clusters in the grid are available. That is necessary because the status of all copies needs to be determined.
These commands are only available if all clusters are on R3.2 or higher.
11.6 Cache throughput / Cache bandwidth
The TS7700 cache has a finite bandwidth, depending on your actual environment (number of installed cache drawers and models). Other influences are compression ratio and blocksizes used. For more information, see the performance white paper, IBM TS7700 R4 (TS7760) Performance, WP102652. Always use the most recently published performance white papers available on the Techdocs website at the following address:
This TVC bandwidth is shared between the host I/O (compressed), copy activity (from and to other clusters), premigration activity, recalls for read and remote writes from other clusters. The TS7700 balances these tasks by using various thresholds and controls to prefer host I/O.
This section gives you some information how to determine if your cache bandwidth is a limitation, for what actions the cache bandwidth is used, and tuning actions that you can consider.
11.6.1 Interpreting Cache throughput: Performance graph
As described in “Historical Summary” on page 634, the Historical Summary throughput chart give you an overview about the cache bandwidth. Due to the nature of the MI, you can see only the throughput of maximum 1 day, and only one cluster at a time. However, it is a good starting point if you are looking to an actual performance problem to identify, if the cache bandwidth is exhausted.
11.6.2 Interpreting cache throughput: VEHSTATS HOURFLOW
In the VEHSTATS reports, you find a report called “HOURFLOW.” This report gives you per cluster, on a 15-minute or 1-hour interval, the overall compressed throughput, and which task is using the cache bandwidth. Figure 11-22 shows such a report.
Figure 11-22 Hourflow Report of VEHSTATS
The MiB/s Total Xfer is an average value. Notice that some peaks maybe higher.
In this example, the maximum value is 497 MBps, but that is driven by the premigration task. If that would cause an performance issue, you should review the PMPRIOR and PMTHLVL setting for tuning.
11.6.3 Tuning Cache bandwidth: Premigration
To tune the usage of cache bandwidth, you have several possibilities.
•When premigration will be run (PMPRIOR and PMTHLVL)
•Amount of drives for premigration
•Use delayed premigration to either never premigrate data or delay the premigration to a more suitable time slot
Fast Host Write premigration algorithm
The Fast Host Write algorithm limits the number of premigration tasks to two, one, or zero. This limit occurs when the compressed host write rate is greater than 30 MiB/s, the CPU is more than 99% bus, and the total I/O rate (read and write) against the cache is greater than 200 MBps (not MiB/s).
The circle on the graph (Figure 11-23) illustrates this algorithm in effect. During the 16:15 to 16:45 intervals, the amount of premigrate activity is limited. During the next six intervals, the premigration activity is zero. After this period of intense host activity and CPU usage, the premigrate tasks are allowed to start again.
Figure 11-23 Fast Host Write premigration algorithm
11.6.4 Premigration and premigration throttling values
These two parameters are used to define actions, triggered by the amount of non-premigrated data in the cache. Both values are applicable only to the TS7700T and the TS7740.
Premigration Priority threshold (PMPRIOR)
When this threshold is crossed, premigration processes start and increase, and the host throughput tends to decrease from the peak I/O rate. The default value is 1600 GB. The TS7740 and TS7700T uses various criteria to manage the number of premigration tasks. The TS7700 looks at these criteria every 5 seconds to determine whether more premigration tasks must be added. Adding a premigration task is based on the following factors,
among others:
•Host-compressed write rate
•CPU activity
•The amount of data that needs to be premigrated per pool
•The amount of data that needs to be premigrated in total
Premigration Throttling threshold (PMTHLVL)
When this threshold is crossed, the host write throttle and copy throttle are both started. The purpose is to slow incoming data to allow the amount of non-premigrated data to be reduced and not rise above this threshold. The default value is 2000 GB.
As stated before, the workload creating data (Host I/O, Copy, or remote write) in the TS7700T CP0 is not subject to throttling for these values.
Premigration Throttling threshold (PMTHLVL)
When this threshold is crossed, the host write throttle and copy throttle are both started. The purpose is to slow incoming data to allow the amount of non-premigrated data to be reduced and not rise above this threshold. The default value is 2000 GB.
As stated before, the workload creating data (Host I/O, Copy, or remote write) in the TS7700T CP0 is not subject to throttling for these values.
Determining the values for your environment: PMPRIOR and PMTHLVL
When you define the values for PMPRIOR and PMTHLVL, they have several dependencies and consequences. Especially after hardware replacements, you need to review these parameters to ensure that the parameters are adjusted.
There is no guideline about the values of PMPRIOR and PMTHLVL. The following aspects need to be considered:
•Installed number of FC for Cache enablement for a TS7740
•Installed number of FC 5274 for premigration queue size for a TS7700T
•Workload profile
•Requirement regarding how long data should stay in cache unpremigrated
You need to determine the balance. If throttling occurs, it can be monitored with the MI or the VEHSTATS reports. You should review the values periodically.
To adjust the parameters, use the Host Console Request command. When you try to define a not allowed value, the TS7700 automatically uses an appropriate value. Details about these settings are described in the IBM Virtualization Engine TS7700 Series z/OS Host Command Line Request User’s Guide, which is available on the Techdocs website:
Use the following keywords:
•SETTING, CACHE, PMPRIOR
•SETTING, CACHE, PMTHLVL
PMPRIOR setting
If the value of PMPRIOR is crossed, the TS7740/TS7700T starts the premigration. This might decrease the resources available for other tasks in the TS7740/TS7700T and shorten the peak throughput period of a workload window.
Raising this threshold increases the timeline where the TS7740/TS7700T can run in Peak mode. However, the exposure is more for non-premigrated data in cache.
Having a low PMPRIOR causes data to be premigrated faster and avoids hitting the premigration throttling threshold.
The default is 1600 GB, and the maximum is the value of PMTHLVL minus 500 GB.
PMTHLVL setting
After the PMTHLVL is crossed, the Host I/O, remote writes, and copies into a TS7740 and TS7700T are throttled. In a TS7700T, only workload to the Tape partitions is subject to throttling. If the data continues to increase after you hit the PMTHLVL, the amount of delay for throttling will continue to increase.
Raising the threshold avoids the application of the throttles, and keeps host and copy throughput higher. However, the exposure is more for non-premigrated data in cache.
The default value is 2000 GB. The maximum is:
•TS7740: The number of installed TB cache enablement (FC 5276) times 1 TB minus
500 GB
•TS7700T: The number of installed premigration queue size (FC 5274) times 1 TB
How to determine the Premigration Queue Size Feature Codes or Tape Cache Enablement
The size of the installed feature codes can be determined in the following ways:
•The MI, in the window “Feature Code License”
•The VEHSTATS Report H30TVCx:
 – TVC_SIZE: For a TS7740, the enabled tape cache (not the installed size) is displayed. For a TS7700D or TS7700T the installed cache size is displayed.
 – PARTITION SIZE: For a TS7740 the enabled tape cache size is displayed again, for the TS7700T the partition size is reported.
 – PRE-MIG THROT VALUE: Shows the PMTHLVL value. In a TS7700T, the recommendation is to set the PMTHLVL equal to the amount of FC 5274. If that recommendation was used, this is the amount of FC installed.
11.7 TS7700 throughput: Host I/O increments
In a TS7700, the read/write throughput from the Host is generally limited by the number of Host I/O increments that are installed. This can be from 1 increment (100 MBps uncompressed data) up to 25 increments (unlimited) in a TS7700 with R3.2 and 8 GB Ficon installed. To understand how many MiB/s Host I/O a TS7700 can absorb as a maximum, the following aspects need to be considered:
•The configuration:
 – The amount and type of drawers installed
 – The FICON attachment
•The compression ratio
•The replication mode
•The blocksize of the I/O
•The read/write ratio
Especially in a TS7720T, the cache bandwidth can be a limit for the Host I/O as well.
To determine information about the Host I/O, you have multiple choices.
11.7.1 Host I/O in the performance graphs
In the metrics, you can select the Host I/O information, as shown in Figure 11-24.
Figure 11-24 Select the Host I/O metrics
Then, the Host I/O is displayed in the performance graph, as shown in Figure 11-25.
Figure 11-25 Performance Graph with Host I/O
11.7.2 Host I/O in the VEHSTATS
In the VEHSTATS report H20VIRT you find the statistic regarding the host I/O. Reported are the actual installed Host I/O increments, and the so called attempted throughput. As long the host I/O did not reach the limit, “LESS” is reported.
The most important fields on this page give you information regarding your Host I/O (throughput) behavior. The MAX THRPUT is the enabled Host Throughput limit with FC 5268.
The ATTMPT THRPUT is the amount of MBps the host wanted to deliver to the TS7700. In combination with the next three values, you can determine whether the installed FC 5268 is sufficient, or if an upgrade would provide a better performance. The DELAY MAX, DELAY AVG, and PCT of 15 Sec Intervals fields tell you how much delay is applied to each second during the 15-second sample interval. VEHSTATS reports thousandths of a second with a decimal point, which is the same as milliseconds if no decimal point is present. Statistics records are created every 15 minutes (900 seconds), so there are 60 of the 15-second intervals that are used to report the 15-minute interval values. Most especially, the PCT of 15 sec INTVLS tells you how often a delay occurred.
Our example in Figure 11-26 shows a TS7700 that benefits from more host throughput increments to provide a better job workflow.
This calculation is only an indicator. If you want to enhance the number of host I/O increments, talk to your IBM representative for sizing help.
Figure 11-26 Throughput Host I/O in VEHSTATS
11.7.3 Host Throughput Feature Codes
If the TS7700 is equipped or has been upgraded with 8-Gb FICON cards, take into account the fact that more throughput increments might need to be considered to unleash the full data transfer capabilities.
The previous 4-Gb FICON system had a maximum of 10 throughput increments, any data rate above 1 gigabyte per second (GBps) were given for free. With the new 8-Gb cards (or after an upgrade occurs), the new throughput increments limit is 25 GBps.
If a cluster was able to achieve speeds faster than 1 GBps with 4-Gb FICON cards and 10 throughput increments in the past, that will no longer be true because the TS7700 limits them to exactly 1 GBps, supposing that 8-Gb FICON cards were installed and the same 10 throughput increments were licensed. Thus, consider purchasing enough throughput increments (up to 25) to allow TS7700 cluster to run at unthrottled speeds.
Figure 11-27 shows the feature code license entry picture.
Figure 11-27 Feature Code license entry picture
Figure 11-28 shows the installed increments (Feature Code (FC) 5268). In this example, four increments are installed. The throughput is limited to 400 megabytes per second (MBps) because of number of the installed 100 MBps increments.
Figure 11-28 Feature Code licenses example
11.7.4 Tuning for Host I/O
Tuning for the Host I/O is limited. If the increments are exhausted, there is no possibility in the TS7700 to solve this issue. The only possibility is to change your workload profile in the attached hosts.
If the Host I/O is limited because the cache bandwidth is on the limit, see 11.6.3, “Tuning Cache bandwidth: Premigration” on page 648.
11.8 Grid link and replication performance
The following section gives an insight into what aspects and definitions influence the gridlink performance. It also provides information how to monitor and to tune the performance of the copy actions.
The grid link and replication performance depends on the following aspects:
•Installed grid link hardware
•Sufficient bandwidth and quality of the provided network
•Chosen replication mode
•Defined amount of concurrent copy tasks
•Number of remote write/read operations
Remember that cache bandwidth is always an influencing factor, but was already described in the last paragraphs. So this information is not included in this section.
11.8.1 Installed grid link hardware: Mixing of different Grid link adapters
It is not supported to have different grid link adapter types in one single cluster. However, in a grid, there can be a situation in which some clusters are connected to the grid link with 10 GB adapters, and other clusters are connected with 1 GB adapters. That is especially true for migration or upgrade scenarios.
In the TS7700 grid, there is a 1:1 relationship between the primary and primary adapters, and the secondary and secondary adapters. Due to that reason, in a mixed environment of 2*10 GB and 4*1 GB adapters, the clusters with the 4*1 GB links cannot use the full speed of the installed grid link adapters.
Remember, that 4*10 GB can be installed only in a VEC. A VEB/C07 with R4.0 cannot be upgraded to use 4*10 GB.
11.8.2 Bandwidth and quality of the provided network
The network between the TS7700s must have sufficient bandwidth to account for the total replication traffic. If you are sharing network switches among multiple TS7700 paths or with other network traffic, the total bandwidth on that network needs to be sufficient to account for all of the network traffic.
The TS7700 uses the TCP/IP protocol for moving data between each cluster. In addition to the bandwidth, other key factors affect the throughput that the TS7700 can achieve. The following factors directly affect performance:
•Latency between the TS7700s
•Network efficiency (packet loss, packet sequencing, and bit error rates)
•Network switch capabilities
•Flow control to pace the data from the TS7700s
•Inter-switch link (ISL) capabilities, such as flow control, buffering, and performance
The TS7700s attempt to drive the network links at the full 1-Gb rate for the two or four 1-Gbps links, or at the highest possible load at the two 10-Gbps links, which might be much higher than the network infrastructure is able to handle. The TS7700 supports the IP flow control frames to have the network pace the rate at which the TS7700 attempts to drive the network. The best performance is achieved when the TS7700 is able to match the capabilities of the underlying network, resulting in fewer dropped packets.
 
Important: When the system attempts to give the network more data than it can handle, it discard packets that it cannot handle. This process causes TCP to stop, resynchronize, and resend amounts of data, resulting in a less efficient use of the network.
To maximize network throughput, you must ensure the following items regarding the underlying network:
•The underlying network must have sufficient bandwidth to account for all network traffic that is expected to be driven through the system. Eliminate network contention.
•The underlying network must be able to support flow control between the TS7700s and the switches, allowing the switch to pace the TS7700 to the wide-area LANs (WANs) capability.
•Flow control between the switches is also a potential factor to ensure that the switches are able to pace with each other’s rate.
•Be sure that the performance of the switch can handle the data rates that are expected from all of the network traffic.
Latency between the sites is the primary factor. However, packet loss, because of bit error rates or because the network is not capable of the maximum capacity of the links, causes TCP to resend data, which multiplies the effect of the latency.
11.8.3 Selected replication mode
As already mentioned, the number of concurrent running copy tasks, and the replication mode has an influence to the overall performance of a TS7700 grid. The following shared resources are consumed:
•CPU cycles of the TS7700
•Grid bandwidth
•Cache bandwidth
Synchronous mode copy
Synchronous mode copy can have a positive effect to the cache bandwidth. In opposite to all other replication modi, where the data needs to be read from cache to produce the copy, the SYNC is written directly to the remote sync cluster.
In addition, the synchronous mode copy does also not adhere to the same rules as the other copy modes, as shown in Table 11-1.
Table 11-1 Synchronous mode rules comparison to Run or Deferred modes
Subject
Synchronous mode copy
Run or Deferred mode
Data direction
Data is pushed from the primary cluster.
Data is pulled from the secondary cluster.
Throttling
Synchronous mode copies will not be throttled.
These copies can be throttled.
Number of concurrent copies can be controlled by a setting.
No
Yes
Copies can be halted by a disable gridlink command.
No
Yes
Synchronous mode showing in the MI
In the MI the synchronous mode mounts are only visible in the Virtual Tape Device view, of the Host I/O cluster. The chosen second synchronous target cluster does not show any mount. That is true, because for the synchronous mount an internal device is used, which is not shown in the Virtual Tape drive panel, as shown in Figure 11-29.
Figure 11-29 Virtual Tape Drive view for synchronous mode copy
Synchronous mode monitoring in the VEHSTATS
The information for synchronous mode are shown in H30TVCx, independent for each partition. As in the MI, all numbers that are related to the synchronous mode operation are only shown in the Host I/O cluster for these mounts. The “secondary” synchronous mode does not reflect synchronous mounts.
In the following picture, you see that in total 44 scratch mounts (FAST NUM MNTS) where made, all of them are scratch mounts. In addition, you see the same number in the SYNC NUM MNTS field, which means, that the same number of mounts to a remote cluster has been run.
The receiver of the synchronous copy reports nothing, as shown in Figure 11-30.
Figure 11-30 Synchronous mode copies shown in VEHSTATS
There is no further information (such as the number of concurrent sync copies, inconsistency at interval, and so on) for the synchronous mode available, because none of them is applicable.
Only if the synchronous mode copy could not be processed, and sync-deferred was activated, are reports written. However, then these copies are reported with “DEFERRED” and there is no possibility for a further drill down.
Run copies monitoring in the VEHSTATS
The information about the RUN copies are contained in the H33GRID report in the receiving cluster, as shown in Figure 11-31 on page 659. To see the whole grid performance, you need to look at each cluster individually. To understand how much RUN copies (amount of lvols and MB) were processed in an interval, look to LVOLS TO_TVC_BY_RUN_COPY.
To understand, if RUN copies were queued for processing in that interval, look at AV_RUN_QUEAGE -- MINUTES--. Having numbers in here means that RUN could not be processed accordingly. Having RUN lvols waiting also means that the job cannot process further. If the report shows multiple indications for this behavior, take a closer look at the number of concurrent copy activities and the grid link usage.
You might want to consider increasing the number of concurrent RUN tasks. Also check if all receiving clusters where available, or if a cluster went to service or had an outage during that interval.
Figure 11-31 shows the H33GRID report.
Figure 11-31 H33Grid report to see copy behavior for RUN and Deferred copies
Deferred copies monitoring in the VEHSTATS
For deferred copies you have in report H33GRID only the AV_DEF QUEAGE -- MINUTES --- Usually the deferred copy queue is constantly increasing during batch processing. The reason is that deferred copies are throttled by the sending cluster, when specific workload values are reached.
To understand, if that throttling applies and is the reason for the increase, look in the H30TVCx report. The H30TVC1 report contains the information of CP0, and the H30TVC2-8 report contains the CP1- CP7 information.
The deferred copy throttling is for all partitions identically, so it is sufficient to look into one of the H30TVCx reports. As shown in Figure 11-32, the H30TVC report contains detail information about the occurred throttling.
Figure 11-32 Deferred copy throttling
The following information is displayed:
•NUM 15 MIN INTVL: Shows the number of intervals in the hour the deferred throttling occurred. A 4 means that in every interval of the 1-hour report, deferred throttling occurred. A 1 means, that only in one interval deferred copy throttling happened.
•NUM 30 SEC SMPLS: Shows in how many 30-second samples in the reported intervals the throttling occurred. That means, that in an hour report, you have a maximum of 120 samples (60 minutes * 2 samples of 30 seconds each). If 1 is reported, only in 30 seconds of the whole our a deferred throttling occurred.
•AVG SEC INTVL: Shows the actual amount of penalty in seconds given for each deferred copy action in this interval.
Looking to the Figure 11-32 on page 659, you find one interval where deferred copy throttling occurred in all 120 samples. This results in the maximum value of .125 s penalty for each copy operation. However, in the report shown, there was no interval where no deferred copy throttling occurred, but in some intervals were limited throttling measured.
Be aware, that depending on your network, a throttling higher than 20 ms normally results in little or no deferred copy action. For further information how to influence the deferred copy throttling, see 11.8.5, “Tuning possibilities for copies: Deferred Copy Throttling” on page 661.
11.8.4 Tuning possibilities for copies: COPYCOUNT Control
There can be several reasons for tuning the counts of the number of concurrent copy jobs over the grid links.
Values can be set for the number of concurrent RUN copy threads and the number of Deferred copy threads. The allowed values for the copy thread count are 5 - 128. The default value is 20 for clusters with two 1-Gbps Ethernet links, and 40 for clusters with four 1-Gbps or two 10-Gbps Ethernet links. Use the following parameters with the LIBRARY command:
•SETTING, COPYCNT, RUN
•SETTING, COPYCNT, DEF
Increase the copy count
Increasing the copy count can be beneficial, if the following conditions occur:
•The gridlinks are not saturated
•The number of very small logical volumes is high
•An upgrade from 2 to 4 grid links was made - and the number of copy tasks were not adjusted.
In this case, an increase of the copy count might reduce the RPO.
Be aware, that usually one gridlink with 1 Gbps can be saturated by 10 copies running in parallel. If the logical volumes are small, you might see gaps in the grid link usage, when only a few copies are running, because some volumes finished, and new lvols where selected for copy processing. In this situation, it might be beneficial to have more copies running concurrently.
Take into account that if too many copies are running concurrently, you will overflow the grid link. That can result in package loss and retries, and can lead overall to a lower performance of the grid link environment.
If you want to increase the number, do that in smaller steps (5 or 10) to get experience with the new setting. In addition, do not define values that are too high, especially if you use synchronous mode copy concurrently to the RUN and Deferred traffic.
Decrease the copy count
Decreasing the copy count can be beneficial in the following situations:
•If you have limited network bandwidth (for example, less than the 100 MiB/s).
•If the bandwidth is limited, running too many copies in parallel prolongs the single copy time. This can result in time outs. When you cross this threshold, the system switches from RUN to IMMED-Deferred. For a deferred copy, the copy is deleted from the actual copy tasks and will be scheduled back to the copy queue.
•If packet loss is reported and hardware issue or the grid link quality is not the reason.
•If too many copies are running in parallel, a single copy stream might run into a packet time out, which is reported as packet loss.
11.8.5 Tuning possibilities for copies: Deferred Copy Throttling
The deferred copy throttle (DCT) value is used to regulate outgoing deferred copies to other clusters to prefer host throughput. For some, host throughput is more important than the deferred copies, but for others, deferred copies are as important. Adjusting the DCT value and threshold can enable you to tune the performance of the deferred copies.
Deferred Copy throttle value
When the DCT threshold is reached, the TS7700 adds a delay to each block of deferred copy data that is sent across the grid links from a cluster. The larger the delay, the slower the overall copy rate becomes.
The performance of the grid links is also affected by the latency time of the connection. The latency has a significant influence on the maximum grid throughput. For example, with a one-way latency of 20 - 25 milliseconds (ms) on a 2 x 1 Gb grid link with 20 copy tasks on the receiving cluster, the maximum grid bandwidth is approximately 140 MBps. Increasing the number of copy tasks on the receiving cluster increases the grid bandwidth closer to
200 MBps.
The default DCT is 125 ms. The effect on host throughput as the DCT is lowered is not linear. Field experience shows that the knee of the curve is at approximately 30 ms. As the DCT value is lowered toward 30 ms, the host throughput is affected somewhat and deferred copy performance improves somewhat. At and below 30 ms, the host throughput is affected more significantly, as is the deferred copy performance.
If the DCT needs to be adjusted from the default value, try an initial DCT value of 30 - 40 ms. Favor the value toward 30 ms if the client is more concerned with deferred copy performance, or toward 40 ms if the client is concerned about sacrificing host throughput.
After you adjust the DCT, monitor the host throughput and Deferred Copy Queue to see whether the wanted balance of host throughput and deferred copy performance is achieved. Lowering the DCT improves deferred copy performance at the expense of host throughput.
A DCT of “0” eliminates the penalty completely, and deferred copies are treated equally as host I/O. Depending on your RPO requirements that is also a feasible setting.
The DCT value can be set by using the Host Console Request command. The setting of this throttle is discussed in detail in the IBM Virtualization Engine TS7700 Series z/OS Host Command Line Request User’s Guide, which is available from the Techdocs website by using the keywords SETTING, THROTTLE, and DCOPYT:
Deferred Copy throttle value threshold
This value is used to determine the average host I/O rate at which to keep deferred copy throttling on. The average host I/O rate is a rolling average of the I/O rate over a 20-minute period. When this average rate exceeds the DCT threshold, the deferred copies are delayed as specified by the DCOPYT value.
The DCTAVGTD – DCT 20-Minute Average Threshold looks at the 20-minute average of the compressed host read and write rate. The threshold defaults to 100 MBps. The Cache Write Rate – Compressed writes to disk cache includes host write, recall write, grid copy-in write, and cross-cluster write to this cluster. The threshold is fixed at 150 MBps. Cluster Utilization looks at both the CPU usage and the disk cache usage. The threshold is when either one is 85% busy or more.
DCT is applied when both of the following conditions are true:
•Cluster utilization is greater than 85% or the cache write rate is more than 150 MBps.
•The 20-minute average compressed host I/O rate is more than DCTAVGTD.
The preceding algorithm was added in R2.0. The reason to introduce the cache write rate
at R2.0 was due to the increased CPU power on the IBM POWER7 processor. The CPU usage is often below 85% during peak host I/O periods.
Before R2.0, the cache write rate was not considered. Use the following parameters with the LIBRARY command to modify the DCT value and the DCTAVGTD:
•SETTING, THROTTLE, DCOPYT
•SETTING, THROTTLE, DCTAVGDT
 
Note: The recommendation is to not change DCTAVGDT and instead use for the tuning the DCOPTY only. Changing DCTAVGDT might not reduce the throttling as expected.
Application of Deferred Copy Throttle
The next two charts illustrate the use of DCT. In Figure 11-33 on page 663, the amount of data that is being copied out is small because the DCT is being applied. DCT is applied because the compressed host I/O is above the DCT threshold, which is set to the default of 100 MBps.
Figure 11-34 on page 663 shows the compressed host I/O dropping below the 100 MBps threshold. As a result, the rate of deferred copies to other clusters is increased substantially.
Figure 11-33 shows the behavior when the DCT is used. The deferred copies are limited (light blue bars), and Host I/O (green bar) and Premigration (yellow bar) are preferred.
Figure 11-33 DCT being applied
In Figure 11-34, you see the effect when DCT is “turned off” because the host throughput drops under 100 MBps (green bar). The number of deferred copy writes in MBps increases (light blue bar).
Figure 11-34 DCT turned off
11.8.6 Grid link performance monitoring
The TS7700 generates a host message when it detects the grid performance is degraded. If the degraded condition persists, a call-home link is generated. The performance of the grid links is monitored periodically, and if one link is performing worse than the other link by an IBM Service Support Representative (IBM SSR)-alterable value, a warning message is generated and sent to the host. The purpose of this warning is to alert you that an abnormal grid performance difference exists. The value must be adjusted so that warning messages are not generated because of normal variations of the grid performance.
For example, a setting of 60% means that if one link is running at 100%, the remaining links are marked as degraded if they are running at less than 60% of the 100% link. The grid link performance is available with the Host Console Request function, and on the TS7700 MI. The monitoring of the grid link performance by using the Host Console Request function is described in detail in the IBM Virtualization Engine TS7700 Series z/OS Host Command Line Request User’s Guide, which is available on Techdocs. Use the STATUS and GRIDLINK keywords:
The grid link degraded threshold also includes two other values that can be set by the SSR:
•Number of degraded iterations: The number of consecutive 5-minute intervals that link degradation was detected before reporting an attention message. The default value is 9.
•Generate Call Home iterations: The number of consecutive 5-minute intervals that link degradation was detected before generating a Call Home. The default value is 12.
The default values are set to 60% for the threshold, nine iterations before an attention message is generated, and 12 iterations before a Call Home is generated. Use the default values unless you are receiving intermittent warnings and support indicates that the values need to be changed. If you receive intermittent warnings, let the SSR change the threshold and iteration to the suggested values from support.
For example, suppose that clusters in a two-cluster grid are 2000 miles apart with a round-trip latency of approximately 45 ms. The normal variation that is seen is 20 - 40%. In this example, the threshold value is at 25% and the iterations are set to 12 and 15.
11.9 Considerations for the backend TS7740 / TS7700T
In the backend, we advise you to monitor the two different resources, backend drives and backend cartridges. To determine if you have sufficient backend resources, how they are actually used, and how the use increases over time, you have several possibilities. The most difficult question is if you have sufficient backend drives.
11.9.1 Amount of Back-end drives
It is important to ensure that enough back-end drives are available. If not enough backend drives are available, you might see the following issues:
•Recalls are too slow, because no free backend drive was available
•Premigration cannot be processed sufficiently, and therefore throttling occurred due to reaching the limit of premigration queue sizes
•More backend cartridges are needed, because no drives where available to run the reclaim
If there are insufficient back-end drives, the performance of the TS7740/TS7700T diminishes.
In a TS7740, there was a direct dependency between Host I/O Increments throughput and the number of backend drives. This was understandable, because the TS7740 had a limited cache and all data that was written to the cache also needed to be premigrated to backend cartridges.
This strict dependency does not exist any more in a TS7700T. First of all, part of the Host I/O can be written in the CP0, and this data will never be premigrated. Second, a part of the data might be expired in cache, even if they were written to CPx using the Delay Premigration parameter.
Therefore, such a strict relationship does not exist any more.
As a guideline for the TS7740, use the ranges of back-end drives that are listed in Table 11-2 based on the host throughput that is configured for the TS7740. The lower number of drives in the ranges is for scenarios that have few recalls. The upper number is for scenarios that have numerous recalls. Remember, these are guidelines, not rules.
Table 11-2 Performance increments versus back-end drives
Throughput (MiB/s)
Back-end drives
Up to 400
4 - 6
400 - 800
6 - 8
800 - 1200
8 - 12
1200 - 1600
10 - 16
1600 or higher
16
So, if the Host increments cannot be used for guidance anymore, the question is how to determine the number of needed backend drives.
As described previously, there is no overall rule. Here are some general statements:
•The more physical backend pools you use, the more physical drives you need. Each physical pool is treated independently (premigration, reclaim).
•Depending on the used physical cartridge type and the reclaim value, the amount of still valid data can be high. Therefore, a reclaim has to copy a high amount of data from one physical tape to another tape. This uses two drives, and the more data that has to be transferred, the longer these drives will be occupied. Reclaim is usually a low priority task, but if not enough reclaims can be run, the number of necessary tapes increases.
•Tape drives are also needed for Copy Export and Secure data overwrite.
•Data expires in cache without premigration and data in CP0 does not need tape drives.
•Low hit ratio requires more recalls from the backend.
Installing the correct number of back-end drives is important, along with the drives being available for use. Available means that they are operational and might be idle or in use. The Host Console Request function can be used to set up warning messages for when the number of available drives drops. Setting the Available Physical Drive Low and High warning levels is discussed in detail in the IBM Virtualization Engine TS7700 Series z/OS Host Command Line Request User’s Guide, which is available on Techdocs. Use these keywords:
•SETTING, ALERT, PDRVLOW
•SETTING, ALERT, PDRVCRIT
Use the following website:
11.9.2 Monitor Backend drives in the MI
In the MI you can see the number and status of the backend drives, the actual mount situation, and in the performance section you find a snapshot of mount times.
11.9.3 Monitor Backend drives in the VEHSTATS
The report H32TDU12 gives you an overview about the usage of the backend drives. Example 11-1 shows this information:
•How many physical tape drives were installed, and how many were available
•How many drives (MIN/AVG/MAX) were mounted
•How much time (MIN/AVG/MAX in seconds) the mount took
•The number of physical mounts sorted by purpose:
 – STG: Recalls of logical volumes back into cache
 – MIG: Premigration of logical volumes from cache to physical tape
 – RCM: Reclamation
 – SDE: Secure Data Erase
Example 11-1 VEHSTATS for Physical Drives Activity
08JUL10TH -----------PHYSICAL_DRIVES_3592-E06-------------------
RECORD --MOUNTED-- -MOUNT_SECS- ----MOUNTS_FOR-----
TIME INST AVL MIN AVG MAX MIN AVG MAX STG MIG RCM SDE TOT
01:00:00 16 16 2 9 16 20 32 53 3 15 0 0 18
02:00:00 16 16 3 8 16 20 25 39 6 4 0 0 10
03:00:00 16 16 1 4 9 20 20 21 4 2 0 0 6
04:00:00 16 16 1 2 3 19 21 23 0 2 0 0   2
The following fields are the most important fields in this report:
•PHYSICAL_DRIVE_MOUNTED_AVG: If this value is equal or close to the maximum drives available during several hours, this might mean that more physical tape drives
are required.
•MOUNT_FOR (RCL MIG RCM SDE): This field presents the reason for each physical mount. If the percentage value in the Recall (RCL) column is high compared to the total number of mounts, this might indicate a need to evaluate the cache size or cache management policies. However, this is not a fixed rule and further analysis is required. For example, if HSM migration is into a TS7740, you might see high recall activity during the morning, which can be driven by temporary development or user activity. This is normal and not a problem in itself.
Be aware, that the number of tape drives might be misleading. The report does not recognize the “IDLE” state. Idle means that the tape drive is mounted, but not in use. Therefore, you might see a maximum - or even an average - usage that is equal to the installed drives. That might or might not be a performance issue. To identify if that really is a bottleneck, it is necessary to have a closer look at the overall situation.
To do so, first review which mounts are run. If a reclaim is still processed, there was no performance issue (except if a panic reclaim occurred).
If no reclaim was run, have a closer look at how long in average a physical cartridge could be mounted. To calculate this, take the total mounts and divide it by the number of installed drives and the amount of intervals sample. That shows how long a physical tape cartridge can be mounted on a physical tape drive. If this value is lower than 10 minutes, further investigations should be done. If this value is lower than 4 minutes, a performance issue is likely.
The H32GUPxx (General Pool Use) report is shown in Example 11-2. A single report always shows two pools. In this example, the report shows Pool 01 and Pool 02. You can see the following details per pool for each recorded time frame:
•The number of active logical volumes
•The amount of active data in GB
•The amount of data written in MB
•The amount of data read in MB
•The current reclamation threshold and target pool
Example 11-2 VEHSTATS report for General Pool Use
(C) IBM REPORT=H32GUP01(10210) HNODE LIBRARY HIST GUP/POOLING ACTIVITY RUN ON 18JUL2010 @ 16:57:51        PAGE 03
GRID#=CC001 DIST_LIB_ID= 0 VNODE_ID= 0 NODE_SERIAL=CL0ABCDE VE_CODE_LEVEL=008.006.000.0110 - UTC NOTCHG 18JUL10 POOL 01 3592-E05 3592JA READ UN- POOL 02 3592-E05 READ UN-
RECORD ACTIVE ACTIVE MB MB VOL_COUNT RECLAIM- ONLY AVAI ACTIVE ACTIVE MB MB VOL_COUNT RECLAIM- ONLY AVAI
TIME LVOLS GB WRITTN READ SCR PRIV PCT POOL VOLS VOLS LVOLS GB WRITTN READ SCR PRIV PCT POOL VOLS VOLS
UPD INT=> -ON_THE_HOUR- ON_THE_HR -ON_THE_HOUR- ON_THE_HR
4:15:00 65079 18052 5412 0 2 56 25 01 00 00 0 0 0 0 0 0 25 02 00 00
4:30:00 65079 18052 37888 0 2 56 25 01 00 00 0 0 0 0 0 0 25 02 00 00
  4:45:00 65079 18052 83895 0 2 56 25 01 00 00 0 0 0 0 0 0 25 02 00 00
5:00:00 65630 18206 94721 0 2 57 25 01 00 00 0 0 0 0 0 0 25 02 00 00
5:15:00 65630 18206 98630 0 2 57 25 01 00 00 0 0 0 0 0 0 25 02 00 00
5:30:00 65630 18206 124490 0 2 57 25 01 00 00 0 0 0 0 0 0 25 02 00 00
5:45:00 65630 18206 119979 0 2 57 25 01 00 00 0 0 0 0 0 0 25 02 00 00
6:00:00 67069 18610 108854 0 2 57 25 01 00 00 0 0 0 0 0 0 25 02 00 00
6:15:00 67069 18610 108854 0 2 57 25 01 00 00 0 0 0 0 0 0 25 02 00 00
6:30:00 67069 18610 97126 0 2 57 25 01 00 00 0 0 0 0 0 0 25 02 00 00
Check the ODERV12 statements from the BVIR jobs to select which types of cartridges are used in your environment. Only four different types of media can be reported at the same time.
11.9.4 Monitor Backend drives with a LI REQ command
The LI REQ,distlib,PDRIVE command gives you a snapshot about the physical tape drive environment. It shows the installed drives, the model, and the status. In addition, you can see what action is performed, for which pool the drive is working, and the actual mounted physical volume. If the status is not idle, also the actual logical volume in use is provided.
Figure 11-35 shows an output of this command.
LI REQ,DTS7720,PDRIVE
CBR1020I Processing LIBRARY command: REQ,DTS7720,PDRIVE.
CBR1280I Library DTS7720 request. 523
Keywords: PDRIVE
-----------------------------------------------------------------
PHYSICAL DRIVES V2 .1
SERIAL NUM TYPE MODE AVAIL ROLE POOL PVOL LVOL
0000078D1224 3592E07 Y IDLE 00
0000078D0BAA 3592E07 Y IDLE 00
0000078DBC65 3592E08 Y IDLE 00
0000078DBC95 3592E08 E08 Y MIGR 01 R00011 A51571
0000078DBC5C 3592E08 E08 Y MIGR 01 R00001 A66434
0000078DBC87 3592E08 E08 Y MIGR 01 R00002 A66462
Figure 11-35 LI REQ PDRIVE command example
11.9.5 Tune the usage of back-end drives
If you need to tune the back-end drive usage, you have multiple possibilities:
•Review the usage of delay premigration for data with short lifecycle
•Review the number of physical pools
•Review reclaim operations
•Review the amount of premigration drives
•Review Copy Export operation
Delay premigration usage
To eliminate the necessity to premigrate and reclaim data, you might consider using delay premigration for some of your data. To determine if this is a viable solution, you need to analyze the information from your tape management system. This shows if any data has such a short lifetime cycle, and how much cache space would be needed to do so.
Contact your IBM representative if you need further assistance to do so.
Number of physical pools
As mentioned, each physical pool is treated independently regarding the backend drive usage. That is true for premigration and reclaim. Reducing the number of active pools can be helpful if the backend drive environment shows bottlenecks.
Reclaim operations
Reclaim operations use two drives per reclaim task. Reclaim operations also use CPU MIPs, but does not use any cache bandwidth resources, because the data will be copied from physical tape to physical tape directly. If needed, the TS7740/TS7700T can allocate pairs of idle drives for reclaim operations, making sure to leave one drive available for recall.
Reclaim operations affect host performance, especially during peak workload periods. Tune your reclaim tasks by using both the reclaim threshold and Inhibit Reclaim schedule.
Reclaim threshold
The reclaim threshold directly affects how much data is moved during each reclaim operation. The default setting is 35% for each pool. Clients tend to raise this threshold too high because they want to store more data on their stacked volumes. The result is that reclaim operations must move larger amounts of data and use drive resources that are needed for recalls and premigration. After a reclaim task is started, it does not free its back-end drives until the volume being reclaimed is empty.
Table 11-3 shows the reclaim threshold and the amount of data that must be moved, depending on the stacked tape capacity and the reclaim percentage. When the threshold is reduced from 40% to 20%, only half of the data needs to be reclaimed. This change cuts the time and resources that are needed for reclaim in half. However, it raises the needed number of backend cartridges and slots in the library.
Table 11-3 Reclaim threshold by cartridge capacity
Cartridge
capacity
Reclaim threshold
10%
20%
30%
40%
300 GB
30 GB
60 GB
90 GB
120 GB
500 GB
50 GB
100 GB
150 GB
200 GB
640 GB
64 GB
128 GB
192 GB
256 GB
700 GB
70 GB
140 GB
210 GB
280 GB
1000 GB
100 GB
200 GB
300 GB
400 GB
4000 GB
400 GB
800 GB
1200 GB
1600 GB
10000 GB
1000 GB
2000 GB
30200 GB
4000 GB
Inhibit Reclaim schedule
Use the Inhibit Reclaim schedule to inhibit reclaims during your busy periods, leaving back-end drives available for recalls and premigrates tasks. Generally, start the inhibit 60 minutes before the heavy workload period. This setting allows any started reclaim tasks to complete before the heavy workload period.
Adjusting the maximum number of reclaim tasks
Reclaim operations use two back-end drives per task, and CPU cycles as well. For this reason, use the Inhibit Reclaim schedule to turn off reclaim operations during heavy production periods. When reclaim operations are not inhibited, you might want to limit the number of reclaim tasks. For example, moderate host I/O during the uninhibited period and reclaim might use too many back-end drives, CPU cycles, or both.
With the Host Library Request command, you can limit the number of reclaim tasks in the TS7740/TS7700T. The second keyword RECLAIM can be used along with the third keyword of RCLMMAX. This expansion applies only to the TS7740/TS7700T. Also, the Inhibit Reclaim schedule is still acknowledged.
The maximum number of reclaim tasks is limited by the TS7740/TS7700T, based on the number of available back-end drives, as listed in Table 11-4. These values have changed during the evolution of the product, and might be different in previous releases.
Table 11-4 Reclaim tasks
Number of available drives
Maximum number of reclaim tasks
3
1
4
1
5
1
6
2
7
2
8
3
9
3
10
4
11
4
12
5
13
5
14
6
15
6
16
7
Limiting the number of premigration drives (maximum drives)
Each storage pool enables you to define the maximum number of back-end drives to be used for premigration tasks. Several triggers can cause the TS7740/TS7700T to ramp up the number of premigration tasks. If a ramp-up of premigration tasks occurs, followed by the need for more than one recall, the recall must wait until a premigration task is complete for a back-end drive to be freed. A single premigration task can move up to 30 GB at one time. Having to wait for a back-end drive delays a logical mount that requires a recall.
If this ramping up is causing too many back-end drives to be used for premigration tasks, you can limit the number of premigration drives in the Pool Properties window. For a V06, the maximum number of premigration drives per pool must not exceed 6. Extra drives do not increase the copy rate to the drives. For a V07, TS7720T, or TS7760T, premigration can benefit from having 8 - 10 drives available for premigration, the default value is 10. There is no benefit to have more than 10 running premigration.
The limit setting is in the TS7740/TS7700T MI. For Copy Export pools, set the maximum number of premigration drives. If you are exporting a small amount of data each day (one or two cartridges’ worth of data), limit the premigration drives to two. If more data is being exported, set the maximum to four. This setting limits the number of partially filled export volumes.
To determine the number of drives to set, consider MB/GB written to a pool, compute MiB/s, compute maximum and average, and the number of premigration drives per pool. Base the number of drives by using 50 - 70 MBps per drive for models up to the TS1140, approximately 100 MBps for a TS1140, and approximately 140 MBps for a TS1150. These values are different from the native data rate because of the resources used by a TS7700 (for example, database updates and logical volume selection for premigration).
Avoiding Copy Export during heavy production periods
Because a Copy Export operation requires each physical volume to be exported to be mounted, the best approach is to run the operation during a slower workload time.
11.9.6 Number of back-end cartridges
The number of needed back-end cartridges is determined not only by the amount of data being stored on the cartridges. Some other parameters can influence the number of cartridges you need:
•Reclaim value
•Number of physical pools and pool properties
•Delete Expire setting
To monitor your actual situation, but also the trend of the cartridge usage, you have several possibilities.
11.9.7 Monitor the usage of back-end cartridges on the MI
To view the active pools, select them from the Physical volumes → Active data distribution menu. Figure 11-36 shows the active pools and correspondent data distribution (number of cartridges by occupancy percentage range).
Figure 11-36 Pool Active Data Distribution
Click a pool link. Information for the pool is displayed, as shown in Figure 11-37.
Figure 11-37 Information Display of selected pool
Review your Active Data Distribution. A low utilization percentage results in a higher number of stacked volumes. Also, ensure that you monitor the number of empty stacked volumes to avoid an “out of stacked volumes” condition. If you have defined multiple physical pools, you might need to check this on a per pool basis, depending on your Borrow/Return policies. In this example, Pool 3 has the borrow,return parameter enabled.
11.9.8 Monitor the usage of back-end cartridges with VEHSTATS
You can use the H32GUPxx report to view the cartridges on a per pool base. In addition, to see the trend of empty cartridges, you should use the H32CSP report. This report provides an overview of the empty cartridges in the common scratch pool on a cartridge type basis, as presented in Example 11-3.
Example 11-3 VEHSTATS report for Common Scratch Pool
18JUL10  ----------SCRATCH_STACKED_VOLUMES_AVAILABLE_BY_TYPE-----------
RECORD 3590J 3590K 3592JA 3592JJ NONE NONE 3592JB NONE
TIME MEDIA0 MEDIA1 MEDIA2 MEDIA3 MEDIA4 MEDIA5 MEDIA6 MEDIA7
4:15:00 0 ...... 0............. 42 ........... 0........... 0.............. 0............. 0 ........... 0
4:30:00 0 ..... 0 ............ 42 .......... 0........... 0.............. 0 ............ 0............ 0
4:45:00 0 ....0 ........42 ...... 0 .....0 ........0 .......0 ......0
5:00:00 0 ....0 ........41 ......0 .....0 ........0 .......0 ............0
Remember that it is not sufficient to check only the scratches in the common scratch pool. In addition, you need to check that all pools can borrow from the CSP and will return the empty cartridges to the CSP. If a pool is set to no borrow, you need to ensure that always enough empty cartridges are in inside this pool. This number is reflected in the H32GUPxx reports.
Keep in mind that in a heterogeneous environment, back-level cartridges (JA/JB) can only be used for read and not for write purposes.
11.9.9 Tuning of the usage of Back-end cartridges with VEHSTATS
As stated, you have several possibilities to influence the number of used backend cartridges.
Reclaim value
As explained in the backend cartridge section, you can change the reclaim value to gain more empty cartridges. The lower the percentage of the reclaim value, the more cartridges are needed. The higher this value is, the more valid data needs to be transferred and physical tape drives are required.
To find a good balance, review the active data distribution. Some times, it is sufficient to change to a slightly higher reclaim value.
Amount of physical pool and pool properties
For each physical pool usually two empty cartridges are kept inside the pool. Especially for smaller configurations with JD cartridges, that might increase the need for more cartridges.
In addition, the pool properties should be reviewed. No borrow/Keep has a negative influence.
Amount of physical pool and pool properties
The delete expire value in the scratch categories defines how long a logical volume and the data on it will still be treated as valid data. For more information about delete expire, see 3.3.8, “Logical Volume Delete Expire Processing versus previous implementations” on page 125.
Keep in mind, that a short delete expire value might reduce your cartridge usage, but a short value does not enable you to rescue any unintentional deleted data. We suggest not to use a value below 5 days. A best practise is to use 7 days.
11.10 Throttling the TS7700
As explained previously, Throttling is the mechanism adopted to control and balance several tasks that run at the same time within the TS7700, prioritizing certain tasks over others.
Generally speaking, we distinguish three different types of throttling:
•Host Write Throttling: Host I/O will be throttled
•Copy Throttling: RUN copies pulled from other clusters will be throttled
•Deferred Copy Throttling: Deferred copies pulled from other clusters will be throttled
The reasons for using DCT and how to tune the DCT have already been explained in 11.8.5, “Tuning possibilities for copies: Deferred Copy Throttling” on page 661.
11.10.1 Monitoring throttling with the MI
In the grid summary and the cluster summary, there is a throttling indicator on the head of the cluster. Hover over the indicator to see the throttling type and the throttling impact. This is only a snapshot view.
In addition, you can see the throttling on the performance graph, as explained in Figure 11-17 on page 640.
11.10.2 Monitoring throttling with VEHSTATS
In the H30TVCx report for each interval the throttling is reported for each of the different throttling types. To interpret these values, you need to review the throttling reasons in the following figures. Figure 11-38 shows the reasons for host write throttling.
Figure 11-38 Host Write Throttling reasons
Figure 11-39 shows the reasons for copy throttling.
Figure 11-39 Copy Throttling reasons
If a value of “x03” is shown in the H30TVC, that means reason X01 and X02 are applicable at the same time.
To understand if the throttling is a real performance issue, analyze in how many samples of the interval throttling happened, and the relative throttling impact value (%RLTV IMPAC VALUE). Even if throttling occurred, this might be only in a few samples during the interval, which means that the real impact to the write might or might not influence the overall production run time.
11.10.3 Tuning to avoid the throttling
For the different types and reasons for throttling several tuning possibilities can be considered.
Some of them are parameter changes (PMTHLVL adjustments, ICOPYT), whereas other issues can only be solved by providing higher bandwidth or more resources (cache, drives).
While adjusting a parameter might be beneficial for a specific situation, it might have another impact to other behaviors in the grid. Therefore, we recommend to discuss such tuning measurements with your IBM representative up front.
However, the next section describe a common tuning action.
Disabling host write throttle because of immediate copy
Host write throttle can be turned on because of immediate copies taking too long to copy to other clusters in the grid. Host write throttling is applied for various reasons, including when the oldest copy in the queue is 10 or more minutes old (R1.7). Since Release 2.0, the value was changed to 20 or more minutes. The TS7700 changes an immediate copy to immediate-deferred if the immediate copy has not started after 40 minutes in the immediate copy queue.
The reason for this approach is to avoid triggering the 45-minute missing interrupt handler (MIH) on the host. When a copy is changed to immediate-deferred, the RUN task is completed, and the immediate copy becomes a high priority deferred copy. See “Immediate-copy set to immediate-deferred state” on page 694 for more information.
You might decide to turn off host write throttling because of immediate copies taking too long (if having the immediate copies take longer is acceptable). However, avoid the 40-minute limit where the immediate copies are changed to immediate-deferred.
In grids where a large portion of the copies is immediate, better overall performance has been seen when the host write throttle because of immediate copies is turned off. You are trading off host I/O for length of time that is required to complete an immediate copy. The enabling and disabling of the host write throttle because of immediate copies is discussed in detail in the IBM Virtualization Engine TS7700 Series z/OS Host Command Line Request User’s Guide, which is available on Techdocs. Use the keywords SETTING, THROTTLE, ICOPYT:
11.11 Adjusting parameters in the TS7700
As described in the previous section, the LI REQ command gives you various adjustment possibilities. They can be subparameters of the following elements:
•SETTING
•SETTING2
•Independent parameters
Some of them have changed the cluster behavior, whereas others have an influence on the grid allocation behavior (for example, SAA, DAA, LOWRANK).
Although these settings can be modified by using z/OS or the MI, we suggest that you first check the parameter settings in case of any issue, and determine if they have been modified. Remember, that most of the settings are persistent, so after a Service or a Power Off of the cluster, they are still active.
 
Important: Library commands change the behavior of the whole cluster. If a cluster is attached to multiple LPARs from the same client, or to a multi-tenant environment, the change that is run from one LPAR influences all attached LPARs.
If you have a shared TS7700, consider restricting the usage of the Library command.
11.12 Monitoring after service or outage
Use this window (Figure 11-40) to view the pending updates for the IBM TS7700 Grid. The existence of pending updates indicates that updates occurred while a cluster was offline, in service prep mode, or in service mode. Before any existing pending updates can take effect, all clusters must be online.
Figure 11-40 Grid Pending Updates window
This window provides a summary of the number of outstanding updates for each cluster in an IBM TS7700 Grid. You can also use this window to monitor the progress of pending immediate-deferred copies, which, like pending updates, normally result from changes that are made while a cluster is Offline, in service prep mode, or in service mode.
 
Remember: Pending immediate-deferred copies need to be avoided. They might be a result of overload or grid network problems.
With Release 3.2, the download section also includes the tape with a hot token. Hot tokens are volumes that have been changed during an unavailability of the cluster and now need a reconciliation. The reconciliation is run during the cluster online setting.
11.13 Performance evaluation tool: Plotting cache throughput from VEHSTATS
To change the “knobs” settings to alter the behavior of the TS7700, you must be able to collect the statistics data from your clusters, use the available tools to format and plot the binary data, and understand the resulting graphs.
Support is available at the Techdocs website:
Table 11-41 shows the cache throughput plotted from VEHSTATS.
Figure 11-41 Cache throughput plotted from VEHSTATS
When evaluating performance, a graph that reveals a significant amount of information succinctly is the cache throughput for a cluster graph.
There are performance tools available on Techdocs that take 24 hours of 15-minute VEHSTATS data, seven days of 1-hour VEHSTATS data, or 90 days of daily summary data and create a set of charts for you.
The following material does not contain any information about the specifics of a Tape Attach model. However, the other information is still valuable, and has not changed, especially how to create the excel spreadsheet and the charts.
See the following Techdocs site for the performance tools, and the class replay for detailed information about how to use the performance tools:
•Tools:
•Class replay:
The 24-hour, 15-minute data spreadsheets include the cache throughput chart. The cache throughput chart has two major components: The uncompressed host I/O line and a stacked bar chart that shows the cache throughput.
The cache throughput chart includes the following components (all values are in MiB/s):
•Compressed host write: This is the MiB/s of the data that is written to cache. This bar is hunter green.
•Compressed host read: This is the MiB/s of the data read from cache. This bar is lime green.
•Data that is copied out from this cluster to other clusters: This is the rate at which copies of data to other clusters are made. This cluster is the source of the data and includes copies to all other clusters in the grid. The DCT value that is applied by this cluster applies to this data.
For a two-cluster grid, this is a single value. For a three-cluster grid, there are two values, one for copies to each of the other clusters. For a four-cluster grid, there are three values, one for copies to each of the other clusters. The following descriptions are for plotting Cluster 0’s cache throughput. Use the appropriate columns when plotting other clusters. These bars are cyan.
•Data that is copied to this cluster from other clusters: This is the rate at which other clusters are copying data into this cluster. This cluster is the target of the data and includes copies from all other clusters in the grid. The same rules for DCT apply for data copied out. These bars are light blue.
•Compressed data premigrated from cache to tape: This is the rate at which data is being read from cache and written to physical tape. This bar is yellow.
•Compressed data recalled from tape to cache: This is the rate at which data is being read from tape into cache for a mount that requires a recall. This bar is dark blue.
•Compressed remote reads from this cluster: This is the rate that other clusters use this TVC as I/O cache for read. This bar is orange.
•Compressed remote writes to this cluster: This is the rate of synchronous copies. This bar is burnt orange.
This tool contains spreadsheets, data collection requirements, and a 90-day trending evaluation guide to assist you in the evaluation of the TS7700 performance. Spreadsheets for a 90-day, 1-week, and 24-hour evaluation are provided.
One 90-day evaluation spreadsheet can be used for one-cluster, two-cluster, three-cluster, or four-cluster grids and the other evaluation spreadsheet can be used for five-cluster and six-cluster grids. There is an accompanying data collection guide for each. The first worksheet in each spreadsheet has instructions for populating the data into the spreadsheet. A guide to help with the interpretation of the 90-day trends is also included.
There are separate one-week spreadsheets for two-cluster, three-cluster, four-cluster, five-cluster, and six-cluster grids. The spreadsheets use the one-hour interval data to produce charts for the one-week period. There is also a data collection guide.
There are separate 24-hour spreadsheets for two-cluster, three-cluster, four-cluster, five-cluster, and six-cluster grids. The spreadsheets use the 15-minute interval data to produce charts for the 24-hour period. There is also a data collection guide.
These spreadsheets are intended for experienced TS7700 users. A detailed knowledge of the TS7700 is expected, and familiarity with using spreadsheets.
11.14 Bulk Volume Information Retrieval
With the potential to support hundreds of thousands of logical volumes in a TS7700 subsystem, providing a set of information for all of those volumes through normal channel control type commands is not practical. Luckily, the functions of a TS7700 subsystem that allow it to virtualize a tape volume also allow for a simple and effective method to transfer the information to a requesting application.
The TS7700 converts the format and storage conventions of a tape volume into a standard file that is managed by a file system within the subsystem. With BVIR, you are able to obtain information about all of the logical volumes that are managed by a TS7700.
The following data is available from a TS7700:
•Volume Status Information
•Cache Contents Information
•Physical Volume to Logical Volume Mapping Information
•Point-in-Time Statistics
•Historical Statistics
•Physical Media Pools
•Physical Volume Status
•Copy Audit
For more information, see the IBM TS7700 Series Bulk Volume Information Retrieval Function User’s Guide at the following URL:
11.14.1 Overview of the BVIR function
The TS7700 converts the format and storage conventions of a tape volume into a standard file that is managed by a file system within the subsystem. It uses an IBM-standard labeled tape volume to both initiate a request for information and return the results. By using a standard tape volume, no special interfaces or access methods are needed for an application to use this facility. In practice, no specific applications are required because standard IBM utilities, such as IEBGENER, provide the function that is needed to request and obtain the information.
The following steps obtain information by using this function:
1. A single data set with the information request is written to a logical volume. The logical volume can be any logical volume in the subsystem from which the information is to be obtained. Either a scratch or specific volume request can be used. The data set contains a minimum of two records and a maximum of three records that specify the type of data
that is being requested. The records are in human-readable form, containing lines of character data.
The data set can be cataloged or uncataloged (although cataloging the data set can make it easier for subsequent access to the data). On closing the volume, the TS7700 server recognizes it as a request volume and “primes” the subsystem for the next step.
 
Remember: Some information that is obtained through this function is specific to the cluster on which the logical volume is written, for example, cache contents or a logical-physical volume map. In a TS7700 grid configuration with multiple clusters, use an MC for the volume to obtain statistics for a specific cluster. Historical statistics for a multi-cluster grid can be obtained from any of the clusters.
2. The request volume is again mounted, this time as a specific mount. Seeing that the volume was primed for a data request, the TS7700 appends the requested information to the data set. The process of obtaining the information and creating the records to append can take up to several minutes, depending on the request and, from a host’s viewpoint, is part of the mount processing time.
After the TS7700 has completed appending to the data set, the host is notified that the mount is complete. The requested data can then be accessed like any other tape data set. In a job entry subsystem 2 (JES2) environment, the job control language (JCL) to complete the two steps can be combined into a single job. However, in a JES3 environment, they must be run in separate jobs because the volume will not be unmounted and remounted between job steps in a JES3 environment.
After the response data set has been written to the request logical volume, that logical volume functions identically to any other logical volume in the subsystem. Subsequent mount requests and read accesses to the logical volume do not affect its contents. Write accesses to the logical volume overwrite its contents. The logical volume can be returned to SCRATCH status and reused by any application.
 
Note: Due to the two-step approach, BVIR volumes cannot be written with LWORM specifications. You need to assign a Data Class without LWORM for BVIR volumes.
Figure 11-42 shows the process flow of BVIR.
Figure 11-42 BVIR process flow
The building of the response information requires a small amount of resources from the TS7700. Do not use the BVIR function to “poll” for a specific set of information and only issue one request at a time. Certain requests, for example, the volume map, might take several minutes to complete.
To prevent “locking” out another request during that time, the TS7700 is designed to handle two concurrent requests. If more than two concurrent requests are sent, they are processed as previous requests are completed.
Although the requested data is always in a human-readable format, depending on the request, the data that is returned from the TS7700 can be in human-readable or binary form. See the response sections for the specifics of the returned data.
The general format for the request/response data set is shown in Example 11-4.
Example 11-4 BVIR output format
123456789012345678901234567890123456789012345
VTS BULK VOLUME DATA REQUEST
VOLUME MAP
11/20/2008 12:27:00 VERSION 02
S/N: 0F16F LIB ID: DA01A
  
PHYSICAL LOGICAL P/B  ORDER   PART       SIZE
P00024   GK0000    P 000001 1 OF 1    23.45 M
P00024   GK0020    P 000002 1 OF 1    76.50 M
P00024   GK0010    P 000003 1 OF 1   134.24 M
 
Clarification: When records are listed in this chapter, there is an initial record showing “1234567890123...” This record does not exist, but it is provided to improve readability.
Record 0 is identical for all requests, and it is not part of the output; it is for support for records 1 - 5 only. Records 6 and higher contain the requested output, which differs depending on
the request:
•Records 1 and 2 contain the data request commands.
•Record 3 contains the date and time when the report was created, and the version
of BVIR.
•Record 4 contains both the hardware serial number and the distributed library ID of the TS7700.
•Record 5 contains all blanks.
Records 6 - N and higher contain the requested data. The information is described in general terms. Detailed information about these records is in the IBM TS7700 Series Bulk Volume Information Retrieval Function User’s Guide at the following URL:
11.14.2 Prerequisites
Any logical volume that is defined to a TS7700 can be used as the request/response volume. Logical volumes in a TS7700 are formatted as IBM standard-labeled volumes. Although a user can reformat a logical volume with an ANSI standard label or as an unlabeled tape volume, those formats are not supported for use as a request/response volume. There are no restrictions regarding the prior use of a volume that is used as a request/response volume, and no restrictions regarding its subsequent use for any other application.
Use normal scratch allocation methods for each request (that is, use the DISP=(NEW,CATLG) parameter). In this way, any of the available scratch logical volumes in the TS7700 can be used. Likewise, when the response volume’s data is no longer needed, the logical volume must be returned to SCRATCH status through the normal methods (typically by deletion of the data set on the volume and a return-to-scratch policy based on data set deletion).
11.14.3 Request data format
Several types of data can be requested. The type of data that is requested is indicated in the request data set. The request data set must be the only data set on the volume, and must be written with a record format of “F” (fixed block) and a logical record size of 80 bytes in uncompressed data format (TRTCH=NOCOMP). Request information is in EBCDIC character form, beginning in the first character position of the record and padded with blank characters on the right to complete the record.
 
Important:
•The request fields must be as shown. Not beginning with the first character position of the record, or by using extra blanks between words, results in a failed request.
•The file must be written in uncompressed format to have it correctly interpreted by the TS7700.
Although the request data format uses fixed records, not all response records are fixed. For the point-in-time and historical statistics responses, the data records are of variable length and the record format that is used to read them is the Undefined (U) format. See Appendix E, “Sample job control language” on page 853 for more information.
In a multi-site TS7700 grid configuration, the request volume must be created on the cluster for which the data is being requested. The MC assigned to the volume needs to specify the particular cluster that is to have the copy of the request volume.
The format for the request data set records is listed in the following sections.
Record 1
Record 1 must contain the command exactly as shown in Example 11-5.
Example 11-5 BVIR request record 1
1234567890123456789012345678
VTS BULK VOLUME DATA REQUEST
The format for the request’s data set records is shown in Table 11-5.
Table 11-5 BVIR request record 1
Record 1: Request identifier
Bytes
Name
Contents
1 - 28
Request identifier
VTS BULK VOLUME DATA REQUEST
29 - 80
Blanks
Blank padding
Record 2
With Record 2, you can specify which data you want to obtain. The following options are available:
•VOLUME STATUS zzzzzz
•CACHE CONTENTS
•VOLUME MAP
•POINT IN TIME STATISTICS
•HISTORICAL STATISTICS FOR xxx
•HISTORICAL STATISTICS FOR xxx-yyy
•PHYSICAL MEDIA POOLS
•PHYSICAL VOLUME STATUS VOLUME zzzzzz
•PHYSICAL VOLUME STATUS POOL xx
•COPY AUDIT COPYMODE INCLUDE/EXCLUDE libids
The format for the request’s data set records is shown in Table 11-6.
Table 11-6 BVIR request record 2
Record 2: Request identifier
Bytes
Name
Contents
1 - 80
Request
‘VOLUME STATUS zzzzzz’ or
‘CACHE CONTENTS’ or
‘VOLUME MAP’ or
‘POINT IN TIME STATISTICS’ or
‘HISTORICAL STATISTICS FOR xxx-yyy’ or
‘PHYSICAL MEDIA POOLS’ or
‘PHYSICAL VOLUME STATUS VOLUME zzzzzz’ or
‘PHYSICAL VOLUME STATUS POOL xx’ or
‘COPY AUDIT COPYMODE INCLUDE/EXCLUDE libids’
Left-aligned, padded with blanks on the right
For the Volume Status and Physical Volume Status Volume requests, ‘zzzzzz’’ specifies the volume serial number mask to be used. By using the mask, one to thousands of volume records can be retrieved for the request. The mask must be six characters in length, with the underscore character ( _ ) representing a positional wildcard mask.
For example, assuming that volumes in the range ABC000 - ABC999 have been defined to the cluster, a request of VOLUME STATUS ABC1_0 returns database records that exist for ABC100, ABC110, ABC120, ABC130, ABC140, ABC150, ABC160, ABC170, ABC180, and ABC190.
For the Historical Statistics request, xxx specifies the Julian day that is being requested. Optionally, -yyy can also be specified and indicates that historical statistics from xxx through yyy are being requested. Valid days are 001 - 366 (to account for leap year). For leap years, February 29 is Julian day 060 and December 31 is Julian day 366. For other years, Julian day 060 is March 1, and December 31 is Julian day 365. If historical statistics do not exist for the day or days that are requested, that will be indicated in the response record.
This can occur if a request is made for a day before the day the system was installed, day or days the system was powered off, or after the current day before a rolling year has been accumulated. If a request spans the end of the year, for example, a request that specified as HISTORICAL STATISTICS FOR 364-002, responses are provided for days 364, 365, 366, 001, and 002, regardless of whether the year was a leap year.
For Copy Audit, INCLUDE or EXCLUDE is specified to indicate which TS7700’s clusters in a grid configuration are to be included or excluded from the audit. COPYMODE is an option for taking a volume’s copy mode for a cluster into consideration. If COPYMODE is specified, a single space must separate it from INCLUDE or EXCLUDE.
The libid parameter specifies the library sequence numbers of the distributed libraries that are associated with each of the TS7700 clusters either to include or exclude in the audit. The parameters are separated by a comma. At least one libid parameter must be specified.
For the Physical Volume Status Pool request, xx specifies the pool for which the data is to be returned. If there are no physical volumes that are assigned to the specified pool, that is indicated in the response record. Data can be requested for pools 0 - 32.
For point-in-time and historical statistics requests, any additional characters that are provided in the request record past the request itself are retained in the response data, but otherwise ignored. In a TS7700 grid configuration, the request volume must be valid only on the specific cluster from which the data is to be obtained.
Use a specific MC that has a copy policy that is defined to indicate that only the wanted cluster is to have a copy of the data. By ensuring that there is a sole copy of the request volume, any virtual device address on any of the clusters in the same grid configuration can be used to request and access the data. You do not have to have host connectivity to the specific cluster. If an MC is used that indicates that more than one cluster is to have a valid copy of the request volume, unpredictable response data results can occur.
11.14.4 Response data format
When the request data set has been written to the volume and then closed and unmounted, when mounted again, the TS7700 validates the contents of the request volume and appends the requested data records to the data set.
Human-readable appended records can vary in length, depending on the reports that are requested and can be 80 - 640 bytes. Binary data appended records can be variable in length of up to 24,000 bytes. The data set is now a response data set. The appropriate block counts in the end of file (EOF) records are updated to reflect the total number of records written to the volume.
These records contain the specific response records based on the request. If the request cannot be understood or was invalid, that is indicated. The record length is fixed; the record length of each response data is listed in Table 11-7.
Table 11-7 Record length of response data
BVIR request
Record length
in bytes
VOLUME STATUS vvvvvv
64
CACHE CONTENTS
80
VOLUME MAP
80
POINT IN TIME STATISTICS
24000
HISTORICAL STATISTICS FOR xxx-yyy
24000
PHYSICAL MEDIA POOLS
80
PHYSICAL VOLUME STATUS VOLUME zzzzzz
440
PHYSICAL VOLUME STATUS POOL xx
440
COPY AUDIT COPYMODE INCLUDE/EXCLUDE libids
80
After appending the records and updating the EOF records, the host that requested the mount is signaled that the mount is complete and can read the contents of the volume. If the contents of the request volume are not valid, one or more error description records are appended to the data set or the data set is unmodified before signaling the host that the mount completed, depending on the problem encountered.
All human-readable response records begin in the first character position of the record and are padded with blank characters on the right to complete the record. All binary records are variable in length and are not padded.
 
Tips:
•In the response records, the dates and times that are presented are all based on the internal clock of the TS7700 handling the request. The internal clock of a TS7700 is not synchronized to the host, but it is synchronized with all other TS7700s.
•The host and the TS7740 can be synchronized to a Network Time Protocol (NTP) server, but they use a different NTP server with a different timing protocol. Slight time differences are still possible when NTP is used.
The response data set contains both request records that are described in 11.14.3, “Request data format” on page 681, and the response data set contains three explanatory records (Records 3 - 5) and starting with Record 6, the actual response to the data request.
The detailed description of the record formats of the response record is in the following white papers:
•IBM TS7700 Series Bulk Volume Information Retrieval Function User’s Guide
•IBM TS7700 Series Statistical Data Format White Paper
The response data set has this general format:
•Records 1- 2
Contains the contents of request records 1- 2.
•Record 3
This record contains the date and time that the response data set was created and a format version number for the results.
•Record 4
This record contains both the five-character hardware serial number of the TS7700, and the five-character distributed library sequence number of the cluster that generated the response.
•Record 5
This record contains all blank characters.
•Record 6 - N and Record 7
These records contain the specific response records based on the request. If the request cannot be understood or was invalid, that is indicated.
11.14.5 Interpreting the BVIR response data
This section explains how to interpret each BVIR Response Data Set for the specific request information, such as the following information:
•Volume Status Information
•Cache Contents Information
•Physical Volume to Logical Volume Mapping Information
•Point in Time Statistics
•Historical Statistics
•Physical Media pools
•Physical Volume Status
•Copy Audit
 
Clarification: When records are listed in this chapter, an initial record shows “1234567890123...”. This record does not exist, but is provided to improve readability.
Volume status information
A database is maintained on each TS7700 cluster that contains information that is related to the management of the logical volumes on the cluster and copy and resynchronization processes when the TS7700s are in a grid configuration. Several returned database fields can be useful in handling operational exceptions at one or more clusters in a grid configuration.
The volume status information that is returned represents the status of the volume on the cluster the requested volume was written to. In a TS7700 grid configuration, separate requests must be sent to each cluster to obtain the volume status information for the individual clusters. Using the volume serial number mask that is specified in the request, a response record is written for each matching logical volume that exists in the cluster.
A response record consists of the database fields that are defined as described in the white paper. Fields are presented in the order that is defined in the table and are comma-separated. The overall length of each record is 640 bytes with blank padding after the last field, as needed. The first few fields of the record that is returned for VOLSER ABC123 are shown in Example 11-6.
Example 11-6 BVIR volume status information
123456789012345678901234567890123456789012345678901234567890123
ABC123,0,2009-04-22-11.56.45.871263,0,0,32,0,N,2548,N,8719,N...
Important information is derived from the records:
•Data Inconsistent
This field indicates whether the cluster has a valid version of the data. If it indicates that the data on the logical volume is not valid, the same volume on another TS7700 in the grid has been modified and it has not yet been copied.
Suppose that you use the deferred Copy Consistency Point (which is typically when there is significant distance between the TS7700s in the grid configuration). In this situation, some number of volumes will not be consistent between the TS7700s at any point in time.
If a situation occurs that renders the site inoperable where the source data is, by sending the Volume Status request to an operable TS7700, this field can be used to identify the volumes that were not copied before the situation so that appropriate recovery steps can be run for them.
•MES Volume
This field indicates that the logical volume was created in the TS7700 Cluster or even created within a VTS before being merged into a grid configuration. Volumes that existed in a TS7700 cluster before being included in a grid configuration are not automatically copied to the other TS7700 clusters in the configuration until they have been accessed
and closed.
This field can be used to determine which volumes in each TS7700 cluster have not been copied to build a set of jobs to access them, and force the copy. The PRESTAGE program from the TAPETOOL FTP site can support you in doing that job in an efficient way. The VEHSYNC job can be used to identify volumes needing copies.
Additional information about various tools available for monitoring your TS7700 is provided in 11.18.1, “VEHSTATS tool overview” on page 704. You can also access the TAPETOOL FTP site at the following URL:
•Copy Required for Cluster n
This field indicates that a copy to another TS7700 Cluster in a grid configuration is required. In cases where Deferred mode copy is used, this field can be used to determine whether a critical set of volumes have completed their copy operations to specific clusters.
•Volume Ownership and Volume Ownership Taken
At any point in time, a logical volume is owned by a specific cluster. If required, ownership is transferred as part of mount processing. If a logical volume is mounted on a virtual drive anywhere in the composite library, ownership will not be transferred until the volume is unloaded. Ownership can transfer in one of two ways:
 – Through communication with the current owning cluster
 – Through a recovery process called ownership takeover
Normally, if the cluster receiving a mount command does not have ownership of the volume, it requests the transfer of volume ownership from the current owning cluster. If the volume is not in use, ownership is transferred.
However, if the cluster receiving the mount request cannot communicate with the owning cluster, that method does not work. In this case, the requesting clusters cannot determine whether the owning cluster has failed or just the grid network links to it have failed. Operator intervention is required to indicate that the owning cluster has failed and that ownership takeover by the other clusters is allowed. Two types of ownership takeover are available:
 – Write ownership takeover (WOT): The cluster taking over ownership of the volume has complete freedom to modify the contents of the volume or modify any of the properties that are associated with the volume. This includes scratch mounts.
 – Read Ownership Takeover (ROT): The cluster taking over ownership of the volume is restricted to reading the volume’s data only. Therefore, a cluster in ROT mode fails a scratch mount request for which it is unable to acquire volume ownership.
•Current and Pending Category
One of the key properties that are associated with a volume is the category that it is assigned. The primary usage for category is to group scratch volumes together. A volume’s category assignment changes as the volume is used. The current category field indicates the category that the volume is assigned to within the TS7700 Integrated Library Manager function.
The pending category field indicates that a new category assignment is in progress for the volume. These fields can be used to determine whether the category assignments are in sync between the clusters and the host databases.
•Data Deleted
As part of normal processing in a TS7700 Cluster, you can specify that after a certain time after being returned to scratch, the contents of a volume can be deleted. This field indicates whether the data that is associated with the volume has been deleted on the cluster.
•Removal State
As part of normal processing in a TS7700 Grid configuration where a mixture of both TS7740 and TS7700D clusters exists, a data removal or migration process occurs where data is removed from TS7700D clusters to prevent TS7700D clusters from overrunning their TVC. This field, and the removal time stamp, can be used to determine whether the data that is associated with the volume has been removed.
•Hot
This field represents the cluster’s view of which clusters have obsolete token or volume metadata information as a result of a cluster outage. When clusters are unavailable because of expected or unexpected outages, the remaining clusters mark the unavailable cluster for pending reconciliation by updating this hot mask. The field represents both Insert or Eject pending updates, or regular pending updates.
Insert/Eject updates are related to volumes being inserted or ejected during the outage. Regular pending updates are for updates that occur to the volume during an outage as a result of normal operations, such as host I/O. Each bit within the mask represents which clusters are viewed as needing reconciliation.
Cache content information
This report provides a list of all volumes that are currently kept in cache. The contents of the cache that are associated with the specific cluster that the request volume is written to are returned in the response records. In a TS7700 grid configuration, separate requests must be sent to each cluster to obtain the cache contents.
The response records are written in 80-byte fixed block (FB) format.
 
Remember:
•The generation of the response might take several minutes to complete depending on the number of volumes in the cache and how busy the TS7700 cluster is at the time of the request.
•The contents of the cache typically are all private volumes. However, some might have been returned to SCRATCH status soon after being written. The TS7700 does not filter the cache contents based on the private or SCRATCH status of a volume.
Physical volume to logical volume mapping information
The TS7700 maintains the mapping between logical and physical volumes in a database on each cluster. It is possible that there are inconsistencies in the mapping information provided with this function. This results when a logical volume is being moved from one physical volume to another. For a while, the volume is shown on more than one physical volume. This can result in a few logical volumes that are reported as being on physical volumes that they were on in the past, but are not presently on.
Even with inconsistencies, the mapping data is useful if you want to design jobs that recall data efficiently from physical volumes. If the logical volumes that are reported on a physical volume are recalled together, the efficiency of the recalls is increased. If a logical volume with an inconsistent mapping relationship is recalled, it recalls correctly, but an extra mount of a separate physical volume might be required.
The physical volume to logical volume mapping that is associated with the physical volumes managed by the specific cluster to which the request volume is written is returned in the response records. In a TS7700 grid configuration, separate requests must be sent to each cluster to obtain the mapping for all physical volumes.
The response records are written in 80-byte FB format.
This report is only available for the TS7740 and TS7700T.
 
Tip: The generation of the response can take several minutes to complete depending on the number of active logical volumes in the library and how busy the TS7700 cluster is at the time of the request.
Physical media pools
The TS7740/TS7700T supports separating the physical volumes that it manages into pools. The supported pools include a pool that contains scratch (empty) volumes that are common, and up to 32 pools that can contain scratch (empty) and data (filling/full) volumes. Pools can borrow and return volumes from the common scratch pool. Each pool can contain several types of media.
For pool 0 (common scratch pool), because it contains only empty volumes, only the empty count is returned. Volumes that have been borrowed from the common pool are not included.
For pools 1 - 32, a count of the physical volumes that are empty, are empty and waiting for erasure, are being filled, and have been marked as full, is returned. The count for empty includes physical volumes that have been assigned to the pool and volumes that were borrowed from the common scratch pool but have not yet been returned.
The count of volumes that are marked as Read Only or Unavailable (including destroyed volumes) is returned. Also, the full data volumes contain a mixture of valid and invalid data. Response records are provided for the distribution of active data on the data volumes that are marked as full for a pool.
Information is returned for the common pool and all other pools that are defined and have physical volumes that are associated with them.
The physical media pool information that is managed by the specific cluster to which the request volume is written is returned in the response records. In a TS7700 grid configuration, separate requests must be sent to each cluster to obtain the physical media pool information for all clusters.
The response records are written in 80-byte FB format. Counts are provided for each media type associated with the pool (up to a maximum of eight).
Physical volume status
A database is maintained on each TS7740/TS7700T cluster that contains information that is related to the management of the physical volumes on the cluster. The physical volume status information that is returned represents the status of the volume or volumes on the cluster to which the request volume is written.
In a TS7700 grid configuration, separate requests must be sent to each cluster to obtain the physical volume status information for the individual clusters. A response record is written for each physical volume, selected based on the volume serial number mask or pool number that is specified in the request that exists in the cluster. A response record consists of the database fields that are defined as shown in Example 11-7 for Volume A03599. The overall length of each record is 400 bytes with blank padding after the last field, as needed.
Example 11-7 Sample response record for VOLSER A03599
A03599,2,FULL,READ-WRITE,2007-05-05-06.40.08.030061,2007-05-04-13.45.15.918473,...
 
Tip: The generation of the response might take several minutes to complete depending on the number of volumes that is requested and how busy the TS7700 cluster is at the time of
the request.
Copy audit
A database is maintained on each TS7700 cluster that contains status information about the logical volumes that are defined to the grid. Two key pieces of information are whether the cluster contains a valid copy of a logical volume and whether the copy policy for the volume indicates that it must have a valid copy.
This request runs an audit of the databases on a set of specified TS7700 distributed libraries to determine whether any volumes do not have a valid copy on at least one of them.
If no further parameter is specified, the Audit checks whether a logical volume has a copy on the specified cluster or not. There is no validation if that cluster should have a copy or not. To take the copy modes into account, you need to specify a second parameter COPYMODE.
Using COPYMODE
If the COPYMODE option is specified, whether the volume is supposed to have a copy on the distributed library is considered when determining whether that distributed library has a valid copy. If COPYMODE is specified and the copy policy for a volume on a specific cluster is “S”, “R”, “D”, or “T”, that cluster is considered during the audit.
If the copy policy for a volume on a specific cluster is “N”, the volume’s validity state is ignored because that cluster does not need to have a valid copy.
The request then returns a list of any volumes that do not have a valid copy, subject to the copy mode if the COPYMODE option is specified, on the TS7700 clusters specified as part of the request.
The specified clusters might not have a copy for several reasons:
•The copy policy that is associated with the volume did not specify that any of the clusters that are specified in the request were to have a copy and the COPYMODE option was not specified. This might be because of a mistake in defining the copy policy or because it
was intended.
For example, volumes that are used in a disaster recovery test need to be only on the disaster recovery TS7700 and not on the production TS7700s. If the request specified only the production TS7700 tape drives, all of the volumes that are used in the test are returned in the list.
•The copies have not yet been made from a source TS7700 to one or more of the specified clusters. This can be because the source TS7700 or the links to it are unavailable, or because a copy policy of Deferred was specified and a copy has not been completed when the audit was run.
•Each of the specified clusters contained a valid copy at one time, but has since removed it as part of the TS7700 hybrid automated removal policy function. Automatic removal can take place on TS7700D or TS7700T clusters in all configuration scenarios (hybrid or homogeneous). In a TS7700T, only data in CP0 is subject for autoremoval.
The Copy Audit is intended to be used for the following situations:
•A TS7700 is to be removed from a grid configuration. Before its removal, you want to ensure that the TS7700s that are to remain in the grid configuration have a copy of all the important volumes that were created on the TS7700 that is to be removed.
•A condition has occurred (because of a site disaster or as part of a test procedure) where one of the TS7700s in a grid configuration is no longer available and you want to determine which, if any, volumes on the remaining TS7700s do not have a valid copy.
In the Copy Audit request, you need to specify which TS7700 clusters are to be audited. The clusters are specified by using their associated distributed library ID (this is the unique five-character library sequence number that is defined when the TS7700 Cluster was installed). If more than one distributed library ID is specified, they are separated by a comma. The following rules determine which TS7700 clusters are to be included in the audit:
•When the INCLUDE parameter is specified, all specified distributed library IDs are included in the audit. All clusters that are associated with these IDs must be available or the audit fails.
•When the EXCLUDE parameter is specified, all specified distributed library IDs are excluded from the audit. All other clusters in the grid configuration must be available or the audit fails.
•Distributed library IDs specified are checked for being valid in the grid configuration. If one or more of the specified distributed library IDs are invalid, the Copy Audit fails and the response indicates the IDs that are considered invalid.
•Distributed library IDs must be specified or the Copy Audit fails.
•Here are examples of valid requests (assume a three-cluster grid configuration with distributed library IDs of DA01A, DA01B, and DA01C):
 – COPY AUDIT INCLUDE DA01A: Audits the copy status of all volumes on only the cluster that is associated with distributed library ID DA01A.
 – COPY AUDIT COPYMODE INCLUDE DA01A: Audits the copy status of volumes that also have a valid copy policy on only the cluster that is associated with distributed library ID DA01A.
 – COPY AUDIT INCLUDE DA01B,DA01C: Audits the copy status of volumes on the clusters that are associated with distributed library IDs DA01B and DA01C.
 – COPY AUDIT EXCLUDE DA01C: Audits the copy status of volumes on the clusters in the grid configuration that is associated with distributed library IDs DA01A and DA01B.
On completion of the audit, a response record is written for each logical volume that did not have a valid copy on any of the specified clusters. Volumes that have never been used, have had their associated data deleted, or have been returned to scratch are not included in the response records. The record includes the volume serial number and the copy policy definition for the volume. The VOLSER and the copy policy definitions are comma-separated, as shown in Example 11-8. The response records are written in 80-byte FB format.
Example 11-8 BVIR message when Copy Audit is requested
123456789012345678901234567890123456789012
L00001,R,D,D,N,N,N,N,N,N,R,N,R,N,N,N,N,N,R
 
Tips:
•The output for Copy Audit includes Copy Consistency Points for up to eight TS7700 clusters. This is to provide for future expansion of the number of clusters that are supported in a TS7700 Grid to the designed maximum.
•Copy Audit might take more than an hour to complete depending on the number of logical volumes that have been defined, how many clusters are configured in the grid configuration, and how busy the TS7700 tape drives are at the time of the request.
Unknown or invalid request
If the request file does not contain the correct number of records or the first record is incorrect, the request file on the volume is unchanged and no error is indicated.
If the request file contains the correct number of records and the first record is correct but the second is not, the response file indicates in Record 6 that the request is unknown, as shown in Example 11-9.
Example 11-9 BVIR message when an unknown or invalid request is submitted
12345678901234567890
UNKNOWN REQUEST TYPE
If the request file contains the correct number of records, the first record is correct, and the second is recognized but includes a variable that is not within the range that is supported for the request, the response file indicates in record 6 that the request is invalid, as shown in Example 11-10.
Example 11-10 BVIR message when an invalid variable is specified
12345678901234567890123456
INVALID VARIABLE SPECIFIED
11.15 Alerts and exception and message handling
The following section provides an overview about user-defined alerts in a TS7700, possible exceptions in a TS7700 environment, and upcoming messages. These messages can be used as input for an automation system and, depending on the user requirements, they should be sent with the automation to an email or alarm system.
11.15.1 Alerting of specific events
The TS7700 offers you a broad variety of threshold and timeout alerts. This section includes a brief introduction. For a detailed description, see the following white paper:
Most threshold alerts allow two thresholds per setting. The first one issues the “alarm” message that the threshold is now crossed. A second, lower limit that informs you when the condition is resolved, and that the amount of data has fallen beyond the threshold.
Before you adjust the values of the thresholds alerts, evaluate appropriate values for your installation. Use the performance evaluation tool in advance. Also, review the values after implementation and review them periodically (for example, twice a year) or after installation changes.
Values for the alerting that are too low lead to unnecessary messages and operator actions, Values that are too high might not give you enough time in a critical situation to react.
General statement for alerts
All of the following alerts send messages to the host system log if the pending inbound copy backlog indicates that the amount of uncopied data has exceeded the low or the critical warning limit specified (in GB). All parameters can be set independently, but the values have some dependencies that must be acknowledged. For all alerts, a message is created if the values are exceeded.
The default value for each parameter is 0, which indicates that no warning limit is set and messages are not generated.
Inbound backlog value (PCPYLOW and PCPYCRIT)
These values specify the threshold in GB of volumes waiting in the incoming copy queue. The PCPYLOW value defines the first level of warning. The PCPYCRIT represents the critical state of inbound copy backlog in GB.
These alerts are applicable for all TS7700s, and can be set for each distributed library independently.
Change the value with the following commands:
LI REQ,distributed library,SETTING,ALERT,PCPYLOW,value
LI REQ,distributed library,SETTING,ALERT,PCPYCRIT,value
Uncopied data in cache (COPYLOW and COPYRIT)
These values specify the threshold in GB of volumes waiting to be copied to another cluster in the TS7700 grid configuration. The COPYLOW value defines the first level of warning. The COPYCRIT represents a critical state of inbound copy backlog (in GB). These alerts are applicable for all TS7700s, and can be set for each distributed library independently.
Change the value with the following commands:
LI REQ,distributed library,SETTING,ALERT,COPYLOW,value
LI REQ,distributed library,SETTING,ALERT,COPYCRIT,value
Data in cache (RESDLOW and RESDHIGH)
These values specify the amount of data in cache. The RESDLOW value defines the first level of warning. The RESDHIGH represents a critical state of data in cache.
The following values are measured:
•TS7700D: All data in cache
•TS7740: Resident data in the cache that has not yet been copied to the back end
•TS7700T: Resident data in CP0
Change the value with the following commands:
LI REQ,distributed library,SETTING,ALERT,RESDLOW,value
LI REQ,distributed library,SETTING,ALERT,RESDHIGH,value
 
Important: The monitored values are different for each TS7700 model.
For the TS7740, if RESDLOW/RESDHIGH was exceeded, check the number of available tape drives and empty cartridges in the TS7740, especially if you have multiple physical pools. Also, review the number of premigration tasks and reclaim schedules.
For a TS7700D and TS7700T CP0, if RESDLOW/RESDHIGH was exceeded, check the amount of Pinned data, and the amount of data that is subject to auto removal. If the data is filled up with Pinned data and the data will not be expired by the host in the near future, you might run out of cache.
Data in cache (RSDTLOW and RSTDHIGH)
These alerts are similar to those described previously, but now for the TS7700T tape partitions. The same actions apply as for the TS7740 when the thresholds are exceeded. The RESTDLOW value defines the first level of warning. The RESTDHIGH represents a critical state of data in cache of non-premigrated data from all tape partitions.
Change the value with the following commands:
LI REQ,distributed library,SETTING,ALERT,RSDTLOW,value
LI REQ,distributed library,SETTING,ALERT,RSDTHIGH,value
Physical drive availability (PDRVLOW and PDRVCRIT)
These values specify the number of available backend drives in a TS7700T or TS7740. The PDRVLOW value defines the first level of warning. The PDRVCRIT represents a critical state. They are applicable only for TS7700T and TS7740 and can be set for each distributed library independently.
Change the value with the following commands:
LI REQ,distributed library,SETTING,ALERT,PDRVLOW,value
LI REQ,distributed library,SETTING,ALERT,PDRVCRIT,value
Physical scratch availability (PSCRLOW and PSCRCRIT)
These values specify the number of available backend cartridges in a TS7700T or TS7740. The PDRVLOW value defines the first level of warning. The PDRVCRIT represents a
critical state.
Change the value with the following commands:
LI REQ,distributed library,SETTING,ALERT,PSCRLOW,value
LI REQ,distributed library,SETTING,ALERT,PSCRCRIT,value
11.15.2 Handling Replication Exceptions
In certain conditions, the selected replication mode cannot be run. This is true for Synchronous mode copy and Immediate (RUN) copies.
To ensure, that the production is not impacted, a switch from the origin replication mode to a “Sync-Deferred” or “Immed-Deferred” is possible. For the synchronous mode copy, the user can define whether the synchronous mode copy will change to a “SYNC-Deferred” state, or if the job will fail.
Immediate-copy set to immediate-deferred state
The goal of an immediate copy is to complete one or more RUN consistency point copies of a logical volume before surfacing status of the RUN command to the mounting host. If one or more of these copies cannot complete, the replication state of the targeted volume enters the immediate-deferred state.
The volume remains in an immediate-deferred state until all of the requested RUN consistency points contain a valid copy. The immediate-deferred volume replicates with a priority greater than standard deferred copies, but lower than non-deferred immediate copies.
There are numerous reasons why a volume might enter the immediate-deferred state. For example, it might not complete within 40 minutes, or one or more clusters that are targeted to receive an immediate copy are not available. Independently of why a volume might enter the immediate-deferred state, the host application or job that is associated with the volume is not aware that its previously written data has entered the immediate-deferred state.
The reasons why a volume moves to the immediate-deferred state are contained in the Error Recovery Action (ERA) 35 sense data. The codes are divided into unexpected and expected reasons. From a z/OS host view, the ERA is part of message IOS000I (Example 11-11).
Example 11-11 Message IOS000I
IOS000I 1029,F3,EQC,0F,0E00,,**,489746,HADRMMBK  745                          
.50408035000000206011(B31000011C005800)2182(50000FFF)CE1F0720EED40900         
IMMEDIATE MODE COPY - EXPECTED FAILURE - REASON CODE = 82                    
COPY COUNT - 1 OF 2 COMPLETED SUCCESSFULLY
New failure content is introduced into the CCW(RUN) ERA35 sense data:
•Byte 14 FSM Error. If set to 0x1C (Immediate Copy Failure), the additional new fields are populated.
•Byte 18 Bits 0:3. Copies Expected: Indicates how many RUN copies were expected for this volume.
•Byte 18 Bits 4:7. Copies Completed: Indicates how many RUN copies were verified as successful before surfacing Sense Status Information (SNS).
•Byte 19. Immediate Copy Reason Code:
 – Unexpected - 0x00 to 0x7F: The reasons are based on unexpected failures:
 • 0x01 - A valid source to copy was unavailable.
 • 0x02 - Cluster that is targeted for a RUN copy is not available (unexpected outage).
 • 0x03 - 40 minutes have elapsed and one or more copies have timed out.
 • 0x04 - Is reverted to immediate-deferred because of health/state of RUN target clusters.
 • 0x05 - Reason is unknown.
 – Expected - 0x80 to 0xFF: The reasons are based on the configuration or a result of planned outages:
 • 0x80 - One or more RUN target clusters are out of physical scratch cache.
 • 0x81 - One or more RUN target clusters are low on available cache
(95%+ full).
 • 0x82 - One or more RUN target clusters are in service-prep or service.
 • 0x83 - One or more clusters have copies that are explicitly disabled through the Library Request operation.
 • 0x84 - The volume cannot be reconciled and is “Hot” against peer clusters.
The additional data that is contained within the CCW(RUN) ERA35 sense data can be used within a z/OS custom user exit to act on a job moving to the immediate-deferred state.
Because the requesting application that results in the mount has already received successful status before sending the CCW(RUN) command, it cannot act on the failed status. However, future jobs can be suspended or other custom operator actions can be taken by using the information that is provided within the sense data.
Handling of the Immed-Deferred related messages
For each volume that cannot be replicated in the immed-Deferred mode, an IOS000I can be reported in the system log. Although that allows a message automation on the host to detect possible problems, the thousands of messages that are generated might overflow the host log. This is especially true in maintenance situations. A Host Console Request (HCR) command enables you to handle all OS000I triggered by “IMMED-Deferred.” The following options are available:
LI REQ,distributed library,SETTING,COPY,IMMSNS,[ALL|NONE|UNEXP]
•ALL: All messages are presented to the host.
•NONE: All messages are not presented to the host, except if no valid copy source is available. This is the default.
•UNEXP: Only unexpected failures will be presented to the host. All messages during a maintenance situation or during “Gridlink Disable” will not be presented because they are the result of a customer action.
Synchronous mode copy set to synchronous deferred
The default behavior of Synchronous mode copy (SMC) is to fail a write operation if both clusters with the “S” copy policy are not available or become unavailable during write operations. You can enable the Synchronous Deferred on Write Failure (SDWF) option to permit update operations to continue to any valid consistency point in the grid. If there is a write failure, the failed “S” locations are set to a state of “synchronous-deferred.”
After the volume is closed, any synchronous-deferred locations are updated to an equivalent consistency point through asynchronous replication. If the SDWF option is not selected (default) and a write failure occurs at either of the “S” locations, host operations fail and you must view only content up to the last successful sync point as valid.
For example, imagine a three-cluster grid and a copy policy of Sync-Sync Deferred (SSD), Sync Copy to Cluster 0 and Cluster 1, and a deferred copy to Cluster 2. The host is connected to Cluster 0 and Cluster 1. With this option disabled, both Cluster 0 and Cluster 1 must be available for write operations. If either one becomes unavailable, write operations fail. With the option enabled, if either Cluster 0 or Cluster 1 becomes unavailable, write operations continue. The second “S” copy becomes a synchronous-deferred copy.
In the previous example, if the host is attached to Cluster 2 only and the option is enabled, the write operations continue even if both Cluster 0 and Cluster 1 become unavailable. The “S” copies become synchronous-deferred copies.
The synchronous-deferred volume replicates with a priority greater than immediate-deferred and standard-deferred copies.
For more information about Synchronous mode copy, see the IBM Virtualization Engine TS7700 Series Best Practices - Synchronous Mode Copy white paper on Techdocs.
Handling of composite status change due to replication issues
When a cluster detects a “SYNC-Deferred” or an “Immed-Deferred” condition, a degradation of the composite library is reported. Although these conditions might have an impact to your disaster recovery capability, the production jobs might not be impacted at all. This is especially true if the cluster is not available due to maintenance purposes.
Therefore, a Host Console Request Command is provided, which enables you to define whether these conditions report the composite library as degraded or not:
LI REQ,composite library,SETTING,ALERT,DEFDEG, [ENABLE|DISABLE]
Grid link exception handling
The TS7700 generates a host message when it detects the grid performance is degraded. If the degraded condition persists, a call-home link is generated. The performance of the grid links is monitored periodically, and if one link is performing worse than the other link by an IBM Service Support Representative (IBM SSR)-alterable value, a warning message is generated and sent to the host. The purpose of this warning is to alert you that an abnormal grid performance difference exists. The value must be adjusted so that warning messages are not generated because of normal variations of the grid performance. Here is the warning message format:
CBR3750I, G0030, Library xx Primary, Alternate, Primary2, and Alternate2 grid links are degraded. The degraded grid link port is reported.
A second message with a variable text “EA480E” is reported in the syslog.
For example, a setting of 60% means that if one link is running at 100%, the remaining links are marked as degraded if they are running at less than 60% of the 100% link. The grid link performance is available with the Host Console Request function, and on the TS7700 MI. The monitoring of the grid link performance by using the Host Console Request function is described in detail in the IBM Virtualization Engine TS7700 Series z/OS Host Command Line Request User’s Guide, which is available on Techdocs:
Use the STATUS and GRIDLINK keywords.
The grid link degraded threshold also includes two other values that can be set by the SSR:
•Number of degraded iterations: The number of consecutive 5-minute intervals that link degradation was detected before reporting an attention message. The default value is 9.
•Generate Call Home iterations: The number of consecutive 5-minute intervals that link degradation was detected before generating a Call Home. The default value is 12.
The default values are set to 60% for the threshold, nine iterations before an attention message is generated, and 12 iterations before a Call Home is generated. Use the default values unless you are receiving intermittent warnings and support indicates that the values need to be changed. If you receive intermittent warnings, let the SSR change the threshold and iteration to the suggested values from support.
For example, suppose that clusters in a two-cluster grid are 2000 miles apart with a round-trip latency of approximately 45 ms. The normal variation that is seen is 20 - 40%. In this example, the threshold value is at 25% and the iterations are set to 12 and 15.
11.16 IBM Tape Tools
A set of tape tools is available on an as-is basis to help you monitor your tape environment. Several of these tools are specific to the TS7700 and are based on BVIR data, such as VEHSTATS. Before describing VEHSTATS and the reports that you can obtain by running the VEHSTATS jobs, a general overview of the IBM Tape Tools is provided to guide you through the installation of these tools in a z/OS environment.
11.16.1 Introduction to IBM Tape Tools
All the TS7700 monitoring and evaluating tools that are described are at the following FTP site:
Example 11-12 lists the content of the Readme.txt file that provides basic information about the tape tools.
Example 11-12 Readme.txt from the IBM Tape Tools website
IMPORTANT IMPORTANT
Program enhancements will be made to handle data format changes when they occur. If you try to run new data with old program versions, results will be unpredictable. To avoid this situation, you need to be informed of these enhancements so you can stay current. To be informed of major changes to any of the tools that are distributed through this FTP site, send an email message to:
 
 
In the subject, specify NOTIFY. Nothing else is required in the body of the note. This will add you to our change distribution list.
 
The UPDATES.TXT file will contain a chronological history of all changes made to the tools. You should review that file on a regular basis, at least monthly, perhaps weekly, so you can see whether any changes apply to you.
 
Look in file, OVERVIEW.PDF, for an overview of all currently available tools.
The JCL, CNTL, and LOAD libraries for all the tools are compressed into IBMTOOLS.EXE.
IBMTOOLS.TXT explains the complete installation procedure.
Most tools have their own xxxxxx.txt file with more detail. There are no formal documentation manuals. The intent is to have enough JCL comment to allow the user to run the jobs without difficulty and to have adequate column headings and footnotes to make report output obvious without needing additional documentation.
 
If you feel that the JCL or report output needs more explanation, send an email to the address above indicating the area needing attention.
Most of these tools are z/OS-based and included in the ibmtools.exe file. A complete list of all tools that are included in the ibmtools.exe file is available in the overview.pdf file.
Table 11-8 Tape tools selection
Tool
Major use
Benefit
Inputs
Outputs
BADBLKSZ
Identify small VTS blocksizes
Improve VTS performance, make jobs run faster.
Logrec MDR & CA1, TLMS, RMM, ZARA,CTLT
VOLSER, Jobname, and Dsname for VTS volumes with small blocksizes.
BVIRHSTx
Get historical stats from TS7700
Creates U, VB, SMF format.
TS7700
Statistics file.
 
BVIRPOOL
Identify available scratch by pool
Reports all pools at the same time.
BVIR file
Physical media by pool.
BVIRPRPT
Reclaim Copy Export volumes
Based on active GB,
not %.
BVIR file
Detailed report of data on volumes.
BVIRRPT
Identify VTS virtual volumes by owner
Determine which applications or users have virtual volumes.
BVIR data & CA1, TLMS, RMM, ZARA, CTLT
Logical volumes by jobname or dsname, logical to physical reports.
BVPITRPT
Point in Time stats as write to operator (WTO)
Immediately available.
TS7700
Point in Time stats as WTO.
COPYVTS
Copy lvols from old VTS
Recall lvols based on selected applications.
BVIR data & CA1, TLMS, RMM, ZARA, CTLT
IEBGENER to recall lvols and copy to new VTS.
DIFFEXP
Identify multi-file volumes with different expiration dates
Prevent single file from not allowing volume to return to scratch.
CA1, TLMS, RMM, ZARA, CTLT
List of files not matching
file 1 expiration date .
FSRMATCH
Replace *.HMIGTAPE.DATASET in SMF 14 with actual recalled dsname
Allows TapeWise and other tools by using SMF 14/15 data to report actual recalled data set.
FSR records plus SMF 14, 15, 21, 30, 40, and so on
Updated SMF 14s plus all other SMF records as they were.
GETVOLS
Get VOLSERs from list of dsns
Automate input to PRESTAGE.
CA1, TLMS, RMM, ZARA, CTLT
VOLSERs for requested dsns .
IOSTATS
Report job elapsed times
Show runtime improvements.
SMF 30 records
Job-step detailed reporting.
MOUNTMON
Monitor mount pending and volume allocations
Determine accurate mount times and concurrent drive allocations.
Samples tape UCBs
Detail, summary, distribution, hourly, TGROUP, and system reporting.
ORPHANS
Identify orphan data sets in Tape Management Catalog (TMC)
Cleanup tool.
CA1, TLMS, RMM, ZARA, CTLT
Listing file showing all multiple occurrence generation data group (GDGs) that have not been created in the last nn days.
PRESTAGE
Recall lvols to VTS
Ordered and efficient.
BVIR VOLUME MAP
Jobs that are submitted to recall lvols.
SMFILTER
IFASMFDP exit or E15 exit
Filters SMF records to keep just tape activity. Generates “tape” records to simulate optical activity.
SMF data
Records for tape activity plus optional TMM or optical activity.
TAPECOMP
Show current tape compression ratios
See how well data will compress in VTS.
Logrec MDR or EREP history file
Shift and hourly reports showing current read and write compression ratios.
TAPEWISE
Identify tape usage improvement opportunities
Shows UNIT=AFF, early close, UNIT=(TAPE,2), multi-mount, DISP=MOD, recalls.
SMF 14, 15, 21, 30, and 40
Detail, summary, distributions, hourly, TGROUP, and system reporting.
TCDBMCH
Identify tape configuration database (TCDB) versus Tape Catalog mismatches
List VOLSER mismatches.
CA1, TLMS, RMM, ZARA, CTLT
ERRRPT with mismatched volumes.
TMCREUSE
Identify data sets with create date equal to last ref date
Get candidate list for VTS PG0.
CA1, TLMS, RMM, ZARA, CTLT F
Filter list of potential PG0 candidates.
VEHGRXCL
Graphing package
Graphs TS7700 activity.
VEHSTATS flat files
Many graphs of TS7700 activity.
VEHSCAN
Dump fields in historical statistics file
Individual field dump.
BVIR stats file
DTLRPT for selected interval.
VEHSTATS
TS7700 historical performance reporting
Show activity on and performance of TS7700.
BVIRHSTx file
Reports showing mounts, data transfer, and box usage.
VEPSTATS
TS7700 point-in-time statistics
Snapshot of last 15 seconds of activity plus current volume status.
BVIRPIT data file
Reports showing current activity and status.
VESYNC
Synchronize TS7700 after new cluster added
Identify lvols that need copies.
BVIR data and CA1, TLMS, RMM, ZARA, CTLT
List of all VOLSERs to recall by application.
VOLLIST
Show all active VOLSERs from TMC. Also, get volume counts by group, size, and media.
Used to get a picture of user data set naming conventions. See how many volumes are allocated to different application s.
CA1, TLMS, RMM, ZARA, CTLT
Dsname, VOLSER, create date, and volseq. Group name, counts by media type.
11.16.2 Tools download and installation
Public access is provided to the IBM Tape Tools library, which contains various tools that can help you analyze your tape environment. This set of utilities also includes the VEHSTATS and VEPSTATS tools, which use the Bulk Volume Information Retrieval (BVIR) reports for comprehensive performance analysis.
Figure 11-43 shows several tools that are available from the FTP site.
Figure 11-43 Tape tools catalog
The index is at the following web address:
IBM employees can access IBM Tape Tools on the following website:
IBM Business Partners can access IBM Tape Tools on the following website:
For most tools, a text file is available. In addition, each job to run a tool contains a detailed description of the function of the tool and parameters that need to be specified.
 
Important: For the IBM Tape Tools, there are no warranties, expressed or implied, including the warranties of merchantability and fitness for a particular purpose.
To obtain the tape tools, download the ibmtools.exe file to your computer or use FTP from Time Sharing Option (TSO) on your z/OS system to directly upload the files that are contained in the ibmtools.exe file.
The ibmtools.exe file is a self-extracting compressed file that is expanded into four separate files:
IBMJCL.XMI Contains the execution JCL for current tape analysis tools.
IBMCNTL.XMI Contains parameters that are needed for job execution, but that do not need to be modified by the user.
IBMLOAD.XMI Contains the load library for executable load modules.
IBMPAT.XMI Contains the data pattern library, which is only needed if you run the QSAMDRVR utility.
The ibmtools.txt file contains detailed information about how to download and install the tools libraries.
After you have created the three or four libraries on the z/OS host, be sure that you complete the following steps:
1. Copy, edit, and submit userid.IBMTOOLS.JCL($$CPYLIB) to create a new JCL library that has a unique second node (&SITE symbolic). This step creates a private JCL library for you from which you can submit jobs while leaving the original as is. CNTL and LOAD can then be shared by multiple users who are running jobs from the same system.
2. Edit and submit userid.SITENAME.IBMTOOLS.JCL($$TAILOR) to tailor the JCL according to your system requirements.
The updates.txt file contains all fixes and enhancements made to the tools. Review this file regularly to determine whether any of the programs that you use have been modified.
To ensure that you are not working with outdated tools, the tools are controlled through an EXPIRE member. Every three months, a new EXPIRE value is issued that is good for the next 12 months. When you download the current tools package any time during the year, you have at least nine months remaining on the EXPIRE value. New values are issued in the middle of January, April, July, and October.
If your IBM tools jobs stop running because the expiration date has passed, download the ibmtools.exe file again to get the current IBMTOOLS.JCL(EXPIRE) member.
11.16.3 IBM Tape Tools for TS7700 monitoring
Several tape tools that can be used to help you better understand your tape processing regarding TS7700 operation and migration are described.
IOSTATS
IOSTATS tool is part of the ibmtools.exe file, which is available at the following URL:
You can use IOSTATS tool to measure job execution times. For example, you might want to compare the TS7700 performance before and after configuration changes.
IOSTATS can be run for a subset of job names for a certain period before the hardware installation. SMF type 30 records are required as input. The reports list the number of disk and tape I/O operations that were done for each job step, and the elapsed job execution time.
With the TS7700  running in a multi-cluster grid configuration, IOSTATS can be used for the following purposes:
•To evaluate the effect of the multi-cluster grid environment and to compare job execution times before implementation of the multi-cluster grid to those after migration, especially if you are operating in immediate copy (RUN, RUN data consistency point) mode.
•To evaluate the effect of hardware upgrades and to compare job execution times before and after upgrading components of the TS7700. For example, you might want to verify the performance impact of a larger TVC capacity or the number of TS1130/TS1120/3592 tape drives.
•To evaluate the effect of changing the copy mode of operation on elapsed job execution time.
TAPEWISE
As with IOSTATS, TAPEWISE tool is available from the IBM Tape Tools FTP site. TAPEWISE can, based on input parameters, generate several reports:
•Tape activity analysis, including reads and Disp=mod analysis
•Mounts and MBs processed by hour
•Input and output mounts by hour
•Mounts by SYSID during an hour
•Concurrent open drives used
•Long VTS mounts (recalls)
MOUNTMON
As with IOSTATS, MOUNTMON is available from the IBM Tape Tools FTP site. MOUNTMON runs as a started task or batch job and monitors all tape activity on the system. The program must be authorized program facility (APF)-authorized and, if it runs continuously, it writes statistics for each tape volume allocation to SMF or to a flat file.
Based on data that is gathered from MOUNTMON, the MOUNTRPT program can report on the following information:
•How many tape mounts are necessary
•How many are scratch
•How many are private
•How many by host system
•How many by device type
•How much time is needed to mount a tape
•How long tapes are allocated
•How many drives are being used at any time
•What is the most accurate report of concurrent drive usage
•Which jobs are allocating too many drives
11.17 Using Volume Mount Analyzer
The Volume Mount Analyzer (VMA) is part of the DFSMS suite and is integrated in the DFSMS code. Based on the SMF 14, 15, 21, and 30, you can produce several reports, similar to some of the functions of Tapewise:
•Mounts and MBs processed by hour
•Concurrent logical drives used
•Logical volume distribution (single-file or multifile, and single-volume or multivolume)
The VMA has a filter option, which can be used to create separated reports for the following specific items:
•SYSID
•Drive Types
•Device Ranges
•High-level qualifier
•Accounting codes
Also, some so-called “Top” reports can be produced to get an overview of the tape usage. For more information, see z/OS DFSMS Using the Volume Mount Analyzer, SC23-6895.
11.18 Using VEHSTATS and VEHGRXCL for monitoring and reporting
This section shows how to work with the binary reports for point-in-time and historical statistics after you use the BVIR functions that are described in Appendix E, “Sample job control language” on page 853. Some of the response data of the BVIR functions is already in a readable format. For the remaining binary format data provided by the point-in-time statistics and historical statistics, you need a formatting tool. IBM provides a tool called VEHSTATS. Further information about where to download this tool and how to use it is in 11.15, “Alerts and exception and message handling” on page 692.
To convert the binary response record from BVIR data to address your requirements, you can use the IBM tool VEHSTATS when working with historical statistics. When working with point-in-time statistics, you can use the IBM tool VEPSTATS. See 11.16.2, “Tools download and installation” on page 700 for specifics about where to obtain these tools. Details about using BVIR are in the IBM Virtualization Engine TS7700 Series Bulk Volume Information Retrieval function User’s Guide.
The most recently published white papers are available at the Techdocs website by searching for TS7700 at the following address:
With the record layout of the binary BVIR response data, you can decode the binary file or you can use the record layout to program your own tool for creating statistical reports.
Some of the VEHSTATS reports where already discussed in previous sections. This section contains only information that is not yet covered.
11.18.1 VEHSTATS tool overview
The TS7700 activity is recorded in the subsystem. There are two types of statistics:
•Point-in-time statistics: A snapshot of activity in the last 15 seconds
•Historical statistics: Up to 90 days in 15-minute increments
Both sets of statistics can be obtained through the BVIR functions (see Appendix E, “Sample job control language” on page 853).
Because both types of statistical data are delivered in binary format from the BVIR functions, you must convert the content into a readable format.
You can do this task manually by using the information that is provided in the following documents:
•IBM TS7700 Series Statistical Data Format White Paper
•TS7700 VEHSTATS Decoder reference
Alternatively, you can use an existing automation tool. IBM provides a historical statistics tool called VEHSTATS. Like the other IBM Tape Tools, the program is provided as-is, without official support, for the single purpose of showing how the data might be reported. There is no guarantee of its accuracy, and there is no additional documentation available for this tool. Guidance for interpretation of the reports is available in 11.18.3, “VEHSTATS reports” on page 707.
You can use VEHSTATS to monitor TS7700 virtual and physical tape drives, and TVC activity to do trend analysis reports, which are based on BVIR binary response data. The tool summarizes TS7700 activity on a specified time basis, up to 90 days in time sample intervals of 15 minutes or 1 hour, depending on the data reported.
Figure 11-43 on page 701 highlights three files that might be helpful in reading and interpreting VEHSTATS reports:
•The TS7700.VEHSTATS.Decoder file contains a description of the fields that are listed in the various VEHSTATS reports.
•The VEHGRXCL.txt file contains the description for the graphical package that is contained in VEHGRXCL.EXE.
•The VEHGRXCL.EXE file contains VEHSTATS_Model.ppt and VEHSTATS_Model.xls. You can use these files to create graphs of cluster activity based on the flat files that are created with VEHSTATS. Follow the instructions in the VEHSTATS_Model.xls file to create these graphs.
11.18.2 Running the VEHSTATS jobs
You have several output options for VEHSTATS, and you must submit separate jobs, depending on your requirements. The IBMTOOLS.JCL member VEHSTATS (Example 11-13) provides guidance about which job to choose.
Example 11-13 Member VEHSTATS
THERE ARE NOW DIFFERENT VERSIONS OF THE VEHSTATS JOB DEPENDING
ON HOW YOU WANT TO VIEW OR SAVE THE REPORTS.
1. VEHSTSO WRITES REPORTS DIRECTLY TO SYSOUT (THIS IS OLD VEHSTATS)
2. VEHSTPS WRITES FINAL REPORTS TO A SINGLE PHYSICAL SEQUENTIAL
FILE WHERE REPORTS ARE WRITTEN WITH DISP=MOD.
3. VEHSTPO WRITES FINAL REPORTS TO A PDSE WHERE EACH REPORT IS A
SEPARATE MEMBER.
In addition to the VEHSTATS tool, sample BVIR jobs are included in the IBMTOOLS libraries. These jobs help you obtain the input data from the TS7700. With these jobs, you can control where the historical statistics are accumulated for long-term retention.
The TS7700 still maintains historical statistics for the previous 90 days, but you can have the pulled statistics recorded directly to the SMF log file, or continue to use the disk flat file method. The flat files can be recorded as either RECFM=U or RECFB=VB.
Three specific jobs in IBMTOOLS.JCL are designed to fit your particular needs:
BVIRHSTS To write statistics to the SMF log file
BVIRHSTU To write statistics to a RECFM=U disk file
BVIRHSTV To write statistics to a RECFM=VB disk file
BVIR volumes cannot be written with LWORM attributes. Ensure that the BVIR logical volumes have a Data Class without LWORM specification.
The VEHSTATS reporting program accepts any or all of the various formats of BVIR input. Define which input is to be used through a data definition (DD) statement in the VEHSTATS job. The three input DD statements are optional, but at least one of the statements that are shown in Example 11-14 must be specified.
Example 11-14 VEHSTATS input DD statements
//* ACTIVATE ONE OR MORE OF THE FOLLOWING DD STATEMENTS FOR YOUR DATA
//*STATSU DD DISP=SHR,
//* DSN=&USERHLQ..#&VTSID..BVIRHIST.D070205.D070205
//*STATSVB DD DISP=SHR,
//* DSN=&USERHLQ..#&VTSID..BVIRHIST.D070206.D070206
//*STATSMF DD DISP=SHR, RECORDS WILL BE SELECTED BASED ON SMFNUM
//* DSN=&USERHLQ..#&VTSID..SMF194
The SMF input file can contain all SMF record types that are kept by the user. The SMFNUM parameter defines which record number is processed when you specify the STATSMF statement.
The fields that are shown in the various reports depend on which ORDER member in IBMTOOLS.JCL is being used. Use the following steps to ensure that the reports and the flat file contain the complete information that you want in the reports:
1. Review which member is defined in the ORDER= parameter in the VEHSTATS job member.
2. Verify that none of the fields that you want to see have been deactivated as indicated by an asterisk in the first column. Example 11-15 shows sample active and inactive definitions in the ORDERV12 member of IBMTOOL.JCL. The sample statements define whether you want the amount of data in cache to be displayed in MB or in GB.
Example 11-15 Sample statements in the ORDERV12 member
*ORDER=' PG0 MB IN TVC'; PG0 MEGABYTES IN CACHE
*ORDER=' PG1 MB IN TVC'; PG1 MEGABYTES IN CACHE
ORDER=' PG0 GB IN TVC'; PG0 GIGABYTES IN CACHE
ORDER=' PG1 GB IN TVC'; PG1 GIGABYTES IN CACHE
If you are planning to create graphics from the flat file by using the graphics package from the IBM Tape Tools FTP site, specify the ORDERV12 member because it contains all the fields that are used when creating the graphics, and verify that all statements are activated for all clusters in your environment.
If your cluster numbers do not start with 0, or the numbering has gaps, you need to use another parameter adjustment in the JCL. With “Define Distributed Library” = DEFDL, you can specify the order of your clusters, as shown in the following example:
DEFDL= H3833 1;
DEFDL= H6395 2;
11.18.3 VEHSTATS reports
VEHSTATS can be used to monitor TS7700 drive and TVC activity, and to run trend analysis to see where the performance bottlenecks are. Also, comparative analysis can be used to determine whether an upgrade, such as adding more physical tape drives, might improve the overall performance of the TS7740. VEHSTATS is not a projection tool, but it provides the basis for an overall health check of the TS7700.
VEHSTATS gives you a huge amount of information. The following list shows the most important reports available for the TS7700, and the results and analysis that can help you understand the reports better:
•H20VIRT: Virtual Device Historical Records
•H21ADP0x: vNode Adapter Historical Activity
•H21ADPXX: vNode Adapter Historical Activity combined (by adapter)
•H21ADPSU: vNode Adapter Historical Activity combined (total)
•H30TVC1: hnode HSM Historical Cache Partition:
 – For a TS7700D and TS7740, this report represents the cache.
 – For a TS7700T, multiple TVCs (TVC2 and TVC3) are presented. TVC1 contains the data from CP0, TVC2 contains the data of CP1, and so on.
•H31IMEX: hNode Export/Import Historical Activity
•H32TDU12: hNode Library Historical Drive Activity
•H32CSP: hNode Library Hist Scratch Pool Activity
•H32GUPXX: General Use Pools 01/02 through General Use Pools 31/32
•H33GRID: hNode Historical Peer-to-Peer (PTP) Activity
•AVGRDST: Hrs Interval Average Recall Mount Pending Distribution
•DAYMRY: Daily Summary
•MONMRY: Monthly Summary
•COMPARE: Interval Cluster Comparison
•HOURFLAT: 15-minute interval or 1-hour interval
•DAYHSMRY: Daily flat file
 
Tip: Be sure that you have a copy of TS7700 VEHSTATS Decoder and the TS7700 Statistical white paper available when you familiarize yourself with the VEHSTATS reports.
Virtual Device Activity
Example 11-16 on page 708 shows the report for Virtual Device Activity. This report gives you an overview, per 15-minute interval, of the relevant time frame and shows the following information:
•The minimum, average, or maximum (MIN, AVG, or MAX) mounted virtual drives
•The amount of channel blocks written based on blocksize
 
Clarification: The report is provided per cluster in the grid. The report title includes the cluster number in the DIST_LIB_ID field.
Example 11-16 VEHSTATS report for Virtual Drive Activity - first half
1(C) IBM REPORT=H20VIRT (15102) VNODE VIRTUAL DEVICE
GRID#=00186 DIST_LIB_ID= 2 VNODE_ID= 0 NODE_SERIAL=CL2H6395
03JUN15WE -VIRTUAL_DRIVES- _THROUGHPUT_ PCT_OF _
RECORD --MOUNTED-- MAX ATTMPT Delay_/15Sec 15Sec
TIME INST MIN AVG MAX THRPUT THRPUT MAX AVG INTVLS
R2.2 CALC <----R3.0.0063----> <
06:00:00 256 0 6 12 100 507 .803 .200 69
07:00:00 256 5 8 12 100 502 .801 .257 85
08:00:00 256 3 8 12 100 497 .799 .230 81
09:00:00 256 3 8 12 100 515 .806 .256 86
10:00:00 256 0 1 9 100 432 .769 .190 48
11:00:00 256 0 0 0 100 less .000 .000 0
12:00:00 256 0 0 0 100 less .000 .000 0
13:00:00 256 0 10 25 100 684 .854 .413 67
Example 11-17 shows the second half of the report.
Example 11-17 VEHSTATS report for Virtual Drive Activity - second half
HISTORICAL RECORDS RUN ON 13JUL2015 § 13:40:49 PAGE 24
VE_CODE_LEVEL=008.033.000.0025 UTC NOT CHG
____CLUSTER VS FICON CHANNEL______
AHEAD AHEAD BEHIND BEHIND ---------CHANNEL_BLOCKS_WRITTEN_F
MAX AVG MAX AVG <=2048 <=4096 <=8192
------------R3.1.0073+----------->
7585 3551 1064 242 17540 0 25650
7626 4600 1239 326 22632 0 32400
7638 4453 958 325 21943 0 31350
7491 4553 974 353 22913 0 32700
7664 2212 1564 387 14092 0 19500
0 0 0 0 0 0 0
0 0 0 0 0 0 0
8521 4534 713 108 19101 0 32063
With R3.1, new information was included. CLUSTER VS FICON CHANNEL shows you whether the TS7700 can take more workload (called ahead), or if the FICON tries to deliver more data than the TS7700 can accept at this specific point in time (called behind). You will normally see that numbers in both columns are shown.
Use the ratio of both numbers to understand the performance of your TS7700. In our example, the TS7700 is behind only 8% of the time in an interval. The TS7700 can handle more workload than is delivered from the host.
In addition, the report shows the CHANNEL BLOCKS WRITTEN FOR BLOCKSIZES. In general, the largest number of blocks are written at 65,546 or higher blocksize, but this is not a fixed rule. For example, DFSMShsm writes a 16,384 blocksize, and DB2 writes blocksizes up to 256,000. The report contains more differences for blocksizes, but are not shown in the example. From an I/O point of view, analysis of the effect of blocksize on performance is outside the scope of this book.
vNode Host Adapter Activity
The next example report provides details about the vNode Host Adapter Activity. Although there is a large amount of information available (one report per distributed library per FICON adapter), the vNode adapter Historical Activity Combined report is sufficient to provide an overall view of the FICON channel performance. As always, one report exists for each distributed library. This report is on an hourly basis with the following information:
•Total throughput per distributed library every hour
•Read and write channel activity
•Read and write device activity with compression rate achieved
Example 11-18 shows a sample report for Adapter 3 of Cluster 0.
Example 11-18 Adapter 3 sample report
C) IBM REPORT=H21ADP03(10210) VNODE ADAPTOR HISTORICAL ACTIVITY RUN ON 18AUG2010 @ 8:04:29 PAGE 30
GRID#=CC001 DIST_LIB_ID= 0 VNODE_ID= 0 NODE_SERIAL=CL0ABCDE     VE_CODE_LEVEL=008.006.000.0110                  UTC NOT CHG
ADAPTOR 3 FICON-2 (ONLINE ) R DRAWER SLOT# 6
19JUL10MO PORT 0 MiB is 1024 based, MB is 1000 based PORT 1
   RECORD GBS MB/ ----CHANNEL-------------- ----------DEVICE--------- GBS MB/ ---------CHANNEL-------   ----------DEVICE-----
TIME RTE sec RD_MB MB/s WR_MB MB/s RD_MB COMP WR_MB COMP RTE sec RD_MB MB/s WR_MB MB/s RD_MB COMP WR_MB COMP
01:00:00 4 20 25827 7 49676 13 7741 3.33 19634 2.53 0 0   0 0 0 0 0 0
02:00:00 4 7 9204 2 18030 5 2100 4.38 6480 2.78 0 0   0 0 0 0 0 0
03:00:00 4 1 2248 0 4550 1 699 3.21 1154 3.94 0 0   0 0 0 0 0 0
04:00:00 4 0 0 0 69 0 0 24 2.87 0 0   0 0 0 0 0 0
05:00:00 4 0 1696 0 1655 0 550 3.08 540 3.06 0 0   0 0 0 0 0 0
06:00:00 4 9 8645 2 24001 6 3653 2.36 13589 1.76 0 0   0 0 0 0 0 0
07:00:00 4 4 6371 1 10227 2 2283 2.79 3503 2.91 0 0   0 0 0 0 0 0
08:00:00 4 2 5128 1 4950 1 2048 2.50 1985 2.49 0 0   0 0 0 0 0 0
09:00:00 4 3 6270 1 7272 2 2530 2.47 3406 2.13 0 0   0 0 0 0 0 0
The following fields are the most important fields in this report:
•GBS_RTE: This field shows the actual negotiated speed of the FICON Channel.
•RD_MB and WR_MB: The amount of uncompressed data that is transferred by this FICON Channel.
The host adapter activity is summarized per adapter, and as a total of all adapters. This result is also shown in the vNode adapter Throughput Distribution report shown in Example 11-19.
Example 11-19 Extract of the Adapter Throughput Distribution report
1(C) IBM REPORT=H21ADPSU(10210) VNODE ADAPTOR THROUGHPUT DISTRIBUTION RUN ON 18AUG2010 @ 8:04:29
GRID#=CC001 DIST_LIB_ID= 0 VNODE_ID= 0 NODE_SERIAL=CL0ABCDE VE_CODE_LEVEL=008.006.000.0110 UTC NOT CHG
    MB/SEC_RANGE #INTERVALS     PCT      ACCUM%
1 - 50 477 64.4 64.4
51 - 100 191 25.8 90.2
101 - 150 52 7.0 97.2
151 - 200 17 2.2 99.5
201 - 250 1 0.1 99.7
251 - 300 2 0.2 100.0
The provided example is an extract of the report and summarizes the overall host throughput and shows how many one-hour intervals have shown which throughput.
For example, look at the second line of the report data:
•The throughput was 51 - 100 MBps in 191 intervals.
•191 intervals are 25.8% of the entire measurement period.
•In 90.2% of the measurement intervals, the throughput was below 100 MBps.
Cache Partition Activity
This report provides details of Cache Partitions Activity in the TS7700. For a TS7740 and a TS7700D, only one H30TVC1 is provided. For a TS7700T, multiple H30TVCx can be shown. The H30TVC1 for a TS7700T contains the information of the resident partition. If you have defined two Tape Partitions, you also get H30TVC2 and H30TVC3. You can identify the following information for each 15-minute interval:
•CPU and Disk Utilization
•The number of scratch (Fast Ready) mounts, cache hits, cache misses, and Sync mounts
•The value that is defined for PMTHLVL
•The percentage of read, write, and deferred copy throttling, and the throttling reasons
•The capacity and number of logical volumes by preference group (0 or 1) in cache, and an indicator for the residency time
The report also shows information about the Preference Groups. The following fields are the most important fields in this report:
•The ratio between FAST_RDY_MOUNTS, CACHE_HIT_MOUNTS, and CACHE_MISS_MOUNTS. In general, a high number of CACHE_MISSES might mean that additional cache capacity is needed, or cache management policies need to be adjusted. For a TS7700T, you might also reconsider the Tape Partition sizes.
•FAST_RDY_AVG_SECS and CACHE_HIT_ AVG_ SECS need to show only a few seconds. CACHE_MIS_AVG_SECS can list values higher than a few seconds, but higher values (more than 2 - 3 minutes) might indicate a lack of physical tape drives. For more information, see Example 11-20.
Example 11-20 List of Values
03:00:00 16 16 1 4 9 20 20 21 4 2 0 0 6
04:00:00 16 16 1 2 3 19 21 23 0 2 0 0   2
Peer-to-Peer Activity
The Peer-to-Peer Activity report that is shown in Example 11-21 provides various performance metrics of grid activity. This report can be useful for installations working in Deferred copy mode. This report enables, for example, the analysis of subsystem performance during peak grid network activity, such as determining the maximum delay during the batch window.
Example 11-21 VEHSTATS report for Peer-to-Peer Activity
(C) IBM REPORT=H33GRID (10210) HNODE HISTORICAL PEER-TO-PEER ACTIVITY RUN ON 18AUG2010 @ 8:04:29 PAGE 37
GRID#=CC001 DIST_LIB_ID= 1 VNODE_ID= 0 NODE_SERIAL=CL1FEDCB VE_CODE_LEVEL=008.006.000.0110 UTC NOT CHG
25JUN10FR LVOLS MiB AV_DEF AV_RUN MiB_TO CALC V_MNTS MiB_XFR MiB_XFR 1-->0 CALC 1-->2 CALC 1-->3 CALC
TO TO QUEAGE QUEAGE TVCBY MiB/ DONE_BY FR_DL TO_DL TVC_BY MiB/ TVC_BY MiB/ TVC_BY MiB/
RECEIVE RECEIVE ---MINUTES--- COPY SEC OTHR_DL RMT_WR RMT_RD COPY SEC COPY SEC COPY SEC
01:00:00 1 13 1 0 139077 38.6 43 1 346 61355 17.0 746 0.2 156 0.0
02:00:00 6 1518 7 0 150440 41.7 84 462 11410 64536 17.9 4448 1.2 1175 0.3
03:00:00 2 3239 3 0 88799 24.6 38 8 44 57164 15.8 1114 0.3 166 0.0
04:00:00 2 574 4 0 241205 67.0 4 82 29 109850 30.5 1409 0.3 401 0.1
05:00:00 3 1055 2 0 70637 19.6 9 390 136 51464 14.2 2488 0.6 0
06:00:00 16 9432 2 0 187776 52.1 33 1519 491 100580 27.9 2526 0.7 463 0.1
07:00:00 0 0 0 0 86624 24.0 19 63 12649 50139 13.9 6036 1.6 1988 0.5
08:00:00 1 484 0 0 46314 12.8 26 30 12292 23216 6.4 9563 2.6 1971 0.5
For the time of the report, you can identify, in 15-minute increments, the following items:
•The number of logical volumes to be copied (valid only for a multi-cluster grid configuration)
•The amount of data to be copied (in MB)
•The average age of copy jobs on the deferred and immediate copy queue
•The amount of data (in MB) to and from the TVC driven by copy activity
•The amount of data (in MB) copied from other clusters (inbound data) to the cluster on which the report was run
 
Tip: Analyzing the report that is shown in Example 11-21, you see three active clusters with write operations from a host. This might not be a common configuration, but it is an example of a scenario to show the possibility of having three copies of a logical volume in a multi-cluster grid.
The following fields are the most important fields in this report:
•MB_TO_COPY: The amount of data pending a copy function to other clusters (outbound).
•MB_FR: The amount of data (MB) copied from the cluster (inbound data) identified in the column heading. The column heading 1-->2 indicates Cluster 1 is the copy source and Cluster 2 is the target.
•CALC_MB/SEC: This number shows the true throughput that is achieved when replicating data between the clusters that are identified in the column heading.
Summary reports
In addition to daily and monthly summary reports per cluster, VEHSTATS also provides a side-by-side comparison of all clusters for the entire measurement interval. Examine this report for an overall view of the grid, and for significant or unexpected differences between
the clusters.
11.18.4 VEHGRXCL tool overview
VEHGRXCL is a tool that can be downloaded from the IBM Tape Tools and used as the graphical interface for the records that are provided by VEHSTATS. The VEHGRXCL.EXE file contains VEHSTATS_Model.ppt and VEHSTATS_Model.xls. You can use these files to create graphs on cluster activity based on the flat files that are created with VEHSTATS. Detailed instructions about how to include your data in the tool are described in the first worksheet in the VEHSTATS_Model.xls file that is created as part of the installation procedure.
The following steps describe the sequence of actions in general to produce the graphs of your grid environment:
1. Run the BVIRHSTV program to collect the TS7700 BVIR History data for a selected period (suggested 31 days). Run the VEHSTATS program for the period to be analyzed (a maximum of 31 days is used).
2. Select one day during the analysis period to analyze in detail, and run the VEHSTATS hourly report for that day. You can import the hourly data for all days and then select the day later in the process. You also decide which cluster will be reported by importing the hourly data of that cluster.
3. File transfer the two space-separated files from VEHSTATS (one daily and one hourly) to your workstation.
4. Start Microsoft Excel and open this workbook, which must be in the directory C:VEHSTATS.
5. Import the VEHSTATS daily file into the “Daily data” sheet by using a special parsing option.
6. Import the VEHSTATS hourly file into the “Hourly data” sheet, by using a special parsing option. Copy 24 hours of data for your selected day and cluster and paste it into the top section of the “Hourly data” sheet.
7. Open the accompanying VEHSTATS_MODEL.PPT Microsoft PowerPoint presentation and ensure that automatic links are updated.
8. Save the presentation with a new name so as not to modify the original VEHSTATS_MODEL.PPT.
9. Verify that the PowerPoint presentation is correct, or make any corrections necessary.
10. Break the links between the workbook and the presentation.
11. Edit or modify the saved presentation to remove blank or unneeded charts. Save the presentation with the links broken.
The following examples of PowerPoint slides give an impression of the type of information that is provided with the tool. You can easily update these slides and include them in your own capacity management reports.
Figure 11-44 gives an overview of all of the sections included in the PowerPoint presentation.
Figure 11-44 Sample VEHGRXCL: Agenda
Figure 11-45 gives an overview of the reported period.
Figure 11-45 Sample VEHGRXCL: Overview
Figure 11-46 is an example throughput, expressed in MBps.
Figure 11-46 Sample VEHGRXCL: Maximum and average throughput
Figure 11-47 is an example of physical mounts.
Figure 11-47 Sample VEHGRXCL: All physical mounts
11.18.5 VEHAUDIT overview
VEHAUDIT is also based on the BVIR information and a part of the TapeTool suite. It has several independent reports,
Copy Distribution report (CPYDST)
This report shows long copies taken from one cluster to the other clusters in the grid. Family relationships can be reflected as well. This report can be used to identify the RPO, and how long in general the creation of copies takes.
Detail Report (DTLRPT)
The DTLRPT provides an audit functionality, and provides information about which logical volumes have a deviation from the requested copy consistency policy on the specified cluster. In opposition to the BVIRAUD function, it does not compare several clusters to determine if copies are missing. Instead, an audit of a specific cluster is performed, and the output report shows every logical volume that is not in the appropriate state.
The report contains not only missing copies, but also inconsistent data level, data corruption of a logical volume (only detected at read), and existing (but unwanted) copies.
For each logical volume reported, it also reports all of the following information:
•Type of issue
•Constructs,
•Creation, last reference and expire date
•Category
•Size
•Data set name
•Physical volume placement (only TS7740/TS770T)
•information from the tape management system (if input is provided)
MES Error Report (MESERRP)
This report shows a list of MES records of corrupted volumes. A volume is corrupted if the “Read_Error” flag is set.
Multi Copy (MLTCPY)
This report shows if more copies exist than requested from the copy consistency policy perspective.
STALE
This report contains logical volumes that are expired, but not yet deleted. This report should usually be empty.
TOSYNC
If fewer copies exist for a logical volume than requested in the Management Class, this report should be generated. The tool is able to eliminate the scratches. The list produced in this report can then be used as input for copy refresh processing.
11.19 IBM z/OS commands for monitoring
In addition to the previously introduced methods and options for monitoring the TS7700, the following extra points offer further subsystem monitoring.
11.19.1 DISPLAY SMS commands
Several DISPLAY SMS commands exist to display the OAM status, the composite and distributed library, and volume details. Several of these commands (shown in bold) and their responses are listed in Example 11-22, separated by a dashed line.
Example 11-22 DISPLAY SMS command responses
D SMS,OAM
F OAM,D,OAM,L=DENEKA-Z
CBR1100I OAM status: 171
TAPE TOT ONL TOT TOT TOT TOT TOT ONL AVL TOTAL
LIB LIB AL VL VCL ML DRV DRV DRV SCRTCH
2 2 0 0 2 0 528 256 256 12
There are also 3 VTS distributed libraries defined.
CBRUXCUA processing ENABLED.
CBRUXEJC processing ENABLED.
CBRUXENT processing ENABLED.
CBRUXVNL processing ENABLED.
----------------------------------------------------------------------
D SMS,LIB(ALL),DETAIL
F OAM,D,LIB,L=DENEKA-Z
CBR1110I OAM library status: 174
TAPE LIB DEVICE TOT ONL AVL TOTAL EMPTY SCRTCH ON OP
LIBRARY TYP TYPE DRV DRV DRV SLOTS SLOTS VOLS
DTS7720 VDL 3957-VEB 0 0 0 0 0 0 Y Y
HYDRAE VDL 3957-V07 0 0 0 185 133 0 Y Y
HYDRAG VCL GRID 512 256 256 0 0 12 Y Y
HYDRAO VDL 3957-V06 0 0 0 400 340 0 Y Y
HYDV06 VCL 3957-V06 16 0 0 0 0 0 Y Y
----------------------------------------------------------------------
D SMS,LIB(ALL),STATUS
IGD002I 13:47:31 DISPLAY SMS 176
LIBRARY CLASS SYSTEM= 1
DTS7720 TAPE +
HYDRAE TAPE +
HYDRAG TAPE +
HYDRAO TAPE +
HYDV06 TAPE +
***************************** LEGEND ****************************
. THE LIBRARY IS NOT DEFINED TO THE SYSTEM
+ THE LIBRARY IS ONLINE
- THE LIBRARY IS OFFLINE
P THE LIBRARY IS PENDING OFFLINE
----------------------------------------------------------------------
D SMS,LIB(HYDRAG),DETAIL
F OAM,D,LIB,HYDRAG,L=DENEKA-Z
CBR1110I OAM library status: 179
TAPE LIB DEVICE TOT ONL AVL TOTAL EMPTY SCRTCH ON OP
LIBRARY TYP TYPE DRV DRV DRV SLOTS SLOTS VOLS
HYDRAG VCL GRID 512 256 256 0 0 12 Y Y
----------------------------------------------------------------------
MEDIA TYPE SCRATCH COUNT SCRATCH THRESHOLD SCRATCH CATEGORY
MEDIA1 7 0 0011
MEDIA2 5 0 0012
----------------------------------------------------------------------
DISTRIBUTED LIBRARIES: HYDRAE DTS7720
----------------------------------------------------------------------
LIBRARY ID: 00186
OPERATIONAL STATE: AUTOMATED
ERROR CATEGORY SCRATCH COUNT: 0
CORRUPTED TOKEN VOLUME COUNT: 0
----------------------------------------------------------------------
Library supports outboard policy management.
Library supports logical WORM.
----------------------------------------------------------------------
D SMS,LIB(HYDRAE),DETAIL
F OAM,D,LIB,HYDRAE,L=DENEKA-Z
CBR1110I OAM library status: 168
TAPE LIB DEVICE TOT ONL AVL TOTAL EMPTY SCRTCH ON OP
LIBRARY TYP TYPE DRV DRV DRV SLOTS SLOTS VOLS
HYDRAE VDL 3957-V07 0 0 0 185 133 0 Y Y
--------------------------------------------------------------------
COMPOSITE LIBRARY: HYDRAG
--------------------------------------------------------------------
LIBRARY ID: 01052
CACHE PERCENTAGE USED: 0
OPERATIONAL STATE: AUTOMATED
SCRATCH STACKED VOLUME COUNT: 12
PRIVATE STACKED VOLUME COUNT: 5
--------------------------------------------------------------------
Library supports import/export.
Library supports outboard policy management.
Library supports logical WORM.
Convenience I/O station installed.
Convenience I/O station in Output mode.
Bulk input/output not configured.
----------------------------------------------------------------------
D SMS,VOL(A13052)
CBR1180I OAM tape volume status: 143
VOLUME MEDIA STORAGE LIBRARY USE W C SOFTWARE LIBRARY
TYPE GROUP NAME ATR P P ERR STAT CATEGORY
A13052 MEDIA1 SGG00001 HYDRAG P N NOERROR PRIVATE
-------------------------------------------------------------------
RECORDING TECH: 36 TRACK COMPACTION: YES
SPECIAL ATTRIBUTE: NONE ENTER/EJECT DATE: 2014-02-12
CREATION DATE: 2014-02-12 EXPIRATION DATE: 2014-02-13
LAST MOUNTED DATE: 2014-02-12 LAST WRITTEN DATE: 2014-02-12
SHELF LOCATION:
OWNER: DENEKA
LM SG: SGG00001 LM SC: SC00000K LM MC: MNSSN068 LM DC: D100N006
LM CATEGORY: 001F
-------------------------------------------------------------------
Logical volume.
Volume is cache resident.
Valid copy in each distributed library.
 
D SMS,VOL(A13051)
CBR1180I OAM tape volume status: 146
VOLUME MEDIA STORAGE LIBRARY USE W C SOFTWARE LIBRARY
TYPE GROUP NAME ATR P P ERR STAT CATEGORY
A13051 MEDIA1 *SCRTCH* HYDRAG S N N NOERROR SCRMED1
-------------------------------------------------------------------
RECORDING TECH: 36 TRACK COMPACTION: YES
SPECIAL ATTRIBUTE: NONE ENTER/EJECT DATE: 2014-02-12
CREATION DATE: 2014-02-12 EXPIRATION DATE:
LAST MOUNTED DATE: 2014-02-12 LAST WRITTEN DATE: 2014-02-12
SHELF LOCATION:
OWNER:
LM SG: LM SC: LM MC: LM DC:
LM CATEGORY: 0011
-------------------------------------------------------------------
Logical volume.
----------------------------------------------------------------------
For more information, see Chapter 10, “Host Console operations” on page 581 and z/OS DFSMS Object Access Method Planning, Installation, and Storage Administration Guide for Tape Libraries, SC23-6867.
11.19.2 LIBRARY command
The LIBRARY command and the LIBRARY REQUEST command, also known as the Host Console Request function, can be used to check for missing virtual drives or for the status of the grid links. Example 11-23 shows the output of the LIBRARY DD command that you can use to verify whether all virtual drives are available.
Example 11-23 Sample response for LI DD,libname command
15.13.36 CBR1220I Tape drive status: 428
DRIVE DEVICE LIBRARY ON OFFREASON LM ICL ICL MOUNT
NUM TYPE NAME LI OP PT CU AV CATEGRY LOAD VOLUME
1C00 3490 ATL3484F N N Y Y N - --N/A-- -
1C01 3490 ATL3484F Y N N N N A NONE N
1C02 3490 ATL3484F N N Y Y N - --N/A-- -
1C03 3490 ATL3484F N N Y Y N - --N/A-- -
1C04 3490 ATL3484F N N Y Y N - --N/A-- -
1C05 3490 ATL3484F N N Y Y N - --N/A-- -
1C06 3490 ATL3484F N N Y Y N - --N/A-- -
1C07 3490 ATL3484F N N Y Y N - --N/A-- -
1C08 3490 ATL3484F N N Y Y N - --N/A-- -
1C09 3490 ATL3484F N N Y Y N - --N/A-- -
1C0A 3490 ATL3484F N N Y Y N - --N/A-- -
1C0B 3490 ATL3484F N N Y Y N - --N/A-- -
1C0C 3490 ATL3484F N N Y Y N - --N/A-- -
1C0D 3490 ATL3484F N N Y Y N - --N/A-- -
1C0E 3490 ATL3484F N N Y Y N - --N/A-- -
1C0F 3490 ATL3484F N N Y Y N - --N/A-- -
For more information about the LIBRARY command, see Chapter 10, “Host Console operations” on page 581 and z/OS DFSMS Object Access Method Planning, Installation, and Storage Administration Guide for Tape Libraries, SC23-6867.
11.20 What to look for and where
This chapter describes tools, provides considerations, and gives you important information to help you monitor and understand the performance indicators of your TS7700 grid. Table 11-9 summarizes where you can find information and what observations you can make.
Table 11-9 Monitoring summary
Information
Source
Tracking
Reporting interval
Observation
All virtual drives online
LI DD,libname
Display each composite library and each system
Each shift
Report or act on any missing drive
TS7700 health check
TS7700 MI
Display each composite library
Each shift
Report any offline or degraded status
Library online and operational
D SMS, LIB(ALL), DETAIL
Display each composite library and each system
Each shift
Verify availability to systems
Exits enabled
D SMS,OAM
Display each system
Each shift
Report any disabled exits
Virtual scratch volumes
D SMS, LIB(ALL), DETAIL
Display each composite library
Each shift
Report each shift
Physical scratch tapes
D SMS, LIB(ALL), DETAIL
Display each composite library
Each shift
Report each shift
Interventions
D SMS, LIB(ALL), DETAIL
Display each composite library
Each shift
Report or act on any interventions
Grid link status
LI REQ,libname, STATUS, GRIDLINK
Display each composite library
 
Each shift
Report any errors or elevated Retransmit%
Number of volumes on the deferred copy queue
TS7700 MI → Logical Volumes → Incoming Copy Queue
Display for each cluster in the grid
Each shift
Report and watch for gradual or sudden increases
Copy queue depths
TS7700 MI
Display for each system
Each shift
Report if queue depth is higher than usual
Virtual mounts per day
VEHSTATS
Rolling weekly trend
Daily
Increase over time. Indicates increased workload
MB transferred per day
VEHSTATS
Rolling weekly trend
Daily
 
Increase over time. Indicates increased workload
Virtual volumes managed
VEHSTATS
Rolling weekly trend
Daily
Capacity planning: maximum one million per grid
MB stored
VEHSTATS
Rolling weekly trend
Daily
 
Capacity planning and general awareness
Back-end drive utilization
VEHSTATS
Rolling weekly trend
Daily
 
Check for periods of 100%
Daily throttle indicators
VEHSTATS
Rolling weekly trend
Daily
 
Key performance indicator
Average virtual mount time
VEHSTATS
Rolling weekly trend
Daily
 
Key performance indicator
Cache hit percentage
VEHSTATS
Rolling weekly trend
Daily
 
Key performance indicator
Physical
scratch count
VEHSTATS
Rolling weekly trend
Daily
 
Capacity planning and general awareness
Available slot count
D SMS, LIB(ALL), DETAIL
Rolling weekly trend
Daily
 
Capacity planning and general awareness
Available virtual scratch volumes
D SMS, LIB(ALL), DETAIL
Rolling weekly trend
Daily
 
Drive insert
Data distribution
BVIRPOOL job
Watch for healthy distribution
Weekly
Use for reclaim tuning
Times in cache
VEHSTATS
Watch for healthy distribution
Weekly
Preference group tuning indicator
Most checks that you need to make in each shift ensure that the TS7700 environment is operating as expected. The checks that are made daily or weekly are intended for tuning and longer-term trend analysis. The information in this table is intended as a basis for monitoring. You can tailor this information to best fit your needs.
11.21 Virtual Device Allocation in z/OS with JES2
z/OS (JES2 only) allocation characteristics in general are described and how allocation algorithms are being influenced by z/OS allocation parameter settings EQUAL and BYDEVICES, by the TS7700 Copy Consistency Points, by Override Settings, and by the Allocation Assistance functions is shown. Carefully plan for your device allocation requirements because improper use of the parameters, functions, and settings can have unpredictable results.
Various scenarios are presented, showing the influence of the algorithms involved in virtual device allocation. A configuration with two 3-cluster grid configurations, named GRID1 and GRID2, is used. Each grid has a TS7700D (Cluster 0) and a TS7740 (Cluster 1) at the primary Production Site, and a TS7740 (Cluster 2) at the Disaster Site. The TS7700D Cluster 0 in the Production Site can be considered a deep cache for the TS7740 Cluster 1 in the scenarios that are described next.
Figure 11-48 gives a general overview of the configuration.
Figure 11-48 Grid configuration overview
In Figure 11-48, the host in the Production Site has direct access to the local clusters in the Production Site, and has access over the extended FICON fabric to the remote clusters in the Disaster Site. The extended FICON fabric can include dense wavelength division multiplexing (DWDM) connectivity, or can use FICON tape acceleration technology over IP. Assume that connections to the remote clusters have a limited capacity bandwidth.
Furthermore, there is a storage management subsystem (SMS) Storage Group (SG) per grid.
The groups are defined in the SMS SG routine as GRID1 and GRID2. SMS equally manages SGs. The order in the definition statement does not influence the allocations.
The following scenarios are described. Each scenario adds functions to the previous scenario so that you can better understand the effects of the added functions:
•EQUAL allocation: Describes the allocation characteristics of the default load-balancing algorithm (EQUAL) and its behavior across the sample TS7700 configuration with two grids. See 11.21.1, “EQUAL allocation” on page 721.
•BYDEVICES allocation: Adds the new BYDEVICES algorithm to the configuration. It explains how this algorithm can be activated and the differences from the default EQUAL algorithm. See 11.21.2, “BYDEVICES allocation” on page 724.
•Allocation and Copy Consistency Point setting: Adds information about the effect of the Copy Consistency Point on the cache data placement. The various TS7700 Virtualization override settings influence this data placement. See 11.21.3, “Allocation and Copy Consistency Point setting” on page 726.
•Allocation and device allocation assistance (DAA): DAA is activated, and the effects on allocation are described. The unavailability of cluster and device information influences the allocation when DAA is enabled. DAA is enabled, by default. See 11.21.4, “Allocation and device allocation assistance” on page 728.
•Allocation and scratch allocation assistance (SAA): SAA is activated, and its effects are described. A sample workload in this scenario is presented to clarify SAA. The advantages of SAA and the consequences of the unavailability of clusters and devices are explained. SAA is disabled, by default. See 11.21.5, “Allocation and scratch allocation assistance” on page 731.
11.21.1 EQUAL allocation
For non-specific (scratch) allocations, by default, MVS device allocation will first randomize across all eligible libraries and then, after a library is selected, will randomize on the eligible devices within that library. In terms of the TS7700, library refers to a composite library because the MVS allocation has no knowledge of the underlying clusters (distributed libraries). The default algorithm (EQUAL) works well if the libraries under consideration have an equal number of online devices.
For example, if two libraries are eligible for a scratch allocation and each library has 128 devices, over time, each library will receive approximately half of the scratch allocations. If one of the libraries has 128 devices and the other library has 256 devices, each of the libraries still receives approximately half of the scratch allocations. The allocations are independent of the number of online devices in the libraries.
 
Remember: With EQUAL allocation, the scratch allocations are randomized across the libraries. EQUAL allocation is not influenced by the number of online devices in the libraries.
In this first scenario, both DAA and SAA are assumed to be disabled. With the TS7700, you can control both assistance functions with the LIBRARY REQUEST command. DAA is ENABLED by default and can be DISABLED with the command. SAA is DISABLED by default and can be ENABLED with the command. Furthermore, none of the TS7700 override settings are used.
Assuming that the MC for the logical volumes has a Copy Consistency Point of [R,R,R] in all clusters and that the number of available virtual drives are the same in all clusters, the distribution of the allocation across the two grids (composite libraries) is evenly spread. The multi-cluster grids are running in BALANCED mode, so there is no preference of one cluster above another cluster.
With the default algorithm EQUAL, the distribution of allocations across the clusters (in a multiple cluster grid) depends on the order in which the library port IDs were initialized during IPL (or input/output definition file (IODF) activate). The distribution of allocations across the clusters also depends on whether the library port IDs in the list (returned by the DEVSERV QLIB,composite-library-id command) randomly represent each of the clusters.
Alternatively, the distribution depends on if the library port IDs in the list tend to favor the library port IDs in one cluster first, followed by the next cluster, and so on. The order in which the library port IDs are initialized and appear in this DEVSERV list can vary across IPLs or IODF activates, and can influence the randomness of the allocations across the clusters.
So with the default algorithm EQUAL, there might be times when device randomization within the selected library (composite library) appears unbalanced across clusters in a TS7700 that have online devices. As the number of eligible library port IDs increases, the likelihood of this imbalance occurring also increases. If this imbalance affects the overall throughput rate of the library, consider enabling the BYDEVICES algorithm described in 11.21.2, “BYDEVICES allocation” on page 724.
 
Remember: Exceptions to this can also be caused by z/OS JCL backward referencing specifications (UNIT=REF and UNIT=AFF).
With z/OS V1R11 and later, and z/OS V1R8 through V1R10 with APAR OA26414 installed, it is possible to change the selection algorithm to BYDEVICES. The algorithm EQUAL, which is the default algorithm that is used by z/OS, can work well if the libraries (composite libraries) under consideration have an equal number of online devices and the previous cluster behavior is understood.
The non-specific (scratch) allocation distribution is shown in Figure 11-49.
Figure 11-49 ALLOC EQUAL scratch allocations
For specific allocations (DAA DISABLED in this scenario), it is first determined which of the composite libraries, GRID1 or GRID2, has the requested logical volume. That grid is selected and the allocation can go to any of the clusters in the grid. If it is assumed that the logical volumes were created with the EQUAL allocation setting (the default), it can be expected that specific device allocation to these volumes will be distributed equally among the two grids.
However, how well the allocations are spread across the clusters depends on the order in which the library port IDs were initialized, and whether this order was randomized across
the clusters.
In a TS7740 multi-cluster grid configuration, only the original copy of the volume stays in cache, normally in the mounting cluster’s TVC for a Copy Consistency Point setting of [R,R,R]. The copies of the logical volume in the other clusters are managed as a TVC Preference Level 0 (PG0 - remove from cache first) unless an SC specifies Preference Level 1 (PG1 - stay in cache) for these volumes.
A number of possibilities can influence the cache placement:
•You can define an SC for the volume with Preference Level 0 (PG0). The logical volume does not stay in the I/O TVC cluster.
•You can set the CACHE COPYFSC option, with a LIBRARY REQUEST,GRID[1]/[2],SETTING,CACHE,COPYFSC,ENABLE command. When the ENABLE keyword is specified, the logical volumes that are copied into the cache from a peer TS7700 cluster are managed by using the actions that are defined for the SC construct associated with the volume as defined at the TS7740 cluster that is receiving the copy.
Therefore, a copy of the logical volume also stays in cache in each non-I/O TVC cluster where an SC is defined as Preference Level 1 (PG1). However, because the TS7700D is used as a deep cache, there are no obvious reasons to do so.
In the hybrid multi-cluster grid configuration that is used in the example, there are two cache allocation schemes, depending on the I/O TVC cluster selected when creating the logical volume. Assume an SC setting of Preference Level 1 (PG1) in the TS7740 Cluster 1 and Cluster 2.
•If the mounting cluster for the non-specific request is the TS7700D Cluster 0, only the copy in that cluster stays. The copies in the TS7740 Cluster 1 and Cluster 2 will be managed as Preference Level 0 (PG0) and will be removed from cache after placement of the logical volume on a stacked physical volume. If a later specific request for that volume is directed to a virtual device in one of the TS7740s, a cross-cluster mount from Cluster 1 or Cluster 2 occurs to Cluster 0’s cache.
•If the mounting cluster for the non-specific request is the TS7740 Cluster 1 or Cluster 2, not only the copy in that cluster stays, but also the copy in the TS7700D Cluster 0. Only the copy in the other TS7740 cluster will be managed as Preference Level 0 (PG0) and will be removed from cache after placement of the logical volume on a stacked physical volume.
Cache preferencing is not valid for the TS7700D cluster. A later specific request for that logical volume creates only a cross-cluster mount if the mount point is the vNode of the TS7740 cluster that is not used at data creation of that volume.
With the EQUAL allocation algorithm that is used for specific mount requests, there are always cross-cluster mounts when the cluster where the device is allocated is not the cluster where the data is. Cache placement can limit the number of cross-cluster mounts but cannot avoid them. Cross-cluster mounts over the extended fabric are likely not acceptable, so vary the devices of Cluster 2 offline.
11.21.2 BYDEVICES allocation
The alternative algorithm BYDEVICES randomizes scratch allocations across all devices. For example, if two libraries are eligible for a scratch allocation and each library has 128 devices, over time each library will receive approximately half of the scratch allocations, similar to the EQUAL algorithm. Again, in terms of the TS7700, “library” refers to a composite library because MVS allocation has no knowledge of the underlying clusters (distributed libraries).
However, if one of the libraries has 128 devices and the other library has 256 devices, over time, the library that has 128 devices will receive 1/3 of the scratch allocations and the library that has 256 devices will receive approximately 2/3 of the scratch allocations. This is different compared to the default algorithm EQUAL, which does not take the number of online devices in a library into consideration.
 
Clarification: With BYDEVICES, the scratch allocation randomizes across all devices in the libraries, and is influenced by the number of online devices.
With z/OS V1R11 and later, and z/OS V1R8 through V1R10 with APAR OA26414 installed, it is possible to influence the selection algorithm. The BYDEVICES algorithm can be enabled through the ALLOCxx PARMLIB member by using the SYSTEM TAPELIB_PREF(BYDEVICES) parameter, or it can be enabled dynamically through the SETALLOC operator command by entering SETALLOC SYSTEM,TAPELIB_PREF=BYDEVICES.
The alternative BYDEVICES algorithm can be replaced by the default EQUAL algorithm by specifying EQUAL through the SETALLOC command or the ALLOCxx PARMLIB member in a similar manner. Before enabling the new load balancing support, care must be taken to ensure that the wanted results are achieved. This is important if the libraries are being shared across multiple systems and the systems are at different levels of support, or if manual actions have already been taken to account for the behavior of algorithms that were used in
previous releases.
 
Consideration: The SETALLOC operator command support is available only in z/OS V1R11 or later releases. In earlier z/OS releases, BYDEVICES must be enabled through the ALLOCxx PARMLIB member.
Assume that GRID1 has a total of 60 virtual devices online and GRID2 has 40 virtual devices online. For each grid, the distribution of online virtual drives is 50% for Cluster 0, 25% for Cluster 1, and 25% for Cluster 2.
The expected distribution of the scratch allocations is as shown in Figure 11-50.
Figure 11-50 ALLOC BYDEVICES scratch allocations
As stated in 11.21.1, “EQUAL allocation” on page 721, DAA is ENABLED by default and was DISABLED by using the LIBRARY REQUEST command. Furthermore, none of the TS7700 override settings are activated.
For specific allocations (DAA DISABLED in this scenario), it is first determined which one of the composite libraries, GRID1 or GRID2, has the requested logical volume. That grid is selected, and the allocations can go to any cluster in the grid and are proportionately distributed based on the number of online devices in each cluster. The logical volume cache placement possibilities and the two allocation schemes, both described in 11.21.1, “EQUAL allocation” on page 721, are also applicable for the BYDEVICES allocation.
With the BYDEVICES allocation algorithm that is used for specific mount requests, there are always cross-cluster mounts when the cluster where the device is allocated is not the cluster where the data is. Cache placement can limit the number of cross-cluster mounts but cannot avoid them. Cross-cluster mounts over the extended fabric are likely not acceptable, so vary the devices of Cluster 2 offline.
11.21.3 Allocation and Copy Consistency Point setting
By defining the Copy Consistency Point, you control if and how a volume will be placed in a determined cluster of the grid. If you plan to use the TS7700D Cluster 0 as a deep cache, you probably prefer to define the MC Construct as [R,D,D]. By defining this, Cluster 0 is the primary placeholder of the data. At job completion time, only this cluster has a valid copy of the data. The other cluster creates a deferred copy of that logical volume afterward.
For more information about Copy Consistency Points, see the IBM TS7700 Best Practices - Synchronous Mode Copy and IBM TS7700 Best Practices - Copy Consistency Point white papers:
It is further assumed that the allocation characteristics apply as described in 11.21.2, “BYDEVICES allocation” on page 724. Both DAA and SAA are DISABLED in this scenario, too, and none of the TS7700 override settings are used.
For non-specific (scratch) allocations, the BYDEVICES algorithm randomizes across all devices, resulting in allocations on all three clusters of each grid. Subsequently, i/O TVC selection assigns the TVC of Cluster 0 as the I/O TVC due to the Copy Consistency Point setting. Many factors can influence this selection, as explained in 2.2.2, “Tape Volume Cache” on page 32.
Normally, the cluster with a Copy Consistency Point of R(un) gets preference over other clusters. As a consequence, the TVC of Cluster 0 is selected as the I/O TVC and cross-cluster mounts are issued from both Cluster 1 and Cluster 2.
By activating the override setting “Prefer Local Cache for Fast Ready Mount Requests” in both clusters in the Disaster Site, cross-cluster mounts are avoided but the copy to Cluster 0 is made before the job ends, caused by the R(un) Copy Consistency Point setting for this cluster. Therefore, by further defining a family for the Production Site clusters, Cluster 1 retrieves its copy from Cluster 0 in the Production Site location, avoiding using the remote links between the locations.
The method to prevent device allocations at the Disaster Site, implemented mostly today, is just varying offline all the remote virtual devices. The disadvantage is that in losing a cluster in the Production Site, an operator action is required to vary online manually the virtual devices of Cluster 2 of the grid with the failing cluster. With the TS7700 R2.0, an alternative solution is using scratch allocation assistance (SAA), which is described in 11.21.5, “Allocation and scratch allocation assistance” on page 731.
For specific allocations, the algorithm that is described in 11.21.2, “BYDEVICES allocation” on page 724 applies when DAA is disabled. It is first determined which of the composite libraries, GRID1 or GRID2, has the requested logical volume. Subsequently, that grid is selected and the allocation over the clusters is randomized. It can be assumed that, if the requested logical volumes were earlier created with the BYDEVICES allocation scenario, these logical volumes are spread over the two grids. The allocation distribution within the grid over the three clusters is determined by the number of the online devices in each of the clusters.
Cluster 0 is likely to have a valid copy of the logical volume in the cache due to the Copy Consistency Point setting of [R,D,D]. If the vNodes of Cluster 1 and Cluster 2 are selected as mount points, it results in cross-cluster mounts. It might happen that this volume has been removed by a policy in place for TS7700D Cluster 0, resulting in the mount point TVC as the
I/O TVC.
In the TS7700, activating the Force Local TVC to have a copy of the data override first results in a recall of the virtual volume from a stacked volume. If there is no valid copy in the cluster or if it fails, a copy is retrieved from one of the other clusters before the mount completes. Activating the Prefer Local Cache for non-Fast Ready Mount Requests override setting recalls a logical volume from tape instead of using the grid links for retrieving the data of the logical volume from Cluster 0. This might result in longer mount times.
With the TS7700, an alternative solution can be considered by using device allocation assistance (DAA) that is described in 11.21.4, “Allocation and device allocation assistance” on page 728. DAA is enabled by default.
Figure 11-51 shows the allocation results of specific and non-specific allocations when the devices of the remote clusters in the Disaster Site are not online. Allocation BYDEVICES is used. GRID1 has a total of 60 devices online and GRID2 has 40 devices online. For each grid, the distribution of online devices is 75% for Cluster 0 and 25% for Cluster 1.
Cross-cluster mounts might apply for the specific allocations in Cluster 1 because it is likely that only the TS7700D Cluster 0 will have a valid copy in cache. The red arrows show the data flow as result of these specific allocations.
Figure 11-51 Allocation and Copy Consistency Points set at R,D,D
11.21.4 Allocation and device allocation assistance
Device allocation assistance (DAA) allows the host to query the TS7700 to determine which clusters must be preferred for a private (specific) mount request before the actual mount is requested. DAA returns to the host a ranked list of clusters (the preferred cluster is listed first) where the mount must be run. If DAA is enabled, it is for the composite library, and it is used by all z/OS JES2 or JES3 LPARs having the proper level of supporting software (z/OS V2R1 or above).
The selection algorithm orders the clusters first by those having the volume already in cache, then by those having a valid copy on tape, and then by those without a valid copy. Later, host processing attempts to allocate a device from the first cluster that is returned in the list.
If an online device is not available within that cluster, it will move to the next cluster in the list and try again until a device is chosen. This enables the host to direct the mount request to the cluster that will result in the fastest mount, typically the cluster that has the logical volume resident in cache.
If the mount is directed to a cluster without a valid copy, a cross-cluster mount results. Thus, in special cases, even if DAA is enabled, cross-cluster mounts and recalls can still occur.
For JES2, if the default allocation algorithm EQUAL is used, it supports an ordered list for the first seven library port IDs returned in the list. After that, if an eligible device is not found, all of the remaining library port IDs are considered equal. The alternative allocation algorithm BYDEVICES removes the ordered library port ID limitation.
With the TS7700, install the additional APAR OA30718 before enabling the new BYDEVICES algorithm. Without this APAR, the ordered library port ID list might not be acknowledged correctly, causing specific allocations to appear randomized.
In the scenario that is described in 11.21.3, “Allocation and Copy Consistency Point setting” on page 726, if you enable DAA (this is the default) by entering the command LIBRARY REQUEST,GRID[1]/[2],SETTING,DEVALLOC,PRIVATE,ENABLE, it influences the specific requests in the following manner. The Copy Consistency Point is defined as [R,D,D]. It is assumed that there are no mount points in Cluster 2.
It is further assumed that the data is not in the cache of the TS7740 Cluster 1 anymore because this data is managed as TVC Preference Level 0 (PG0), by default. It is first determined which of the composite libraries, GRID1 or GRID2, has the requested logical volume. Subsequently, that grid is selected and the allocation over the clusters is determined by DAA. The result is that all allocations select the TS7700D Cluster 0 as the preferred cluster.
You can influence the placement in cache by setting the CACHE COPYFSC option with the LIBRARY REQUEST,GRID[1]/[2],SETTING,CACHE,COPYFSC,ENABLE command. When the ENABLE keyword is specified, the logical volumes that are copied into the cache from a peer TS7700 cluster are managed by using the actions that are defined for the SC construct associated with the volume as defined at the TS7740 cluster receiving the copy.
Therefore, a copy of the logical volume also stays in cache in each non-I/O TVC cluster where an SC is defined as Preference Level 1 (PG1). However, because the TS7700D is used as a deep cache, there are no obvious reasons to do so.
There are two major reasons why Cluster 0 might not be selected:
•No online devices are available in Cluster 0, but are in Cluster 1.
•The defined removal policies in the TS7700D caused Cluster 0 to not have a valid copy of the logical volume anymore.
In both situations, DAA selects the TS7740 Cluster 1 as the preferred cluster:
•When the TS7740 Cluster 1 is selected due to lack of online virtual devices on Cluster 0, cross-cluster mounts might happen unless the TS7700 override settings, as described in 11.21.3, “Allocation and Copy Consistency Point setting” on page 726, are preventing this from happening.
•When the TS7740 Cluster 1 is selected because the logical volume is not in the TS7700D Cluster 0 cache anymore, its cache is selected for the I/O TVS and because the Copy Consistency Point setting is [R,D,D], a copy to the TS7700D Cluster 0 is made as part of successful RUN processing.
Even when DAA is enabled, there might be specific mounts for which the device affinity call is not made. For example, DFSMShsm goes to allocation when appending a volume, requiring that a scratch volume be mounted. Then, when a device is allocated and a volume is to be mounted, it selects from the list of HSM-owned volumes. In this case, because the allocation started as a scratch request, the device affinity is not made for this specific mount.
The MARKFULL option can be specified in DFSMShsm to mark migration and backup tapes that are partially filled during tape output processing as full. This enforces a scratch tape to be selected the next time that the same function begins.
Figure 11-52 shows the allocation result of specific allocations. The devices of the remote clusters in the Disaster Site are not online. GRID1 has in total 60% of specific logical volumes and GRID2 has 40% of the specific logical volumes. This was the result of earlier BYDEVICES allocations when the logical volumes were created.
The expected distribution of the specific allocations is as shown. Cross-cluster mounts might apply in situations where DAA selects the vNode of Cluster 1 as the mount point. The red arrows show the data flow for both the creation of the copy of the data for scratch allocations and for specific allocations.
Figure 11-52 Allocation and DAA
With DAA, you can vary the devices in the Disaster Site Cluster 2 online without changing the allocation preference for the TS7700D cache if the logical volumes exist in this cluster and this cluster is available. If these conditions are not met, DAA manages local Cluster 1 and remote Cluster 2 as equal and cross-cluster mounts over the extended fabric are issued in Cluster 2.
A new copy of the logical volume is created due to the MC setting [R] for Cluster 0. This is likely not an acceptable scenario and so, even with DAA ENABLED, vary the devices of Cluster 2 offline.
If you plan to have an alternative MC setup for the Disaster Site (perhaps for the Disaster Test LPARs), you must carefully plan the MC settings, the device ranges that must be online, and whether DAA is enabled. You will probably read production data and create test data by using a separate category code.
If you do not want the grid links overloaded with test data, vary the devices of Cluster 0 and Cluster 1 offline on the disaster recovery (DR) host only and activate the TS7700 Override Setting “Force Local TVC” to have a copy of the data. A specific volume request enforces a mount in Cluster 2 even if there is a copy in the deep cache of the TS7700D Cluster 0.
11.21.5 Allocation and scratch allocation assistance
The scratch allocation assistance (SAA) function can be used when there is a need for a method to have z/OS allocate to specific clusters (candidate clusters) for a workload. For example, DFSMShsm Migration Level 2 (ML2) migration can be directed to a TS7700D cluster with its deep cache, and the archive workload needs to be directed to a TS7740 cluster within the same grid configuration.
SAA is an extension of the DAA function for scratch mount requests. SAA filters the list of clusters in a grid to return to the host a smaller list of candidate clusters that are designated as scratch mount candidates. By identifying a subset of clusters in the grid as sole candidates for scratch mounts, SAA optimizes scratch mounts to a TS7700 grid.
When a composite library supports/enables the SAA function, the host sends an SAA handshake to all SAA-enabled composite libraries and provide the MC that will be used for the upcoming scratch mount. A cluster is designated as a candidate for scratch mounts by using the Scratch Mount Candidate option on the MC construct, which is accessible from the TS7700 MI, as shown in Figure 11-53. By default, all clusters are considered candidates. Also, if SAA is enabled and no SAA candidate is selected, all clusters are considered as candidates.
Figure 11-53 Scratch Mount Candidate definition
The targeted composite library uses the provided MC definition and the availability of the clusters within the same composite library to filter down to a single list of candidate clusters. Clusters that are unavailable or in service are excluded from the list. If the resulting list has zero clusters present, the function then views all clusters as candidates.
In addition, if the filtered list returns clusters that have no devices that are configured within z/OS, all clusters in the grid become candidates. The candidate list is not ordered, meaning that all candidate clusters are viewed as equals and all clusters that are excluded from the list are not candidates.
Because this function introduces system burden into the z/OS scratch mount path, a new LIBRARY REQUEST option is introduced to globally enable or disable the function across the entire multi-cluster grid. SAA is disabled, by default. When this option is enabled, the z/OS JES software obtains the candidate list of mount clusters from a given composite library.
Use the LIBRARY REQUEST,GRID[1]/[2],SETTING,DEVALLOC,SCRATCH,ENABLE command to enable SAA. All clusters in the multi-cluster grid must be at R2.0 level before SAA is operational. A supporting z/OS APAR OA32957 is required to use SAA in a JES2 environment of z/OS. For JES3, the minimum supported release is z/OS R2.1. Any z/OS environment with earlier code can exist, but it continues to function in the traditional way regarding scratch allocations.
Assume that there are two main workloads. The application workload consists of logical volumes that are created and then retrieved on a regular, daily, weekly, or monthly basis. This workload can best be placed in the TS7700D deep cache. The backup workload is normally never retrieved and can best be placed directly in the TS7740 Cluster 1. SAA helps direct the mount point to the most efficient cluster for the workload:
•The application workload can best be set up in the following manner. In the MC construct, the MC is defined with a Copy Consistency Point of [R,D,D]. Cluster 0 is selected in all clusters as Scratch Mount Candidate. In Cluster 1, the SC can best be set as TVC Preference Level 1. This is advised because in cases where Cluster 0 is not available or no online devices are available in that cluster, Cluster 1 can be activated as the mount point. Cluster 2 can set Preference Level 0.
You can control the placement in cache per cluster by setting the SETTING CACHE COPYFSC option. When the ENABLE keyword is specified, the logical volumes that are copied into the cache from a peer TS7700 cluster are managed by using the actions that are defined for the SC construct associated with the volume as defined at the TS7740 cluster receiving the copy. The SC in Cluster 0 needs to have a Volume Copy Retention Group of Prefer Keep. Logical volumes can be removed from the TS7700D deep cache if more space is needed.
•The Backup workload can best be set up in the following manner. In the MC construct, the MC is defined with a Copy Consistency Point of [D,R,D] or [N,R,D]. Cluster 1 is selected in all clusters as Scratch Mount Candidate. In Cluster 1 and Cluster 2, the SC can best be set as TVC Preference Level 0. There is no need to keep the data in cache.
The SC in Cluster 0 can have a Volume Retention Group of Prefer Remove. If Cluster 0 is activated as mount point because of the unavailability of Cluster 1 or because there are no online devices in that cluster, the logical volumes with this MC can be removed first when cache removal policies in the TS7700D require the removal of volumes from cache.
With these definitions, the scratch allocations for the application workload are directed to TS7700D Cluster 0 and the scratch allocations for the Backup workload are directed to TS7740 Cluster 1. The devices of the remote clusters in the Disaster Site are not online. Allocation “BYDEVICES” is used. GRID1 has in total 60 devices online and GRID2 has 40 devices online. For each grid, the distribution of online devices is now not determined within the grid by the number of online devices, as in the scenario BYDEVICES, but by the SAA setting of the MC.
The expected distribution of the scratch allocations is shown in Figure 11-54.
Figure 11-54 Allocation and SAA
Clusters not included in the list are never used for scratch mounts unless those clusters are the only clusters that are known to be available and configured to the host. If all candidate clusters have either all their devices varied offline to the host or have too few devices varied online, z/OS will not revert to devices within non-candidate clusters. Instead, the host goes into allocation recovery. In allocation recovery, the existing z/OS allocation options for device allocation recovery (WTOR | WAITHOLD | WAITNOH | CANCEL) are used.
Any time that a service outage of candidate clusters is expected, the SAA function needs to be disabled during the entire outage by using the LIBRARY REQUEST command. If left enabled, the devices that are varied offline can result in zero candidate devices, causing z/OS to enter the allocation recovery mode. After the cluster or clusters are again available and their devices are varied back online to the host, SAA can be enabled again.
If you vary offline too many devices within the candidate cluster list, z/OS might have too few devices to contain all concurrent scratch allocations. When many devices are taken offline, first disable SAA by using the LIBRARY REQUEST command and then re-enable SAA after they have been varied back on.
If you plan to have an alternative MC setup for the Disaster Site (perhaps for the Disaster Test LPARs), carefully plan the MC settings, the device ranges that need to be online, and whether SAA will be used. Read production data and create test data that uses a separate category code. If you use the same MC as used in the Production LPAR and define in Cluster 2 the MC with SAA for Cluster 2 and not for Cluster 0 or 1 (as determined by the type of workload), Cluster 2 might be selected for allocations in the Production LPARs.
SAA randomly selects one of the clusters for determining the scratch mount candidate clusters in the MC constructs. Therefore, the devices in Cluster 2 must not be made available to the Production LPARs and the devices in the clusters in the Production Site must not be made available in the Disaster Site.
Furthermore, the Copy Consistency Point for the MCs in the Disaster Site can be defined as [D,D,R] or even [N,N,R] if it is test only. If it is kept equal with the setting in the Production Site, with an [R] for Cluster 0 or Cluster 1, cross-cluster mount might occur. If you do not want the grid links overloaded with test data, update the Copy Consistency Point setting or use the TS7700 Virtualization override setting “Prefer Local Cache for Fast Ready Mount Requests” in Cluster 2 in the Disaster Site.
Cross-cluster mounts are avoided, but the copy to Cluster 0 or 1 is still made before the job ends, caused by the Production R(un) Copy Consistency Point setting for these clusters. By further defining a family for the Production Site clusters, the clusters source their copies from the other clusters in the Production Site location, optimizing the usage of the remote links between the locations.
Be aware that SAA is only influencing the mount processing. If you have multiple clusters defined as SAA, but the cluster not available is defined as the only TVC cluster (for example, [R,N,N] or [N,N,R]), then the job is not able to run. The mount will be processed, but the job will hang because no TVC can be selected.
 
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.223.33.157