IBM Spectrum Virtualize and IBM Storwize performance monitoring
This chapter provides a brief overview of the performance monitoring capabilities of the IBM Spectrum Virtualize and IBM Storwize storage systems.
However, it is beyond the scope of this book to provide an in-depth understanding of performance statistics, or explain how to interpret them. For a more comprehensive look at the performance of these systems, see IBM System Storage SAN Volume Controller and Storwize V7000 Best Practices and Performance Guidelines, SG24-7521.
For IBM storage systems, the official IBM tool for the collection of performance statistics and to supply performance reporting is IBM Spectrum Control (formerly IBM Tivoli Storage Productivity Center). For more information, see 8.3, “Performance data collection with IBM tools” on page 166.
This chapter describes the following topics:
8.1 Manually gathering performance statistics
The IBM Spectrum Virtualize or Storwize storage system is constantly collecting performance statistics. The lssystem command shows the statistics_status. The default statistics frequency is 5 minutes, which you can adjust by using the startstats -interval <minutes> command.
The statistics data is collected in XML files, with a new file created at the end of each sampling period. Each node of a SAN Volume Controller system and each canister in a Storwize system keeps the most recent 16 files of each type. When the 17th file is created, the oldest file is overwritten.
This design provides statistics for the most recent 80-minute period if the default 5-minute sampling interval is used. You can define the sampling interval by running the startstats -interval <minutes> command to collect statistics at different intervals. The system supports user-defined sampling intervals of 1 - 60 minutes. Running the startstats command resets the statistics timer and gives it a new interval at which to sample.
8.1.1 Statistics file naming
The files that are generated are written to the /dumps/iostats/ directory. The file name is in the following format:
•Nm_stats_<node_serial_number>_<date>_<time> for managed disk (MDisk) statistics
•Nv_stats_<node_serial_number>_<date>_<time> for virtual disk (VDisk) statistics
•Nn_stats_<node_serial_number>_<date>_<time> for node (or canister) statistics
•Nd_stats_<node_serial_number>_<date>_<time> for disk drive statistics
The <node_serial_number> is of the node or canister on which the statistics were collected. The date is in the format <yymmdd> and the time is in the format <hhmmss>. The following example shows an MDisk statistics file name:
Nm_stats_7836640-2_140901_164012
The lsdumps -prefix /dumps/iostats command lists the statistics file names, as shown in Example 8-1. The output is truncated and shows only part of the available statistics.
Example 8-1 The lsdump command output
IBM_IBM Storwize:Cluster_9.113.57.226:superuser>lsdumps -prefix /dumps/iostats
id filename
0 Nd_stats_03C4003-2_150821_114831
1 Nm_stats_03C4003-2_150821_114831
2 Nn_stats_03C4003-2_150821_114831
3 Nv_stats_03C4003-2_150821_114831
4 Nd_stats_03C4003-2_150821_114931
5 Nv_stats_03C4003-2_150821_114931
6 Nn_stats_03C4003-2_150821_114931
7 Nm_stats_03C4003-2_150821_114931
8 Nn_stats_03C4003-2_150821_115031
9 Nv_stats_03C4003-2_150821_115031
10 Nd_stats_03C4003-2_150821_115031
To retrieve all the statistics files of a system, copy the files from the non-configuration nodes onto the configuration node by using the following command:
cpdumps -prefix /dumps/iostats <non_config node id>
Then, download the performance statistics files from the configuration node to a local drive on your workstation by using Secure Copy Protocol (SCP). On a Windows workstation, you can use the pscp.exe command (included with PuTTY), as shown in the following example:
C:Program FilesPuTTY>pscp -unsafe -load ITSO_IBM [email protected]:/dumps/iostats/* c:statsfiles
Use the -load parameter to specify the session that is defined in PuTTY. Specify the -unsafe parameter when you use wildcards.
If you do not use IBM Spectrum Control (or third-party software), you must retrieve and parse these XML files to analyze the long-term statistics. The counters on the files are posted as absolute values. Therefore, the application that processes the performance statistics must compare two samples to calculate the differences from the two files. For the detailed description of the XML file contents, see IBM Knowledge Center.
8.2 Real-time performance monitoring
Real-time performance statistics provide short-term status information for the IBM Storwize storage system. The statistics are shown as graphs in the management GUI or can be viewed from the CLI. With system level statistics, you can quickly view the processor use and the bandwidth of volumes, interfaces, and MDisks. Each graph displays the current bandwidth in either megabytes per second (MBps) or I/O operations per second (IOPS) and a view of bandwidth over time.
Each node collects various performance statistics, mostly at 5-second intervals, and the statistics that are available from the configuration node in a clustered environment. This information can help you determine the performance effect of a specific node. As with system statistics, node statistics help you to evaluate whether the node is operating within normal performance metrics.
Real-time performance monitoring gathers the following system-level performance statistics:
•Central processing unit (CPU) uses
•Port use and I/O rates
•Volume and MDisk I/O rates
•Bandwidth
•Latency
Real-time statistics are always collected and cannot be stopped.
8.2.1 Real-time performance monitoring with the CLI
The following commands are available for monitoring the statistics through the CLI:
•lssystemstats
Run the lssystemstats command to display the most recent values of all of the node or node canister statistics in a clustered system. This command can also be used to display a history of values. You can filter the output to display values only for a subset of available statistics. An example lssystemstats CLI output is shown in Figure 8-1 on page 163.
•lsnodestats and lsnodecanisterstats
Run the lsnodestats (IBM Spectrum Virtualize system) or lsnodecanisterstats (IBM Storwize system) command to display the most recent values of statistics for all of the nodes or node canisters and display all statistics for a particular node canister. Additionally, you can use this command to display a history of values for a subset of available statistics. You can filter the output to display values only from certain nodes or a subset of available statistics.
Example 8-2 shows a lssystemstats command output.
Example 8-2 Example lssystemstats command output
IBM_2145:Redbooks_cluster1:superuser>lssystemstats
stat_name stat_current stat_peak stat_peak_time
compression_cpu_pc 0 0 161117063047
cpu_pc 1 2 161117063022
fc_mb 0 0 161117063047
fc_io 5061 5354 161117062937
sas_mb 0 0 161117063047
sas_io 0 0 161117063047
iscsi_mb 0 0 161117063047
iscsi_io 0 0 161117063047
write_cache_pc 0 0 161117063047
total_cache_pc 0 0 161117063047
vdisk_mb 0 0 161117063047
vdisk_io 0 0 161117063047
vdisk_ms 0 0 161117063047
mdisk_mb 0 0 161117063047
mdisk_io 0 0 161117063047
mdisk_ms 0 0 161117063047
drive_mb 0 0 161117063047
drive_io 0 0 161117063047
drive_ms 0 0 161117063047
vdisk_r_mb 0 0 161117063047
vdisk_r_io 0 0 161117063047
vdisk_r_ms 0 0 161117063047
vdisk_w_mb 0 0 161117063047
vdisk_w_io 0 0 161117063047
vdisk_w_ms 0 0 161117063047
mdisk_r_mb 0 0 161117063047
mdisk_r_io 0 0 161117063047
mdisk_r_ms 0 0 161117063047
mdisk_w_mb 0 0 161117063047
mdisk_w_io 0 0 161117063047
mdisk_w_ms 0 0 161117063047
drive_r_mb 0 0 161117063047
drive_r_io 0 0 161117063047
drive_r_ms 0 0 161117063047
drive_w_mb 0 0 161117063047
drive_w_io 0 0 161117063047
drive_w_ms 0 0 161117063047
iplink_mb 0 0 161117063047
iplink_io 0 0 161117063047
iplink_comp_mb 0 0 161117063047
cloud_up_mb 0 0 161117063047
cloud_up_ms 0 0 161117063047
cloud_down_mb 0 0 161117063047
cloud_down_ms 0 0 161117063047
All three commands list the same set of statistics, but either represent all nodes in the cluster or a particular node (or node canister). The values for these statistics are calculated from the node statistics values in the following way:
Bandwidth Sum of the bandwidth of all nodes.
Latency Average latency for the cluster, which is calculated by using data from the whole cluster and not an average of the single node values.
IOPS Total IOPS of all nodes.
CPU percentage Average CPU percentage of all nodes.
Table 8-1 has a brief description of each of the statistics that are presented by the lssystemstats, lsnodestats, and lsnodecanisterstat commands.
Table 8-1 Field name descriptions for lssystemstats, lsnodestats, and lsnodecanisterstats statistics
|
Field name
|
Unit
|
Description
|
|
compression_cpu_pc
|
Percentage
|
Compression CPU use
|
|
cpu_pc
|
Percentage
|
Use of node CPUs
|
|
fc_mb
|
MBps
|
Fibre Channel (FC) bandwidth
|
|
fc_io
|
IOPS
|
FC throughput
|
|
sas_mb
|
MBps
|
SAS bandwidth
|
|
sas_io
|
IOPS
|
SAS throughput
|
|
iscsi_mb
|
MBps
|
iSCSI bandwidth
|
|
iscsi_io
|
IOPS
|
iSCSI throughput
|
|
write_cache_pc
|
Percentage
|
Write cache fullness (updated every ten seconds)
|
|
total_cache_pc
|
Percentage
|
Total cache fullness (updated every ten seconds)
|
|
vdisk_mb
|
MBps
|
Total VDisk bandwidth
|
|
vdisk_io
|
IOPS
|
Total VDisk throughput
|
|
vdisk_ms
|
Milliseconds (ms)
|
Average VDisk latency
|
|
mdisk_mb
|
MBps
|
MDisk (SAN and RAID) bandwidth
|
|
mdisk_io
|
IOPS
|
MDisk (SAN and RAID) throughput
|
|
mdisk_ms
|
Milliseconds
|
Average MDisk latency
|
|
drive_mb
|
MBps
|
Drive bandwidth
|
|
drive_io
|
IOPS
|
Drive throughput
|
|
drive_ms
|
Milliseconds
|
Average drive latency
|
|
vdisk_w_mb
|
MBps
|
VDisk write bandwidth
|
|
vdisk_w_io
|
IOPS
|
VDisk write throughput
|
|
vdisk_w_ms
|
Milliseconds
|
Average VDisk write latency
|
|
mdisk_w_mb
|
MBps
|
MDisk (SAN and RAID) write bandwidth
|
|
mdisk_w_io
|
IOPS
|
MDisk (SAN and RAID) write throughput
|
|
mdisk_w_ms
|
Milliseconds
|
Average MDisk write latency
|
|
drive_w_mb
|
MBps
|
Drive write bandwidth
|
|
drive_w_io
|
IOPS
|
Drive write throughput
|
|
drive_w_ms
|
Milliseconds
|
Average drive write latency
|
|
vdisk_r_mb
|
MBps
|
VDisk read bandwidth
|
|
vdisk_r_io
|
IOPS
|
VDisk reach throughput
|
|
vdisk_r_ms
|
Milliseconds
|
Average VDisk read latency
|
|
vdisk_w_mb
|
MBps
|
VDisk write bandwidth
|
|
vdisk_w_io
|
IOPS
|
VDisk write throughput
|
|
vdisk_w_ms
|
Milliseconds
|
Average VDisk write latency
|
|
mdisk_r_mb
|
MBps
|
MDisk (SAN and RAID) read bandwidth
|
|
mdisk_r_io
|
IOPS
|
MDisk (SAN and RAID) read throughput
|
|
mdisk_r_ms
|
Milliseconds
|
Average MDisk read latency
|
|
mdisk_w_mb
|
MBps
|
MDisk (SAN and RAID) write bandwidth
|
|
mdisk_w_io
|
IOPS
|
MDisk (SAN and RAID) write throughput
|
|
mdisk_w_ms
|
Milliseconds
|
Average MDisk write latency
|
|
drive_r_mb
|
MBps
|
Drive read bandwidth
|
|
drive_r_io
|
IOPS
|
Drive read throughput
|
|
drive_r_ms
|
Milliseconds
|
Average drive read latency
|
|
drive_w_mb
|
MBps
|
Drive write bandwidth
|
|
drive_w_io
|
IOPS
|
Drive write throughput
|
|
drive_w_ms
|
Milliseconds
|
Average drive read latency
|
|
drive_w_mb
|
MBps
|
Drive write bandwidth
|
|
drive_w_io
|
IOPS
|
Drive write throughput
|
|
drive_w_ms
|
Milliseconds
|
Average drive write latency
|
|
power_w
|
Watts
|
Power consumption
|
|
temp_c
|
Celsius
|
Ambient temperature in Celsius degrees
|
|
temp_f
|
Fahrenheit
|
Ambient temperature in Fahrenheit degrees
|
|
iplink_mb
|
MBps
|
Internet Protocol (IP) link Bandwidth
|
|
iplink_io
|
IOPS
|
IP link Throughput
|
|
iplink_comp_mb
|
MBps
|
IP link compressed Throughput
|
|
cloud_up_mb
|
MBps
|
Average transfer for cloud upload operations
|
|
cloud_up_ms
|
Milliseconds
|
Average response time for cloud upload requests
|
|
cloud_down_mb
|
MBps
|
Average transfer for cloud download operations
|
|
cloud_down_ms
|
Milliseconds
|
Average response time for cloud download requests
|
8.2.2 Real-time performance monitoring with the GUI
The real-time statistics are also available from the IBM Storwize GUI. To view them, click Monitoring → Performance, as shown in Figure 8-1.
Figure 8-1 IBM Storwize Monitoring menu
The performance monitoring console is divided into four sections that provide use views for the following resources:
•CPU Use
– CPU usage (percentage) for general tasks
– Shows the CPU (percentage) usage for compression (when enabled)
•Volumes. This shows the overall volume statistics:
– Read
– Write
– Read latency
– Write latency
•Interfaces. This shows the overall statistics for each of the available interfaces:
– FC
– iSCSI
– Serial-attached SCSI (SAS)
– IP Remote Copy
– IP Remote Copy Compressed
•MDisk. This shows the following overall statistics for the MDisks:
– Read
– Write
– Read latency
– Write latency
Figure 8-2 shows real-time performance graphs.
Figure 8-2 Real-time performance graphs
Each graph represents 5 minutes of collected statistics and provides a means of assessing the overall performance of your system. You can select to view performance statistics for each of the available nodes or canisters of the system, as shown in Figure 8-3.
Figure 8-3 Selecting a system node (canister) for System Statistics
It is also possible to change the metric between MBps or IOPS, as shown in Figure 8-4.
Figure 8-4 Changing a metric to MBps
On any of these views, you can select any point with your cursor to determine the exact value and when it occurred. When you place your cursor over the timeline, it becomes a dotted line with the various values gathered, as shown in Figure 8-5.
Figure 8-5 Detailed resource use information in a graph
There are various values for each of these resources that you can view by selecting the check box next to a value. For example, for the MDisks view that is shown in Figure 8-6, the four available fields are selected.
Figure 8-6 Detailed resource use information in the graph
The following detailed resource use information is available in the graph:
•Read
•Write
•Read latency
•Write latency
8.3 Performance data collection with IBM tools
As explained in 8.1, “Manually gathering performance statistics” on page 158, you can obtain performance reports on an IBM Storwize storage system in standard XML format. However, using .xml files is often an impractical and complex method to analyze the IBM Storwize performance statistics. IBM provides several tools that can be used to view performance information.
8.3.1 IBM Spectrum Control
IBM Spectrum Control is the IBM tool to collect and analyze performance statistics.
For more information about using IBM Spectrum Control to monitor your storage subsystem, see the following resources:
•IBM Spectrum Family: IBM Spectrum Control Standard Edition, SG24-8321
8.3.2 IBM Spectrum Control Storage Insights
IBM Spectrum Control Storage Insights is a SaaS offering with its core running over IBM SoftLayer®. IBM Spectrum Control Storage Insight® capabilities can enhance your storage performance as you lower the cost of storage. This solution is cloud-based, so you can get actionable insights in minutes with deeper insights delivered over time as intelligence about your environment builds in the system. The IT workload for maintaining a storage management infrastructure disappears, which enables you to focus on implementing the insights to optimize your storage environment.
The solution is oriented to small and medium businesses that want to avoid the expensive and time-consuming deployment of an on-premises solution and enable less experienced staff to manage storage environment more efficiently by delivering different insights as simple as looking at the many available dashboards. Large organizations can also deploy IBM Spectrum Control Storage Insights to gain visibility of small storage environments, even if they are already using on-premises solutions such as IBM Virtual Storage Center to manage their core storage systems.
For more information about IBM Spectrum Control Storage Insights to view data for your storage subsystem, see the following information:
•Regain Control of your Environment with IBM Storage Insights, REDP-5231
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.