IBM DS8880 hardware components and architecture
This chapter describes the hardware components of the IBM DS8880. It provides insights into the architecture and individual components.
This chapter covers the following topics:
|
Note: The IBM DS8882F Rack Mounted system has several hardware components that slightly differ from those presented here, and is covered in detail in the IBM Redbooks publication Introducing the IBM DS8882F Rack Mounted Storage System, REDP-5505.
|
2.1 Flash drive terminology of the DS8880
It is important to understand the naming conventions that are used to describe the DS8880 components and features. Although most terms are introduced in other chapters of this book, they are repeated and summarized here.
2.1.1 Storage system
The term storage system describes a single DS8880 (base frame plus optional expansion frames).
Base frame
The DS8880 has five available base frame models in two DS8880 families. The model numbers depend on the hardware release for each: DS8888F, DS8886F, and DS8884F models from the all-flash family; and DS8886 and DS8884 models from the hybrid family system configuration. The base frame models for each are listed in Table 2-1.
Each system is a complete storage system that can be contained within a single base frame. To increase the storage capacity and connectivity, expansion frames can be added. Each base frame is equipped with a Hardware Management Console (HMC). The base frame can also contain a second HMC.
For more information about the base frame configuration, see 2.2.7, “DS8880 base frames” on page 34.
Previously available DS8880 models (980, 98B, 981, 98E, 982, 98F) are still supported, but no longer covered in this edition of the document.
Table 2-1 DS8880 Families, frame models and expansion frames
|
DS8880 Families
|
Base Frame Model
|
Expansion Frame model
|
Max expansion Frames
|
|
All-flash
|
|||
|
DS8888F
|
988
|
88E
|
2
|
|
DS8886F
|
985, 986
|
85E, 86E
|
1
|
|
DS8884F
|
984
|
N/A
|
N/A
|
|
DS8882F
|
983
|
N/A
|
N/A
|
|
Hybrid
|
|||
|
DS8886
|
985, 986
|
85E, 86E
|
4
|
|
DS8884
|
984
|
84E
|
2
|
The DS8882F Rack Mounted system is available starting with R8.5 and is covered in the IBM Redbooks publication, Introducing the IBM DS8882F Rack Mounted Storage System, REDP-5505.
Expansion frame
For each of the DS8888F, DS8886F, DS8886 and DS8884 system configurations, one or more optional expansion frames can be installed. The model numbers for the expansion frames also depend on the hardware release and system configuration of the base frame. The expansion frame models are also listed in Table 2-1.
For each system configuration, the first expansion frame provides additional storage capacity and can also contain additional I/O enclosures. Subsequent expansion frames provide additional storage capacity.
Starting with microcode R8.3, up to two expansion frames can be added to the DS8888F configuration. One expansion frame can be added to the DS8886F. Up to four expansion frames can be added to the DS8886 configuration, and up to two expansion frames can be added to the DS8884 configuration.
To add expansion frames to a DS8888F, the system must first be configured with 2048 GB of total system memory. This configuration includes two additional 12-core processors per server, for a total of 48 cores per server.
To add an expansion frame to a DS8886F system, the system must first be configured with a minimum of 256 GB of total system memory and 16-core processors per server.
To add expansion frames to a DS8886 system, the system must first be configured with a minimum of 256 GB of total system memory and 16-core processors per server. For a DS8884 system, the minimum total system memory that is required to add expansion frames is 128 GB.
All DS8880 system memory and processor upgrades can be performed concurrently.
Expansion frames of previous generation DS8000 storage systems are not supported and cannot be installed in a DS8880 storage system. DS8880 expansion frames can only be added to systems with the same configuration and hardware release. For more information about the expansion frame configuration, see 2.2.8, “DS8880 expansion frames” on page 37.
2.1.2 Management Console
The Management Console, also known as the HMC, is the focal point for management operations of the DS8880. The HMC provides connectivity to the client network, and communications to the system private networks, power subsystem, and other management domains. All storage configuration, user-controlled tasks, and some service actions are managed through the HMC. Although many other IBM products use an HMC, the installed Licensed Internal Code (LIC) makes the DS8880 HMC unique to these systems.
The DS8880 HMC also includes the IBM Copy Services Manager code. For details, see IBM DS8880 Integrated Copy Services Manager and LDAP Client on the HMC, REDP-5356.
2.1.3 Central processor complex
The DS8880 has two POWER8 servers, which are referred to as central processor complexes (CPCs) or Central Electronics Complexes (CECs). The CPC is also known as the processor complex or the internal server.
The CPCs for each model type have these characteristics:
•Each DS8888F CPC can have either 24 or 48 processor cores, and either 512 GB or 1024 GB of processor memory, for 1024 GB or 2048 GB of total system memory.
•Each DS8886F and DS8886 CPC can have 8 - 24 processor cores, and 64 - 1,024 GB of processor memory, for 128 GB - 2048 GB of total system memory.
•Each DS8884F and DS8884 CPC has 6 - 12 processor cores and can have 32 - 128 GB of processor memory, for 64 GB - 256 GB of total system memory.
Both CPCs in a DS8880 system share the system workload. The CPCs are redundant, and either CPC can fail over to the other CPC if a failure occurs, or for scheduled maintenance or upgrade tasks. The CPCs are identified as CPC 0 and CPC 1. A logical partition in each CPC runs the AIX V7.x operating system and storage-specific LIC. This logical partition is called the storage node. The storage servers are identified as Node 0 and Node 1 or server0 and server1.
Similar to earlier generations of the IBM DS8000 series, the DS8880 consists of one base frame, and optional expansion frames. The CPCs reside in the base frame, and can be upgraded with additional processor cores and system memory to accommodate growing performance, or when additional storage capacity or host connectivity is required.
Upgrades from the smallest to the largest configuration in terms of system memory, processors, storage capacity, and host attachment can be performed concurrently.
|
Note: The DS8884, DS8884F, DS8886, DS8886F, and DS8888F hardware platforms are specific to each configuration. Upgrades are not supported from one hardware platform to another.
|
2.2 DS8880 configurations and models
This section presents the current DS8880 configurations and models. The DS8880 storage systems are associated with machine type 283x for Hybrid family, or 533x for All-flash family models. The last digit of the machine type corresponds to the length of warranty option chosen, where x equals the number of years. DS8880 offers a one-year, two-year, three-year, or four-year warranty period. Expansion frames must have the same machine type as the base frame. The main variations between models are the combinations of CPCs, I/O enclosures, storage enclosures, disks, and direct current uninterruptible power sources (DC-UPSs).
|
Note: The IBM DS8882F Rack Mounted system has several hardware components that slightly differ from those presented here, and is covered in detail in the IBM Redbooks publication Introducing the IBM DS8882F Rack Mounted Storage System, REDP-5505.
|
2.2.1 DS8888F Analytic Class all-flash configuration
The DS8888F is available with dual 24-core, or dual 48-core processor complexes, with up to 16 Fibre Channel/Fibre Connection (FICON) host adapters (HAs) in the base frame, and up to 16 additional Fibre Channel/FICON HAs in the optional expansion frame, for a total of up to 128 host ports in the system as shown in Figure 2-1.
Figure 2-1 Front view of fully configured DS8888F All-Flash configuration
The DS8888F can be configured with either 1 TB or 2 TB of system memory, and up to four high-performance flash enclosure (HPFE) Gen2 pairs in the base frame. An additional four HPFE Gen2 pairs can be added in the expansion frame. Each HPFE Gen2 pair can contain either 16, 32, or 48 flash drives, of 400 GB, 800 GB, 1.6 TB, 3.2 TB, 3.84 TB, 7.68 TB, or 15.36 TB capacity. The DS8888F All-Flash configuration offers up to 11,796.48 TB of raw flash storage capacity.
Figure 2-1 shows a newer fully configured DS8888F All-Flash configuration, for hardware available with release 8.2.1. The first frame contains the processor complexes, I/O enclosures, and up to four HPFE Gen2 pairs. The second frame contains four additional I/O enclosures, and up to four additional HPFE Gen2 pairs.
Each frame contains two standard 3-phase DC-UPSs that supply redundant power for all installed components. All DC-UPSs in a system contain one battery service module (BSM) set, whether the extended power line disturbance (ePLD) feature is installed or not.
|
Note: The DS8888F configuration supports only three-phase input power.
|
2.2.2 DS8886F Enterprise Class all-flash configuration
The DS8886F is available with dual 8-core, dual 16-core, or dual 24-core processor complexes, with up to 16 Fibre Channel/Fibre Connection (FICON) host adapters (HAs) in the base frame, and up to 16 additional Fibre Channel/FICON HAs in the optional expansion frame, for a total of up to 128 host ports in the system.
The DS8886F is a high-density, high-performance, high-capacity storage system. The DS8886F can be configured with 128 GB - 2 TB of system memory and supports one expansion frame.
Figure 2-2 shows a fully configured DS8886F configuration. This example shows a 3-phase system (986), but single phase(985) systems are similar. It can be configured with four pairs of HPFE Gen2 enclosures in both frames 1 and 2.
Figure 2-2 Front view of fully configured DS8886F configuration with 3-phase DC-UPSs
|
Note: All DC-UPSs in a system must be either single-phase or 3-phase. DS8886 systems do not support field conversion from single-phase to 3-phase DC-UPSs, or from 3-phase to single-phase DC-UPSs.
|
2.2.3 DS8884F Business Class all-flash configuration
The DS8884F is an entry-level, high-performance, high-capacity storage system that includes only High Performance Flash Enclosures Gen2.
The systems feature 6-core (12-core with zHyperLink support) processor complexes with up to 16 Fibre Channel/FICON HAs in the base frame for a total of up to 64 host ports in the system. The total system memory ranges from 64 to 256 GB. The configuration doesn’t have option for expansion racks.
It features 1 or 2 pairs of IO Enclosures which on full configuration supports up to four High Performance Flash Enclosure Gen2 pair in the base frame supporting up to 192 Flash Tier 0,Flash Tier 1, or Flash Tier 2 drives. The frame is 19 inches wide and 40U high. See below in Figure 2-3 on page 31
Figure 2-3 Front view of fully configured DS8884F configuration
|
Note: The DS8884F configuration supports only single-phase input power.
|
2.2.4 DS8882F Rack Mounted Model all-flash configuration
The DS8882F can be integrated into IBM z14 ZR1, IBM LinuxONE™ Rockhopper II (z14 model LR1) with 16 units of reserved space. The system can be integrated into any 19 inch wide rack that conforms to EIA 310D specifications.
The system features 6-core processors and supports one High Performance Flash Enclosure Gen2 pair with up to 48 Flash Tier 0, Flash Tier 1, or Flash Tier 2 drives.
|
Note: The IBM DS8882F Rack Mounted system has several hardware components that slightly differ from those presented here, and is covered in detail in the IBM Redbooks publication Introducing the IBM DS8882F Rack Mounted Storage System, REDP-5505.
|
2.2.5 DS8886 Enterprise class hybrid configuration
The DS8886 is available with dual 8-core, dual 16-core, or dual 24-core processor complexes, with up to 16 Fibre Channel/FICON host adapters HAs in the base frame, and up to 16 Fibre Channel/FICON HAs in the first expansion frame, for a total of up to 128 host ports in the system.
The DS8886 configuration is optimized for performance. It is highly scalable, offering a wide range of options for long-term growth. The DS8886 can be configured with 128 GB - 2 TB of system memory and supports up to four expansion frames.
Figure 2-4 shows a fully configured, five-frame DS8886 configuration with single-phase DC-UPSs. The leftmost frame is a base frame that contains the processor complexes, I/O enclosures, standard drive enclosures, and HPFE Gen2s. The second frame is the first expansion frame that contains I/O enclosures, HPFEs, and standard drive enclosures. The third and fourth frames are also expansion frames that contain only standard drive enclosures.
Figure 2-4 Front view of fully configured DS8886 configuration with single-phase DC-UPSs
Each frame contains two standard single-phase DC-UPSs or two optional 3-phase DC-UPSs that supply redundant power for all installed components. All DC-UPSs in a system contain one BSM set, whether the ePLD feature is installed or not.
|
Note: All DC-UPSs in a system must be either single-phase or 3-phase. DS8886 systems do not support field conversion from single-phase to 3-phase DC-UPSs, or from 3-phase to single-phase DC-UPSs.
|
Figure 2-5 shows the same configuration with 3-phase DC-UPSs and HPFEs Gen2.
Figure 2-5 Front view of fully configured DS8886 configuration with 3-phase DC-UPSs
2.2.6 DS8884 Business Class hybrid configuration
The DS8884 configuration is available with a 6-core processor complex, with up to eight Fibre Channel/FICON HAs in the base frame, and up to eight Fibre Channel/FICON HAs in the first expansion frame, for a total of up to 128 host ports in the system.
The DS8884 configuration employs a different standard drive enclosure cabling scheme to reduce initial configuration costs compared to the DS8886 configuration. The DS8884 configuration increases device adapter (DA) utilization, prioritizing cost-effective storage capacity growth. The DS8884 configuration can be configured with 64 GB - 256 GB of system memory. The DS8884 configuration supports up to two expansion frames.
Figure 2-6 shows a fully configured three-frame DS8884 configuration. The leftmost frame is the base frame that contains the processor complexes, I/O enclosures, standard drive enclosures, and HPFEs Gen2. The second frame is the first expansion frame that contains I/O enclosures, HPFEs, and standard drive enclosures. The third frame is also an expansion frame that contains only standard drive enclosures.
Each frame contains two standard single-phase DC-UPSs that supply redundant power for all installed components. All DC-UPSs in a system contain one BSM set, whether the ePLD feature is installed or not.
|
Note: The DS8884 configuration supports only single-phase input power.
|
Figure 2-6 Front view of fully configured DS8884 configuration with single-phase DC-UPSs
2.2.7 DS8880 base frames
As mentioned in 2.1.1, “Storage system” on page 26, the DS8880 storage system is available indifferent base frame models. Each base frame model number is determined by the configuration and the hardware version. The specific combinations are shown in Table 2-1 on page 26.
|
Note: All DS8880 frames are 19 inches wide with a 40U capacity. The DS8886 frames and DS8888F optionally can be extended from 40U to 46U to allow greater storage capacity in a smaller footprint.
|
The DS8880 base frames accommodate the following components:
•High-Performance Flash Enclosure Gen2, in the following configurations:
– Up to four HPFE Gen2 enclosure pairs in the DS8888F configuration base frame.
– Up to four HPFE Gen2 enclosure pairs in the DS8886F configuration base frame.
– Up to two HPFE Gen2 enclosure pairs in the DS8886 configuration base frame.
– Up to four HPFE Gen2 enclosure pairs in the DS8884F configuration base frame.
– Up to one HPFE Gen2 enclosure pairs in the DS8884 configuration base frame.
Each HPFE Gen2 enclosure pair can accommodate 16, 32, or 48 flash drives. Flash drives are available in 400 GB, 800 GB, 1.6 TB, 3.2 TB, 3.84 TB, 7.68 TB, and 15.36 TB capacities. For more information about the HPFE, see 2.5, “Storage enclosures and drives” on page 59.
•Standard drive enclosures
Standard drive enclosures can accommodate up to twenty-four 2.5-inch small form factor (SFF) serial-attached SCSI (SAS) drives, or twelve 3.5-inch large form factor (LFF) SAS nearline drives. Standard drive enclosures are installed in pairs. For more information about the standard drive enclosures, see 2.5, “Storage enclosures and drives” on page 59. For supported maximums by configuration, see Figure 2-14 on page 45.
All enclosures have redundant power and integrated cooling, which draw air from front to rear. For more information about cooling requirements, see Chapter 7, “IBM DS8880 physical planning and installation” on page 193.
•The power subsystem
Each DS8880 base frame contains two Direct Current Uninterruptible Power Supplies (DC-UPSs), in a fully redundant configuration. Each DC-UPS contains one integrated Battery Service Module (BSM) set. Each DC-UPS has a dedicated input AC power cord. If input AC is not present on one power cord, it is associated DC-UPS continues to operate by using rectified AC from the partner DC-UPS, with no reduction of system power redundancy. If neither AC input is active, the DC-UPSs switch to battery power. If input power is not restored within 4 seconds (40 seconds with ePLD), the DS8880 initiates an orderly system shutdown. For more information about the DC-UPS, see 2.6, “Power and cooling” on page 68.
The DS8888F configuration uses 3-phase DC-UPSs exclusively. The DS8886F and DS8886 configuration can be ordered with single-phase or 3-phase DC-UPSs, whereas the DS8884F and DS8884 configuration uses single-phase DC-UPSs exclusively.
The power subsystem in the DS8880 complies with the latest directives for the Restriction of Hazardous Substances (RoHS), and is engineered to comply with US Energy Star guidelines.
•Hardware Management Consoles (HMCs)
Each base frame comes with a HMC and optional second HMC that are located below the DC-UPSs, and two Ethernet switches (only seen from the rear). For more information about the Management Console, see 2.7, “Management Console and network” on page 70.
•POWER8 CPCs.
Each base frame accommodates two POWER8 CPCs. The POWER8 processor-based servers contain the processors and memory that drive all functions within the DS8880. System memory and processor cores in the DS8888F, DS8886F and DS8886 configurations can be upgraded concurrently. The DS8884F and DS8884 offers concurrent system memory upgrades. For more information about the processor complexes, see 2.3.1, “IBM POWER8-based servers” on page 42.
•I/O enclosures
The DS8888F, DS8886F, DS8884F, and DS8886 base frames accommodate four I/O enclosures. The DS8884 base frame accommodates two I/O enclosures. I/O enclosures are installed in pairs. The I/O enclosures provide PCIe Gen3 connectivity from the I/O adapters and the processor complexes.
The I/O enclosures house the following PCIe I/O adapters:
– 8 Gbps Fibre Channel Host Adapters (HAs) with up to four HAs for each I/O enclosure:
• Either 4-port or 8-port
• Either shortwave (SW) or longwave (LW).
– 16 Gbps Fibre Channel HAs with up to four HAs for each I/O enclosure:
• 4-port only
• Either shortwave (SW) or longwave (LW)
|
Note: An intermix of 8 Gbps Fibre Channel HAs and 16 Gbps Fibre Channel HAs is supported. This intermixture influences the maximum number of possible host ports. For example, a configuration with 16 Gbps Fibre Channel HAs only, each with four ports, supports a maximum of 128 host ports for a DS8886 and 64 host ports for a DS8884.
|
The HAs can be configured in the following manner:
• Fibre Channel Arbitrated Loop (FC-AL) for open systems host attachment (only supported by 8 Gbps Fibre Channel HAs)
• Switched Fibre Channel Protocol (FCP), which is also for open systems host attachment, and for Metro Mirror and Global Copy
• FICON for IBM Z host connectivity and also for z/OS Global Mirror
•zHyperLink adapter
zHyperLink connections with IBM Z hosts provide low latency for random reads and small block sequential writes. It is a point-to-point connection, as shown in Figure 2-7. Note also that zHyperLink is only supported with 12 or more processor cores per DS8880 CEC, and a minimum of overall 256 GB processor memory.
Figure 2-7 DS8880 zHyperLink connection to the system
• Maximum of 16 zHyperLink Ports (eight ports per base frame) on each Storage Subsystem. However, one connection per I/O enclosure (four ports in base frame) is supported at GA time. Minimum of two zHyperlinks per system is preferable. Each link is required to be in a different I/O enclosure.
• Each zHyperLink connection requires a zHyperLink I/O adapter to connect the zHyperLink cable to the storage system. The I/O adapter is plugged into ports T3 and T4 of I/O Enclosure, see Appendix 2.4.2, “I/O enclosure adapters” on page 52.
• The zHyperlink ports that can be installed into the DS8880 systems depends on the number of cores per CEC. The supported combinations for zHyperlink ports availability shown in Table 2-2.
Table 2-2 zHyperLink availability for DS8880 models
|
System/Model
|
Cores per CEC
(DS8880 server)
|
zHyperLink support
|
Max zHyperLink connections (increments of 2)
|
|
DS8884/DS8884F
|
06
|
No
|
None
|
|
12
|
Yes
|
04
|
|
|
DS8886/DS8886F
|
08
|
No
|
None
|
|
16
|
Yes
|
08
|
|
|
24
|
Yes
|
12
|
|
|
DS8888/DS8888F
|
24
|
Yes
|
08
|
|
48
|
Yes
|
16
|
– 8 Gbps Fibre Channel Device Adapters (DAs) in DS8886 and DS8884:
• Up to two DA pairs for each I/O enclosure pair.
• Four ports for each DA.
• Connectivity that is provided to the standard drive enclosures.
For more information about I/O enclosures and I/O adapters, see 2.4, “I/O enclosures and adapters” on page 49.
2.2.8 DS8880 expansion frames
All D8880 system configurations except DS8884F can also have one or more expansion frames. The total number of frames depends on the configuration and model. The supported expansion frame options are shown in Table 2-1 on page 26. Systems must meet certain minimum processor and system memory features, before adding expansion frames.
The DS8880 base frames accommodate the following components:
•High-Performance Flash Enclosure Gen2
The following configurations are possible:
– Up to six HPFE Gen2 enclosure pairs in each DS8888F expansion frames (expansion frames B and C).
– Up to four HPFE Gen2 enclosure pairs in the first DS8886F expansion frame.
Each HPFE Gen2 enclosure pair can accommodate 16, 32, or 48 flash drives. Flash drives are available in 400 GB, 800 GB, 1.6 TB, 3.2 TB, 3.84 TB, 7.68 TB, and 15.36TB capacities. For more information about the HPFE, see 2.5, “Storage enclosures and drives” on page 59.
•Standard drive enclosures
Standard drive enclosures can accommodate up to twenty-four 2.5-inch small form factor (SFF) serial-attached SCSI (SAS) drives, or twelve 3.5-inch large form factor (LFF) SAS nearline drives. Standard drive enclosures are installed in pairs. For more information about the standard drive enclosures, see 2.5, “Storage enclosures and drives” on page 59. For supported maximums by configuration, see Figure 2-14 on page 45.
All enclosures have redundant power and integrated cooling, which draw air from front to rear. For more information about cooling requirements, see Chapter 7, “IBM DS8880 physical planning and installation” on page 193.
•The power subsystem
Each DS8880 base frame contains two DC-UPSs in a fully redundant configuration. Each DC-UPS contains one integrated Battery Service Module (BSM) set. Each DC-UPS has a dedicated input AC power cord. If input AC is not present on one power cord, it is associated DC-UPS continues to operate by using rectified AC from the partner DC-UPS, with no reduction of system power redundancy. If neither AC input is active, the DC-UPSs switch to battery power. If input power is not restored within 4 seconds (40 seconds with ePLD), the DS8880 initiates an orderly system shutdown. For more information about the DC-UPS, see 2.6, “Power and cooling” on page 68.
The DS8888F configuration uses 3-phase DC-UPSs exclusively. The DS8886F and DS8886 configuration can be ordered with single-phase (model 985) or 3-phase (model 986) DC-UPSs, whereas the DS8884F and DS8884 configuration uses single-phase (model 984) DC-UPSs exclusively.
The power subsystem in the DS8880 complies with the latest directives for the Restriction of Hazardous Substances (RoHS), and is engineered to comply with US Energy Star guidelines.
•I/O enclosures
The DS8888F, DS8886F, and DS8886 expansion frames accommodate four I/O enclosures. The DS8884 expansion frames accommodates two I/O enclosures. I/O enclosures are installed in pairs. The I/O enclosures provide PCIe Gen3 connectivity from the I/O adapters and the processor complexes.
The I/O enclosures house the following PCIe I/O adapters:
– 8 Gbps Fibre Channel Host Adapters (HAs) with up to four HAs for each I/O enclosure:
• Either 4-port or 8-port
• Either shortwave (SW) or longwave (LW)
– 16 Gbps Fibre Channel HAs with up to four HAs for each I/O enclosure:
• 4-port only
• Either shortwave (SW) or longwave (LW)
|
Note: An intermix of 8 Gbps Fibre Channel HAs and 16 Gbps Fibre Channel HAs is supported. This intermixture influences the maximum number of possible host ports. For example, a configuration with 16 Gbps Fibre Channel HAs only, each with four ports, supports a maximum of 128 host ports for a DS8886 and 64 host ports for a DS8884.
|
The HAs can be configured in the following manner:
• Fibre Channel Arbitrated Loop (FC-AL) for open systems host attachment (only supported by 8 Gbps Fibre Channel HAs)
• Switched Fibre Channel Protocol (FCP), which is also for open systems host attachment, and for Metro Mirror and Global Copy
• FICON for IBM Z host connectivity and also for z/OS Global Mirror
– zHyperLink connections to IBM Z hosts. Four zHyperLink connections, one per I/O enclosure, are supported starting with release 8.3.1.
|
Note: I/O Priority Manager must be disabled to use a zHyperLink connection.
|
– 8 Gbps Fibre Channel Device Adapters (DAs) in DS8886 and DS8884:
• Up to two DA pairs for each I/O enclosure pair.
• Four ports for each DA.
• Connectivity that is provided to the standard drive enclosures.
For more information, see 2.4, “I/O enclosures and adapters” on page 49.
2.2.9 Scalable upgrades
The hardware features that are supported in the DS8880 depend on the total system memory that is installed. This design ensures that the performance and capacity scale correctly. For more information about processor and system memory requirements for hardware upgrades, see Figure 2-14 on page 45. Each of the DS8880 configurations can be nondisruptively upgraded from the smallest system memory feature to the largest memory feature that is supported by that configuration.
|
Note: The DS8884 and DS8886 configured systems cannot be upgraded or modified to a DS8886 or DS8888F configuration. All upgrades must be within the same configuration.
|
Flash capacity upgrade for DS8880 all-flash systems
Release 8.3 enables flash capacity upgrade with additional HPFE Gen2. For current DS8880F, MTM 533x all-flash models 984, 985, 986, and 988, capacity can be increased by more than 2x. For MTM283x models 984, 985, and 986, supported flash capacity has not changed.
•DS8884F can be upgraded with an additional three pairs of HPFE Gen2 enclosures. Capacity increases from 48 to 192 flash drives, as shown in Figure 2-8.
Figure 2-8 DS8884 MTM 533x model 984
•DS8886F MTM 533x single phase models 985 and 85E and three phase models 986 and 86E can be fully populated with HPFE Gen2, doubling the capacity. With the release of microcode version 8.3, the maximum configuration goes from 192 to 384 Flash drives. Figure 2-9 shows an example of DS8886F MTM 533x with three-phase power (models 986 and 86E).
Figure 2-9 DS8886 3 Phase power MTM 533x model 986 (Base) and 86E (Expansion)
•Release 8.3 enables the support for additional Flash capacity, up to eight pairs of HPFE Gen2 for DS8888F (Analytic Class) MTM 533x. This configuration includes the second expansion frame with six pairs of HPFE Gen2. The maximum configuration goes from 384 to 768 flash drives, as shown in Figure 2-10.
Figure 2-10 DS8888F MTM 533x Models 988 and 88E
2.2.10 Licensed functions
Several of the IBM DS8880 functions require a license key. For more information about licensed functions, see Chapter 9, “IBM DS8880 features and licensed functions” on page 243.
2.3 DS8880 architecture overview
This section provides an architectural overview of the major components of the DS8880:
•IBM POWER8 central processor complexes
•I/O enclosures and adapters
•PCIe connectivity and communication
•Storage subsystem
•Hardware management
2.3.1 IBM POWER8-based servers
All DS8880 systems include two CPCs. In the DS8888F configuration, these are Power E850C servers, which have four processor sockets. They are populated with either two or four 12-core 3.66 GHz processors, for a combined total of 24 or 48 cores per CPC.
The DS8888F CPCs support a maximum of 32 DDR4 custom dual inline memory modules (CDIMMs), with a maximum memory size of 1 TB per CPC.
The DS8888F CPC is a 4U-high drawer, and features the following configuration:
•Sixteen Double Data Rate-4 (DDR4) CDIMM slots
•One storage cage with two hard disk drives (HDDs)
•Four or eight PCIe x16 Gen3 slots
•Three PCIe x8 Gen3 slots
•Four power supplies
Figure 2-11 shows the processor complex as configured in the DS8888F.
Figure 2-11 DS8888F processor complex front and rear views
For more information about the server hardware that is used in the DS8888F, see IBM Power System E850 Technical Overview and Introduction, REDP-5222.
The DS8886F and DS8886 configurations shares the same Power S824 servers that have two processor sockets. These servers are populated with single or dual 8-core 4.15 GHz processors, or dual 12-core 3.52 GHz processors, for a combined total of 8, 16, or 24 cores per CPC. The CPCs support a maximum of 16 DDR3 CDIMMs, with a maximum memory size of 1 TB per CPC.
The DS8886 CPC is a 4U-high drawer, and features the following configuration:
•Sixteen DDR3 CDIMM slots
•One storage cage with two HDDs
•Four PCIe x16 Gen3 slots
•Six PCIe x8 Gen3 slots
•Four power supplies
Figure 2-12 shows the processor complex as configured in the DS8886.
Figure 2-12 DS8886 processor complex front and rear views
For more information about the server hardware that is used in the DS8886F, DS8886, see IBM Power Systems S814 and S824 Technical Overview and Introduction, REDP-5097.
The DS8884 and DS8884F share the same Power S822 servers, each with a single 6-core 3.89 GHz processor. The CPCs support a maximum of 8 DDR3 CDIMMs, with a maximum memory size of 128 GB per CPC. Figure 2-13 shows the S822 servers processor complex.
The CPC is a 2U-high drawer, and features the following configuration:
•Eight DDR3 CDIMM slots
•One storage cage with two hard disk drives
•Two PCIe x16 Gen3 slots
•Four PCIe x8 Gen3 slots
•Two power supplies
Figure 2-13 DS8884 processor complex front and rear views
For more information about the server hardware that is used in the DS8884F and DS8884, see IBM Power System S822 Technical Overview and Introduction, REDP-5102.
The DS8880 base frame contains two processor complexes. The servers based on POWER8 feature up to four processor single chip modules (SCMs) in the DS8888F. For the DS8886F and DS8886, one or two SCMs are used. The DS8884F and DS8884 feature a single SCM.
In the DS8888F, the SCMs are configured with 12 active cores. In the DS8886F and DS8886, the SCMs are configured with 8 or 12 cores. In the DS8884F and DS8884, the SCM is configured with six cores. Processor cores and system memory are configured according to the hardware that is installed in the storage system. Processors and memory can be upgraded concurrently as required to support storage system hardware upgrades. The supported maximum system hardware components depend on the total processor and system memory configuration.
Figure 2-14 shows the supported components for the DS8880 processor and memory options. NVS values are typically 1/16 of installed system memory.
Figure 2-14 Supported components for the DS8880 processor/memory options
2.3.2 Processor memory
The DS8888F, DS8886F, and DS8886 configurations offer up to 2 TB of total system memory. The DS8884F and DS8884 configuration offers up to 256 GB total. Each processor complex contains half of the total system memory. All memory that is installed in each CPC is accessible to all processors in that CPC. The absolute addresses that are assigned to the memory are common across all processors in the CPC. The set of processors is referred to as a symmetric multiprocessor (SMP) system.
The POWER8 processor that is used in the DS8880 operates in simultaneous multithreading (SMT) mode, which executes multiple instruction streams in parallel. The number of simultaneous instruction streams varies according to processor and Licensed Internal Code (LIC) level. SMT mode enables the POWER8 processor to maximize the throughput of the processor cores by offering an increase in core efficiency.
The DS8880 configuration options are based on the total installed memory, which in turn corresponds with a predetermined number of installed and active processor cores.
The following DS8888F configuration upgrades can be performed nondisruptively:
•Processor upgrade from two processors/24 cores, to four processors/48 cores
•Processor memory upgrade, from 512 GB per server, to 1,024 GB per server
The following DS8886F, DS8886 configuration upgrades can be performed nondisruptively:
•Processor configuration upgrade, with 8, 16, or 24 cores per server
•Processor memory upgrade, 64 - 1,024 GB per server
The following DS8884F, DS8884configuration upgrade can be performed nondisruptively:
•Processor memory upgrade, 32 - 128 GB per server
|
Note: System memory and processor upgrades are tightly coupled. They cannot be ordered or installed independently from each other.
|
Caching is a fundamental technique for reducing I/O latency. Like other modern caches, the DS8880 contains volatile memory that is used as a read and write cache and nonvolatile memory that is used to maintain and back up a second copy of the write cache. If power is lost, the batteries keep the system running until all data in nonvolatile storage (NVS) is written to the CPC’s internal disks.
The NVS scales to the processor memory that is installed, which also helps to optimize performance. NVS is typically 1/16 of installed CPC memory, with a minimum of 4 GB. The exception is that NVS is 1/32 of installed CPC memory on systems with 2 TB of total system memory.
The effectiveness of a read cache depends on the read hit ratio, which is the fraction of requests that are served from the cache without necessitating a read from the disk (read miss).
2.3.3 Flexible service processor
Each Power8 processor complex is managed by a service processor that is called a flexible service processor (FSP). The FSP is an embedded controller that is based on an IBM PowerPC® processor.
The FSP controls power and cooling for the CPC. The FSP performs predictive failure analysis for installed processor hardware, and performs recovery actions for processor or memory errors. The FSP monitors the operation of the firmware during the boot process, and can monitor the operating system for loss of control and take corrective actions.
2.3.4 Ethernet Connections
Each Power8 processor complex has a single four port Ethernet card installed. On all models, two of these ports connect to the internal network switches described in Figure 2-41 on page 71. With R8.3, the bottom two unused connections are available for Transparent Cloud Tiering (TCT).
Available with Release 8.3.3 and later, an optional adapter is available as a chargeable feature code, and gives 2 high speed 10 Gbps (LR optical, SFP+) and two low speed 1Gbps (RJ45 copper) connector.
For details about TCT, see “Transparent Cloud Tiering” on page 169.
Figure 2-15 shows the TCT ports on a DS8884.
Figure 2-15 DS8884 TCT ports
2.3.5 Peripheral Component Interconnect Express adapters
Each DS8880 processor complex contains multiple PCIe adapters. These adapters enable point-to-point connectivity between CPCs, I/O enclosures, I/O adapters, and the HPFEs. Depending on the configuration, up to four PCIe adapters are in the DS8880 processor complex.
A DS8880 processor complex is equipped with the following PCIe adapters. These adapters provide connectivity between the CPCs and I/O enclosures:
•The DS8888F has four multi-port PCIe Gen3 adapters in slots 2, 3, 8, and 10.
•The DS8886F, DS8886 has four multi-port PCIe Gen3 adapters in slots 3, 5, 6, and 7.
•The DS8884F, DS8884 has two multi-port PCIe Gen3 adapters in slots 6 and 7.
Figure 2-16 shows the PCIe adapter locations in the DS8888F CPC.
Figure 2-16 PCIe adapter locations in the DS8888F processor complex
Figure 2-17 shows the PCIe adapter locations in the DS8886 CPC.
Figure 2-17 PCIe adapter locations in the DS8886F, DS8886 processor complex
Figure 2-18 shows the PCIe adapter locations in the DS8884 CPC.
Figure 2-18 PCIe adapter locations in the DS8884F, DS8884 processor complex
2.4 I/O enclosures and adapters
The DS8880 base frame and first expansion frame (if installed) contain I/O enclosures, which are installed in pairs.
The DS8880 systems use an improved I/O enclosure, compared to earlier DS8000 systems. The enclosure is PCIe Gen3-capable, and it is attached to the CPCs with PCIe Gen3 cables. The internal transfer rate to each I/O bay is four times faster than DS8870, which uses PCIe Gen2. The new I/O enclosure now supports six PCIe adapter slots, and it has PCIe attachments that allow direct attachment to the High-Performance Flash Enclosures.
DS8888F, DS8886F, and DS8886 CPCs have up to four 2-port PCIe adapters that provide connectivity to the I/O enclosures. The DS8884F, DS8884 CPCs have two 2-port PCIe adapters that provide connectivity.
Figure 2-19 on page 50 to Figure 2-21 on page 51 show the DS8880 CPC to I/O enclosure connectivity. The DS8886F and DS8886 require a dual 8-core configuration for the first expansion frame, and the DS8884 requires a 6-core processor with 128 GB of total system memory for the first expansion frame.
One or two I/O enclosure pairs can be installed in the base frame of the DS8886. The DS8884 has a single I/O enclosure pair. Two I/O enclosure pairs are installed in the first expansion frame of a DS8886 when it is installed. The DS8884 expansion frame has a single I/O enclosure pair. The DS8886F has two I/O enclosure pairs in the base frame. The DS8884F can have one or two I/O enclosure pairs depending on the configuration.
Each I/O enclosure can have up to four Host Adapters (HAs) and two Device Adapters (DAs). A maximum of 16 host adapter ports are supported in a single I/O enclosure. The I/O enclosure features integrated PCIe cable connections for direct attachment to the HPFE. With release 8.3 the I/O enclosure also includes two zHyperlink connections with 16-core models.
Each I/O enclosure includes the following attributes:
•Half-width 5U rack-mountable enclosure
•Six PCIe slots
•Two PCIe Gen3 connections to the HPFE
•Two zHyperlink connections
•Redundant power and cooling
Figure 2-19 shows the DS8888F CPC to I/O enclosure connectivity.
Figure 2-19 DS8888F I/O enclosure connections to the CPC
Figure 2-20 shows the DS8886 CPC to I/O enclosure connectivity.
Figure 2-20 DS8886 I/O enclosure connections to the CPC
Figure 2-21 shows the DS8884 CPC to I/O enclosure connectivity.
Figure 2-21 DS8884 I/O enclosure connections to the CPC
2.4.1 Cross-cluster communication
Figure 2-22 shows how the DS8880 I/O enclosure hardware is shared between the servers. One CPC is on the left side and one CPC is on the right side. The solid lines denote primary PCIe paths, and the dashed lines denote secondary PCIe paths.
Figure 2-22 DS8880 series PCIe communications paths as configured on the DS8886
The DS8880 uses the PCIe paths through the I/O enclosures to provide high-speed communication paths between the CPCs. Normally, the lowest available even-numbered I/O enclosure is used for communication from server 0 to server 1, whereas the lowest available odd-numbered I/O enclosure is used for communication from server 1 to server 0.
If a failure occurs in one or more I/O enclosures, any of the remaining enclosures can be used to maintain communication between the servers.
2.4.2 I/O enclosure adapters
The DS8880 I/O enclosures provide the connectivity from the host systems to the storage arrays through the processor complexes. Each I/O adapter is optimized for its specific task in the DS8880.
Each I/O enclosure can contain up to four HAs that provide attachment to host systems and up to two DAs to provide attachment for standard drive enclosures. Each I/O enclosure has six PCIe x8 connectors on the I/O enclosure PCIe module. With 16-core models, two ports are for the internal PCIe fabric connections to CPC 0 and CPC 1. Two ports provide attachment for the HPFEs. Two ports (T3 and T4) are for attachment of zHyperLink to z14. Only one zHyperLink per I/O enclosure is supported, T3 or T4.
Figure 2-23 shows the DS8880 I/O enclosure adapter layout.
Figure 2-23 DS8880 I/O enclosure adapter Layout
DS8880 host adapters
Attached host servers interact with software that is running on the processor complexes to access data that is stored on logical volumes. The servers manage all read and write requests to the logical volumes on the storage arrays.
Two different types of HAs are available (8 Gbps and 16 Gbps). Both can auto-negotiate their data transfer rate. 16 Gbps HAs can negotiate to 16, 8, or 4 Gbps full-duplex data transfer. The 8 Gbps HAs can negotiate to 8, 4, or 2 Gbps full-duplex. The 8 Gbps HAs are available in both 4-port or 8-port cards. The 16 Gbps HAs are 4-port cards only.
Figure 2-24 shows the 16 Gbps host adapter. It provides faster single stream and per port throughput, and reduces latency in comparison to the 8 Gbps adapter. The 16 Gbps HA is equipped with a quad-core PowerPC processor that delivers dramatic (two to three times) full adapter input/output operations per second (IOPS) improvements compared to the 8 Gbps adapter.
Figure 2-24 16 Gbps host adapter
The 16 Gbps HA contains a new high-performance application-specific integrated circuit (ASIC). To ensure maximum data integrity, it supports metadata creation and checking. Each Fibre Channel port supports a maximum of 509 host login IDs and 1,280 paths. This configuration enables the creation of large storage area networks (SANs).
|
Note: The maximum number of host adapter ports in any single I/O enclosure is 16, regardless of the HA type. If you install 8-port HAs, they are installed in slots 1 and 4 of the I/O enclosure.
|
Each HA port can be configured as either FICON or FCP. For 8 Gbps HAs only, it is also possible to configure the port for the FC-AL protocol. For both HAs, the card optics can be either Long Wave or Short Wave.
The DS8888F, DS8886F, and DS8886 configurations support a maximum of 16 HAs in the base frame, and an additional 16 HAs in the first expansion frame. The DS8884 configuration supports a maximum of eight in the base frame and an additional eight in the first expansion frame. With sixteen 8-port HAs, the maximum number is 128 HA ports. With sixteen 4-port HAs, the maximum number is 64 HA ports. The DS8884F configuration supports up to 16 HAs in the base frame.
HAs are installed in slots 1, 2, 4, and 5 of the I/O enclosure. Figure 2-23 on page 52 shows the locations for four 16 Gbps HA cards in the DS8880 I/O enclosure. DS8880 supports intermix of both adapter types up to the maximum number of ports, as shown on Table 2-3.
|
Optimum availability: To obtain optimum availability and performance, one HA must be installed in each available I/O enclosure before a second HA is installed in the same enclosure.
|
Table 2-3 DS8886 port configurations
|
16 Gbps
FC adapters
|
8 Gbps
FC adapters
|
16 Gbps
FC ports |
8 Gbps FC ports (4-port/8-port)
|
Maximum ports
|
|
0
|
16
|
0
|
64 - 128
|
128
|
|
1
|
15
|
4
|
60 - 120
|
124
|
|
2
|
14
|
8
|
56 - 112
|
120
|
|
3
|
13
|
12
|
52 - 104
|
116
|
|
4
|
12
|
16
|
48 - 96
|
112
|
|
5
|
11
|
20
|
44 - 88
|
108
|
|
6
|
10
|
24
|
40 - 80
|
104
|
|
7
|
9
|
28
|
36 - 72
|
100
|
|
8
|
8
|
32
|
32 - 64
|
96
|
|
9
|
7
|
36
|
28 - 56
|
92
|
|
10
|
6
|
40
|
24 - 48
|
88
|
|
11
|
5
|
44
|
20 - 40
|
84
|
|
12
|
4
|
48
|
16 - 32
|
82
|
|
13
|
3
|
52
|
12 - 24
|
78
|
|
14
|
2
|
56
|
8 - 16
|
74
|
|
15
|
1
|
60
|
4 - 8
|
70
|
|
16
|
0
|
64
|
0
|
64
|
Figure 2-25 shows the preferred HA installation order for the DS8880. The HA locations and installation order for the four I/O enclosures in the base frame are the same for the I/O enclosures in the first expansion frame.
Figure 2-25 DS8880 HA installation order
The possible host port assignments are shown for host adapters in I/O enclosures, starting with I/O enclosure 0 shown in Figure 2-26.
Figure 2-26 WWPN ports assignment for I/O Enclosure 0
Figure 2-27 shows the ports for enclosure 1.
Figure 2-27 WWPN ports assignment for I/O Enclosure 1
Figure 2-28 shows The WWPN ports for enclosure 2.
Figure 2-28 WWPN ports assignment for I/O Enclosure 2
Figure 2-29 shows The WWPN ports for enclosure 3.
Figure 2-29 WWPN ports assignment for I/O Enclosure 3
Figure 2-30 shows The WWPN ports for enclosure 4.
Figure 2-30 WWPN ports assignment for I/O Enclosure 4
Figure 2-31 shows The WWPN ports for enclosure 5.
Figure 2-31 WWPN ports assignment for I/O Enclosure 5
Figure 2-32 shows The WWPN ports for enclosure 6.
Figure 2-32 WWPN ports assignment for I/O Enclosure 6
Figure 2-33 shows The WWPN ports for enclosure 7.
Figure 2-33 WWPN ports assignment for I/O Enclosure 7
Fibre Channel
The DS8880 uses the Fibre Channel protocol to transmit Small Computer System Interface (SCSI) traffic inside Fibre Channel frames. It also uses Fibre Channel to transmit FICON traffic, which uses Fibre Channel frames to carry IBM Z I/O.
Each of the ports on a DS8880 HA can be independently configured for FCP or FICON. In addition, FC-AL can be configured on 8 Gbps ports only. The type of port can be changed through the DS8880 Storage Management GUI or by using Data Storage Command-Line Interface (DS CLI) commands. A port cannot be FICON and FCP simultaneously, but the protocol can be changed as required.
Fibre Channel-supported servers
The current list of servers that are supported by Fibre Channel attachment is available at this website:
Consult these documents regularly because they contain the current information about server attachment support.
Fibre Channel distances
All ports on each card must be either LW or SW. The two types cannot be intermixed within an a single adapter card. With LW, you can connect nodes at distances of up to 10 km (6.2 miles) (non-repeated). With SW, you are limited to a distance that depends on the Fibre Channel cable type and the data transfer rate. OM3 Fibre Channel cable is required with 8 Gbps HAs and 16 Gbps HAs. For 16 Gbps, use OM3 or OM4. For the link distance limitations, see Table 2-4.
Table 2-4 SW link distance
|
Speed
|
OM3 link distance
|
OM4 link distance
|
|
8 Gbps FC
|
150 m (492 ft.)
|
190 m (623.3 ft.)
|
|
16 Gbps FC
|
100 m (328 ft.)
|
125 m (410.1 ft.)
|
Device adapters
DAs are Redundant Array of Independent Disks (RAID) controllers that provide access to the installed drives in the standard drive enclosures. Each of the DS8880 DAs has four 8 Gbps Fibre Channel ports, which are connected to the drive enclosures by using two dual FC-AL loops, by using a switched topology.
The DAs are installed in the I/O enclosures, and they are connected to the processor complexes through the PCIe network. The DAs are responsible for managing, monitoring, and rebuilding the RAID arrays. The DAs provide remarkable performance because of a high function and high-performance ASIC. To ensure maximum data integrity, the adapter supports metadata creation and checking.
Each DA connects the processor complexes to two separately switched FC-AL networks. Each network attaches to standard drive enclosures. Every drive in the enclosure is connected to both networks. Whenever the DA connects to a drive, it uses a bridged connection to transfer data. Figure 2-23 on page 52 shows the locations of the DAs in the I/O enclosure. The DS8886 configuration supports 1 - 4 DA pairs in the base frame, and up to 4 additional DA pairs in the first expansion frame, for a total of up to 8 DA pairs. The DS8884 configuration supports half of the DS8886 configuration, or 1 - 2 DA pairs in the base frame, and up to 2 additional DA pairs in the first expansion frame, for a total of up to 4 DA pairs.
|
Note: The all-flash models DS8888F, DS8886F and DS8884F do not support the installation of Device Adapter cards.
|
2.5 Storage enclosures and drives
This section covers the storage enclosures and drives, which consist of the following components:
•HPFE Gen2 pairs, and flash drives:
– Flash enclosure pairs connect to Flash RAID controllers using SAS cabling.
– Flash RAID controllers connect to I/O enclosures using 8x PCIe Gen3 cabling.
•Standard drive enclosures, and rotational drives, commonly referred to as disk drive modules (DDMs):
– Fibre Channel interface cards (FCICs) in the standard drive enclosures connect to Device Adapter cards using 8 Gbps Fibre. These connections create switched Fibre Channel arbitrated loop networks to the installed drives.
For more information about the storage subsystem, see 3.5, “RAS on the storage subsystem” on page 90.
2.5.1 Drive enclosures
The DS8880 has two types of drive enclosures:
•HPFE Gen2
•Standard drive enclosure
High-performance flash enclosure Gen2 pairs
The DS8880 HPFE Gen2 consists of pair of high-speed 2U Flash Enclosures. One Gen2 enclosure is shown in Figure 2-34. HPFE Gen2 enclosures are always installed in pairs. Each enclosure pair supports 16, 32, or 48 flash drives.
Figure 2-34 High-Performance Flash Enclosure Gen2
Each HPFE Gen2 pair is connected to a redundant pair of Flash-optimized RAID controllers. The flash RAID adapters are PCIe adapters that are either installed in the DS8880 I/O enclosures, or remotely connected to the I/O enclosures through PCIe cables.
The DS8888F configuration can support up to four HPFE Gen2 pairs in the base frame, and up to six HPFE Gen2 pairs in the first expansion frame, and up to six HPFE Gen2 in the third expansion frame, for a total of sixteen HPFE Gen2 pairs, with a maximum of 768 flash drive.
The DS8886F configuration can support up to four HPFE Gen2 pairs in the base frame, and up to four HPFE Gen2 pairs in the first expansion frame, for a total of eight HPFE Gen2 pairs, with a maximum 384 flash drives.
The DS8884F configuration can support four HPFE Gen2 pairs in the base frame, for a total of four HPFE Gen2 pairs, with a maximum 192 flash drives.
To learn more about the HPFE Gen2, see IBM DS8880 High-Performance Flash Enclosure Gen2, REDP-5422.
Flash drives in the HPFE Gen2
Each HPFE Gen2 pair can contain 16, 32, or 48 flash drives. Flash drives are available in 400 GB, 800 GB, 1.6 TB, 3.2 TB, 3.84 TB, 7.68 TB, or 15.36 TB capacities. All flash drives in a HPFE Gen2 enclosure pair must be of the same type (high performance or high capacity). However, high-capacity and high-performance flash drives cannot be mixed in the same enclosure pair.
Flash drive sets
Flash drives are ordered in drives sets of 16. The HPFE Gen2 pair can contain 16, 32, or 48 flash drives (1, 2 or 3 drive sets). All flash drives in a HPFE Gen2 pair must be the same type. Half the drive set will be installed in each enclosure of the pair.
Storage-enclosure fillers
Storage-enclosure fillers fill empty drive slots in the storage enclosures. The fillers ensure sufficient airflow across populated storage. For HPFE Gen2, one filler feature provides a set of 16 fillers (feature code 1699).
High-performance flash drives
The DS8880 HPFE Gen2 supports 2.5-inch high-performance flash drives, designated as Flash Tier 0 (Table 2-5). All high-performance flash drives are Full Disk Encryption (FDE) capable. For more information about licensed features, see Chapter 9, “IBM DS8880 features and licensed functions” on page 243.
Table 2-5 Supported high-performance flash drives
|
Feature code
|
Drive capacity
|
Drive type
|
Drive speed in RPM (K=1000)
|
RAID support
(Default RAID-6)
|
|
1610
|
400 GB
|
2.5-in Flash Tier 0
|
N/A
|
5, 6, 101
|
|
1611
|
800 GB
|
2.5-in flash tier 0
|
N/A
|
5, 6, 10a
|
|
1612
|
1.6 TB
|
2.5-in Flash Tier 0
|
N/A
|
6, 102
|
|
1613
|
3.2 TB
|
2.5-in Flash Tier 0
|
N/A
|
6, 10b
|
1 RAID-5 is supported, but not recommended
2 RAID-5 is only supported with an RPQ.
|
Note: To learn more about the DS8880 drive features, see the IBM System Storage DS8880 Introduction and Planning Guide, GC27-8525.
|
High-capacity flash drives
The DS8880 HPFE Gen2 supports 2.5-inch high-capacity flash drives (Table 2-6). Each HPFE Gen2 pair can contain 16, 32, or 48 flash drives. High-capacity flash drives, designated as Flash Tier 1 have a 3.84 TB capacity. High-capacity flash drives, designated as
Flash Tier 2 have 7.68 TB and 15.36 TB capacities.
Flash Tier 2 have 7.68 TB and 15.36 TB capacities.
Table 2-6 Supported high-capacity flash drives
|
Feature Code
|
Drive capacity
|
Drive type
|
Drive speed in RPM (K=1000)
|
RAID support
(Default RAID-6)
|
|
1623
|
03.84 TB
|
2.5-in. Flash Tier 1
|
NA
|
6, 101
|
|
1624
|
07.68 TB
|
2.5-in. Flash Tier 2
|
NA
|
6
|
|
1625
|
15.36 TB
|
2.5-in. Flash Tier 2
|
NA
|
6
|
1 RAID-5 is only supported with an RPQ.
Arrays and spares
Each HPFE Gen2 pair can contain up to six array sites. The first set of 16 flash drives creates two 8-flash-drive array sites. RAID 6 arrays will be created by default on each array site. RAID 5 is optional for flash drives smaller than 1 TB, but is not advised. RAID 10 is optional for all flash drive sizes, except for the 7.68 TB and 15.36 TB Flash Tier 2 drives.
During logical configuration, RAID 6 arrays and the required number of spares are created. Each HPFE Gen2 pair will have two global spares, created from the first increment of 16 flash drives. The first two arrays to be created from these array sites are 5+P+Q. Subsequent RAID 6 arrays in the same HPFE Gen2 Pair will be 6+P+Q.
For more information about DS8880 virtualization and configuration, see Chapter 4, “Virtualization concepts” on page 105
System capacity limitations
There are capacity limitations depending on the use of small or large extents. The maximum amount of raw capacities from small and large extents depends on the amount of system cache.
Systems with 512 GB of processor memory or less can support up to 512 TB of raw capacity with small extents or up to 2 PB of raw capacity with large extents. The systems with more than 512 GB of processor memory can support up to 2 PB of raw capacity with small extents or up to 8 PB of raw capacity with large extents. See Table 2-7.
Table 2-7 Maximum physical capacity for large and small extents
|
Maximum physical capacity for large and small extents based on system cache size
|
||
|
Cache
|
Small extents
|
Large extents
|
|
less than or equal to 512 GB
|
512 TiB (FB)
560 TiB (CKD)
|
2024 TiB (FB)
2240 TiB (CKD
|
|
greater than 512 GB
|
2024 TiB (FB)
2240 TiB (CKD)
|
8 PiB (FB)
7.4 PiB (CKD)
|
DS8888F systems always have a minimum of 1 TB cache. This means these systems are affected by the 2PB limitation with small extents and the 8 PB limitation.
Flash RAID adapters
The flash RAID adapter pairs provide high bandwidth and redundant PCIe3 x8 connectivity to the DS8880 processor nodes through the IO enclosures. The flash RAID adapter pairs then provide redundant SAS connectivity to the HPFE Gen2 pairs.
The DS8880 flash RAID adapter is available in three different form factors, depending on the DS8880 model and location within that model. Internally, the three different form factors have the same core hardware and function.
To differentiate between the three form factors, they have unique naming and features. These are as follows:
•SAS flash RAID adapter
– Installed directly into a PCIe3 x8 adapter slot in the IO enclosure
– Connects to HPFE Gen2 pairs 9 - 16 in model 988
•Microbay flash RAID adapter
– Remotely connected to the IO enclosures by a PCIe3 x8 cable to a standalone enclosure
– Connects to HPFE Gen2 pairs in models 984, 985, 986 and the first eight HPFE Gen2 pairs for model 988
– The microbay enclosure has its own power supplies and integrated cooling
Figure 2-35 shows an internal view of a microbay, showing the RAID controller on the left, PCIe bridge on the right, and dual cooling fans. Each microbay is powered by the I/O enclosure pair, through a redundant pair of Power Junction Assemblies (PJAs). Each PJA receives power from an I/O enclosure, and distributes power to both microbays in the pair.
Figure 2-35 Internal view of a microbay
•Base IO expander with flash RAID adapter
– Is a PCIe3 x8 adapter that is physically imbedded into the base PCIe IO expander, which is installed in the DS8882F 2U IO enclosure
– Exclusively available only in the DS8882F model 983
2.5.2 Standard drive enclosures
The DS8880 traditional spinning drives, also known as DDMs, and flash drives, also known as solid-state drives (SSDs), are installed in standard drive enclosures. Standard drive enclosures are installed in pairs.
A standard drive enclosure pair supports either twenty-four 2.5-inch small form factor (SFF) or twelve 3.5-inch large form factor (LFF) drives. Each drive is an industry-standard serial-attached SCSI (SAS) drive.
Flash drives can be installed in standard drive enclosure pairs in drive sets of 8 or 16, depending on the capacity of the drive. The 2.5-inch SFF drives are installed in drive sets of 16. The 3.5-inch LFF drives are in installed in drive sets of eight.
Half of each drive set is installed in each enclosure of the enclosure pair.
|
Note: If the standard drive enclosure pair is not fully populated, you must install fillers in the empty slots.
|
Each drive connects into the enclosure midplane. The midplane provides physical connectivity for the drives, FCICs, and power supply units (PSUs).
Each drive enclosure has a redundant pair of FCICs with two inbound and two outbound 8 Gbps Fibre Channel (FC) connections. The FCICs provide the Fibre Channel to SAS conversion and interconnection logic for the drives. In addition, the FCICs include SCSI Enclosure Services (SES) processors that provide all enclosure services.
Figure 2-36 shows the front and rear views of the standard drive enclosure.
Figure 2-36 DS8880 standard drive enclosures
Switched Fibre Channel Arbitrated Loop
The DS8880 uses switched FC-AL technology to link the Fibre Channel DA pairs to the standard drive enclosures. Switched FC-AL uses the standard FC-AL protocol. However, the physical implementation is different. Switched FC-AL technology includes the following key features:
•Standard FC-AL communication protocol from the DA to drives.
•Direct point-to-point connections are established between the DA and drives.
•Isolation capabilities to provide easy problem determination during drive failures.
•Predictive failure statistics.
•Simplified expansion, where no cable rerouting is required when additional drive enclosure pairs are installed to the Fibre Channel network. The DS8880 architecture uses dual redundant switched FC-AL access to each of the drive enclosures. This configuration features the following key benefits:
– Two independent switched FC-AL networks provide high-performance connections to the drives.
– Four access paths are available to each drive.
– Each DA port operates independently.
– This configuration doubles the bandwidth over traditional FC-AL loop implementations.
Figure 2-37 shows each drive attached to two separate FCICs with connections to the drives. By using two DAs, redundant data paths exist to each drive. Each DA can support two switched Fibre Channel networks.
Figure 2-37 DS8880 drive enclosure (only 16 drives are shown for simplicity)
Arrays across loops
For the DS8880, the upper enclosure connects to one dual loop and the lower enclosure connects the other dual loop, in a drive enclosure pair.
Each enclosure places two FC switches onto each dual loop. Drives are installed in groups of 16. Half of the new drives are installed in one drive enclosure, and the other half is placed into the other drive enclosure of the pair.
A standard drive enclosure array site consists of eight drives: Four from each enclosure of the pair. When a RAID array is created from the drives of an array site, half of the array is in each storage enclosure. The performance is increased because the bandwidth is aggregated across the hardware of both enclosures.
One storage enclosure of the pair is on one FC switched loop, and the other storage enclosure of the pair is on a second switched loop. This configuration splits the array across two loops, which is known as array across loops (AAL), as shown in Figure 2-39. Only 16 drives are shown, with eight in each drive enclosure. When fully populated, 24 drives are in each enclosure.
Figure 2-38 shows the DA pair layout. One DA pair creates two dual switched loops.
Figure 2-38 DS8880 switched loop layout (only eight drives per enclosure are shown for simplicity)
Figure 2-39 shows the layout of the array sites. Array site 1 in green (the darker drives) consists of the four left drives in each enclosure. Array site 2 in yellow (the lighter disks) consists of the four right drives in each enclosure. When an array is created, the array is accessible across both dual loops.
Figure 2-39 Array across loop
Benefits of AAL
AAL is implemented to increase performance. When the DA writes a stripe of data to a RAID array, it sends half of the write to each switched loop. By splitting the workload in this manner, the workload is balanced across the loops. This configuration aggregates the bandwidth of the two loops and improves performance. If RAID 10 is used, two RAID 0 arrays are created. Each loop hosts one RAID 0 array. When the read I/O is serviced, half of the reads can be sent to each loop, which improves performance by balancing the workload across the loops.
Expansion
Device adapters (DAs) are installed in the I/O enclosures in pairs. Each DA of the pair is in a separate I/O enclosure of the I/O enclosure pair. The DS8886 configuration can support up to four DA pairs in the base frame, and four DA pairs in the first expansion frame for a total of eight DA pairs. The DS8884 configuration can support up to two DA pairs in the base frame, and two DA pairs in the first expansion frame, for a total of four DA pairs.
Standard drive enclosure pairs are connected to a DA pair. Each DA pair can support up to four drive enclosure pairs. If more storage capacity is required, an additional DA pair can be installed (up to the maximum supported quantity), and then additional standard drive enclosure pairs can be installed (up to the maximum supported quantity).
Drive sets can also be added to standard drive enclosures that are not fully populated. Half of the drive set is added to each enclosure of the drive enclosure pair. Performance will be superior if drive enclosure pairs are distributed across more DA pairs, aggregating the bandwidth of the installed hardware.
For more information about DA pairs and standard drive enclosure distribution and cabling, see 2.2.1, “DS8888F Analytic Class all-flash configuration” on page 29 and 2.2.6, “DS8884 Business Class hybrid configuration” on page 33.
DS8880 standard and flash drives
The DS8880 supports the following drive types (Table 2-8 and Table 2-9 on page 67):
•2.5-inch High-performance flash drives (see “High-performance flash drives” on page 60)
•2.5-inch High-capacity flash drives
•2.5-inch SAS flash drives, which are also known as SSDs
•2.5-inch SAS enterprise drives (15K or 10K revolutions per minute (RPM))
•3.5-inch SAS nearline drives (7200 RPM)
Table 2-8 shows the flash drive types supported by DS8880.
Table 2-8 DS8880 supported flash drive types
|
Feature code
|
Drive capacity
|
Drive type
|
Drive speed in RPM (K=1000)
|
RAID support
(Default RAID-6)
|
Drives for each set
|
|
6158
|
400 GB
|
Flash drive (SSD)
|
N/A
|
5, 6, 101
|
16
|
|
6258
|
800 GB
|
Flash drive (SSD)
|
N/A
|
5, 6, 10a
|
16
|
|
6358
|
1.6 TB
|
Flash drive (SSD)
|
N/A
|
6, 102
|
16
|
1 RAID 5 is supported, but not recommended
2 RAID 5 is not supported
Table 2-9 shows the enterprise and nearline drive types supported by DS8880.
Table 2-9 DS8880 supported enterprise and nearline drive types
|
Feature code
|
Drive capacity
|
Drive type
|
Drive speed in RPM (K=1000)
|
RAID support
(Default RAID-6)
|
Drives for each set
|
|
5308
|
300 GB
|
2.5-inch SAS Ent
|
15K
|
5, 6, 101
|
16
|
|
5618
|
600 GB
|
2.5-inch SAS Ent
|
15K
|
5, 6, 10a
|
16
|
|
5708
|
600 GB
|
2.5-inch SAS Ent
|
10K
|
5, 6, 10a
|
16
|
|
5768
|
1.2 TB
|
2.5-inch SAS Ent
|
10K
|
6, 102
|
16
|
|
5778
|
1.8 TB
|
2.5-inch SAS Ent
|
10K
|
6, 10b
|
16
|
|
5868
|
4 TB
|
3.5-inch SAS NL
|
7.2K
|
6, 10b
|
8
|
|
5878
|
6 TB
|
3.5-inch SAS NL
|
7.2K
|
6, 10b
|
8
|
1 RAID 5 is supported, but not recommended
2 RAID 5 is not supported
Arrays and spares
During logical configuration, arrays and spares are created from groups of eight drives, which are called array sites. The required number of spares for each DA pair is four drives of the same capacity and speed. For example, the first four RAID 6 arrays that are created are 5+P+Q+S. Additional RAID 6 arrays (of the same capacity and speed) that are configured on the DA pair are 6+P+Q.
The next group of 16 drives creates two 8-drive array sites. These next two arrays to be created from these array sites are 6+P+Q. For more information about DS8880 virtualization and configuration, see Chapter 4, “Virtualization concepts” on page 105.
2.6 Power and cooling
The DS8880 power and cooling system is highly redundant. The components are described in this section. For more information, see 3.6, “RAS on the power subsystem” on page 98.
2.6.1 Rack Power Control
The DS8880 features a pair of redundant rack power control (RPC) cards, which are used to control the power sequencing of the system. As in earlier DS8000 models, the DS8880 RPCs are connected to the FSP in each processor complex, which allows them to communicate to the HMC and the storage system. The DS8880 RPC cards also add a second communication path to each of the processor complex operating partitions. The DS8880 RPCs also feature dedicated communication paths to each DC-UPS.
2.6.2 Direct current uninterruptible power supply
Each DS8880 frame contains two DC-UPSs. Depending on model configuration, these can be either single-phase 8 kW 8U DC-UPSs, or three-phase, 12 kW 12U DC-UPSs. The DC-UPSs cannot be intermixed in any DS8880 storage system.
The DC-UPS provides rectified AC power distribution and power switching for redundancy. Two redundant DC-UPSs are in each frame of the DS8880. Each DC-UPS features internal fans to supply cooling for that power supply.
The frame features two AC power cords. Each cord feeds a single DC-UPS. The DC-UPS distributes rectified AC. If AC is not present at the input line, the partner DC-UPS can provide rectified AC to the DC-UPS that lost input power, with no reduction in DC-UPS redundancy. If no AC input is present for either DC-UPS in the frame, the DC-UPSs switch to battery power. Depending on whether the ePLD feature is installed on the system, the system runs on battery power for either 4 or 40 seconds before it initiates an orderly shutdown.
2.6.3 Battery service module set
The DC-UPS contains integrated battery sets that are known as BSM sets. The BSM set consists of four BSM modules. The BSM sets help protect data if external power to the frame is lost. If AC input power to the frame is lost, the batteries are used to maintain power to the processor complexes and I/O enclosures for sufficient time to allow the contents of NVS memory (modified data that is not yet destaged from cache) to be written to the hard disk drives that are internal to the processor complexes (not the storage enclosure drives).
The BSMs sets consist of two BSM types:
•Each BSM set contains one primary BSM. The primary BSM is the only BSM with an electrical connection to the DC-UPS.
•Each BSM set also contains three secondary BSMs that are cabled to the primary BSM.
2.6.4 Extended Power Line Disturbance feature
The duration that the DC-UPSs can run on battery before system shutdown is initiated depends on whether the ePLD feature is installed.
The optional ePLD feature allows the system to run for up to 40 seconds without line input power and then gracefully shuts down the system if power is not restored. If the ePLD feature is not installed, the system initiates shutdown after 4 seconds if frame power is not restored. For more information about why this feature might be necessary, see 3.6.3, “Line power fluctuation” on page 101.
2.6.5 Power cord options
The power cord must be ordered specifically for the input voltage to meet certain requirements. The power cord connector requirements vary widely throughout the world. The power cord might not include the suitable connector for the country in which the system is installed. In this case, the connector must be replaced by an electrician after the system is delivered. For more information, see the IBM System Storage DS8880 Introduction and Planning Guide, GC27-8525.
2.6.6 Power distribution
In each frame, the two DC-UPSs supply output power to four Power Distribution Units (PDUs).
In the base frame, the PDUs supply power to the processor complexes, the I/O enclosures, the standard drive enclosures, and the HPFEs. In the first expansion frame, the PDUs supply power to the I/O enclosures, standard drive enclosures, and the HPFEs. In the second and third expansion frames, the PDUs supply power only to the standard drive enclosures. No HPFEs, I/O enclosures, or processor complexes are in these frames.
2.6.7 Enclosure power supplies
The CPCs, I/O enclosures, and standard drive enclosures feature dual redundant PSUs for each enclosure. The PSUs are supplied power from the DC-UPS through the PDUs. Each PSU in an enclosure is supplied from different DC-UPSs for redundancy. The PSUs have their own internal cooling fans. Each enclosure also has its own redundant cooling fans. All cooling fans draw cool air from the front of the frame and exhaust hot air to the rear of the frame.
|
Note: The DS8880 is designed for efficient air flow and to be compliant with hot and cold aisle data center configurations.
|
2.6.8 Power junction assembly
The PJA is a component of the DS8880 power subsystem that combines and distributes power from multiple sources. Dual PJAs provide redundant power to the Management Console (HMC) and the Ethernet switches. Additional redundant PJAs distribute power from the I/O enclosures to the High-Performance Flash Enclosure Gen2 microbays in systems that contain HPFE Gen2 pairs.
2.7 Management Console and network
All configuration base frames ship with one Management Console, which is also known as the HMC, and two private network Ethernet switches. An optional second Management Console is available as a redundant point of management, and is also installed in the base frame. The introduction of the 19-inch rack introduced a new HMC, which is an SFF mini PC HMC.
Figure 2-40 shows a diagram of the mini PC HMC and its location in the base frame. The mini PC HMC resides in the bottom of the base frame. As an option, a redundant HMC is available. This secondary HMC no longer needs to be added to a customer rack. This secondary HMC sits next to the primary HMC.
Figure 2-40 Diagram of new mini PC HMC and location in the base frame
The storage administrator runs all DS8880 logical configuration tasks by using the Storage Management GUI or DS CLI. All client communications to the storage system are through the HMC.
Clients that use the DS8880 advanced functions, such Metro Mirror or FlashCopy, for example, communicate to the storage system with Copy Services Manager (CSM). CSM is built into Spectrum Control Standard and Advanced. It replaces the IBM Tivoli Productivity Center for Replication through the Management Console.
The Management Console provides connectivity between the storage system and Encryption Key Manager servers.
The Management Console also provides the functions for remote call-home and remote support connectivity.
For more information about the HMC, see Chapter 8, “IBM DS8880 Management Console planning and setup” on page 219.
2.7.1 Ethernet switches
The DS8880 base frame has two 8-port Ethernet switches. The two switches provide two redundant private management networks. Each CPC includes connections to each switch to allow each server to access both private networks. These networks cannot be accessed externally, and no external connections are allowed. External client network connection to the DS8880 system is through a dedicated connection to the Management Console. The switches receive power from the PJAs and do not require separate power outlets. The ports on these switches are shown in Figure 2-41.
Figure 2-41 Ethernet switch ports
|
Important: The internal Ethernet switches that are shown in Figure 2-41 are for the DS8880 private network only. No client network connection must ever be made directly to these internal switches.
|
2.7.2 DS8880 operator panel
The DS8880 status indicators are on the front door. Figure 2-42 shows the operator panel for a DS8880 storage system.
Figure 2-42 DS8880 frame operator indicators
Each panel has two power cord indicators, one for each power cord (frame input power). For normal operation, both of these indicators are illuminated green if each power cord supplies the correct power to the frame. Another LED is below the line indicators with a normal state of off. If this LED is lit solid yellow (in only the base frame), an open event exists in the problem log, and service is required. If this LED flashes (in any frame), the frame is being serviced.
The DS8880 has no physical power on/off switch because power sequencing is managed through the HMC. This configuration ensures that all data in nonvolatile storage, which is known as modified data, is destaged correctly to the drives before the DS8880 powers down.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.