IBM z15 Model T01 hardware overview
This chapter expands on the descriptions of the key hardware elements of the z15 that were presented in 1.2, “z15 technical description” on page 10. It includes the following topics:
For more information about the key capabilities and enhancements of the z15, see IBM z15 (8561) Technical Guide, SG24-8851.
2.1 Models and upgrade paths
The IBM z15 (machine type 8561) has one model: the T01. The maximum number of characterizable processors is represented by feature names Max34, Max71, Max108, Max145, and Max190.
|
Naming: Throughout this chapter, we refer to the IBM z15 Model T01 (machine type 8561) as z15 T01.
|
As with its predecessors, the z15 T01 central processor complex (CPC) is built by using a processor unit (PU) single-chip modules (SCMs). Each SCM can have 9 - 12 active PUs, or cores. Spare PUs, System Assist Processors (SAPs), and one Integrated Firmware Processor (IFP) are included in the z15 configuration.
The number of characterizable PUs, SAPs, and spare PUs for the various features is listed in Table 2-1. For more information about PU characterization types, see “PU characterization” on page 31.
Table 2-1 z15 T01 processor unit configurations
|
Feature name
|
Number of
CPC drawers
|
Feature code
|
Characterizable processor units
|
Standard SAPs
|
Spares
|
|
Max34
|
1
|
0655
|
1 - 34
|
4
|
2
|
|
Max71
|
2
|
0656
|
1 - 71
|
8
|
2
|
|
Max108
|
3
|
0657
|
1 - 108
|
12
|
2
|
|
Max145
|
4
|
0658
|
1 - 145
|
16
|
2
|
|
Max190
|
5
|
0659
|
1 - 190
|
22
|
2
|
The supported upgrade paths for the z15 T01 are shown in Figure 2-1 on page 25.
.
Figure 2-1 z15 T01 upgrade paths
If an upgrade request cannot be accomplished with the existing configuration, a hardware upgrade is required in which one or more CPC drawers are added to accommodate the wanted capacity. On the z15 T01, more CPC drawers can be installed concurrently from a Max34 to a Max71, and to a Max108.
|
Note: No field upgrade is available to a Max145 or a Max190 because these two features are factory shipped only.
|
On the z15 T01, concurrent upgrades are available for central processors (CPs), Integrated Facilities for Linux (IFLs), Integrated Coupling Facilities (ICFs), IBM Z Integrated Information Processors (zIIPs), and SAPs. However, concurrent PU upgrades require that more PUs are physically installed but not activated previously.
In the rare event of a PU failure, one of the spare PUs is immediately and transparently activated and assigned the characteristics of the failing PU. Two spare PUs always are available on a z15 T01.
In addition, the z15 T01 offers 292 capacity levels. In all, 190 capacity levels, which are based on the number of physically used CPs, are available, plus up to 102 other subcapacity models for the first 34 CPs. For more information, see 5.3.1, “Capacity settings” on page 84.
2.2 Frames and cabling
The z15 T01 uses 19-inch frames and industry-standardized power and hardware. It can be configured as a one-, two-, three-, or four-frame system. Each frame takes up only two standard 24-inch floor tiles of space, which aligns with modern data center layouts.
The z15 T01 packaging introduces new configuration options as compared to previous Z platforms. See Table 2-2.
Table 2-2 z15 configuration options compared to z13 and z14
|
System
|
Number of frames
|
Number of CPC drawers
|
Number of I/O drawers
|
I/O and power connections
|
Power options1
|
Cooling options
|
|
z15
|
1 - 4
|
1 - 5
|
0 - 122
|
Rear only
|
PDU or
BPA
|
Radiator (air) or water-cooling unit (WCU)
|
|
z14
|
2
|
1 - 4
|
0 - 5
|
Front and rear
|
BPA
|
Radiator (air) or WCU
|
|
z13
|
2
|
1 - 4
|
0 - 5
|
Front and rear
|
BPA
|
Radiator (air) or WCU
|
1 The Power Distribution Unit (PDU) option supports the air-cooling (radiator) option, while the Bulk Power Assembly (BPA) option supports both air-cooling and water-cooling options.
2 Maximum of 12 if ordered with a PDU or maximum of 11 if ordered with a BPA.
The number of PCIe+ I/O drawers can vary based on the number of I/O features, power options (PDU or BPA), and number of CPC drawers installed. For a PDU system, a maximum configuration of up to 12 PCIe+ I/O drawers can be installed. PCIe+ I/O drawers can be added concurrently.
In addition, the z15 T01 supports top-exit options for the fiber optic and copper cables that are used for I/O and power. These options give you more flexibility in planning where the system is installed, which potentially eliminates cables to be run under a raised floor and increases air flow over the system.
The radiator-cooled z15 T01 supports installation on raised floor and non-raised floor environments. For the water-cooled system, only the raised floor option is available.
Figure 2-2 on page 27 shows the front view of a fully configured z15 T01 with radiator cooling, five CPC drawers, and 12 PCIe+ I/O drawers.
Figure 2-2 Front view of a fully configured z15 T01 with radiator cooling
Figure 2-3 shows the rear view of a fully configured z15 T01 with water cooling.
Figure 2-3 Rear view of a fully configured z15
The IBM configurator that is used during the order process calculates the number of frames that is required and placement of CPC and PCIe+ I/O drawers.
Factors that determine the number of frames for z15 T01 configuration include the following examples:
•Number of CPC drawers
•Plan ahead features for more CPC drawers
•Number of I/O features (determines the number of PCIe+ I/O drawers)
•Radiator or water cooling
•PDU or BPA power
2.3 CPC drawers
The z15 T01 can hold up to five CPC drawers (three in the A Frame and two in the B Frame). Each CPC drawer contains the following elements:
•SCMs:
– Four PU SCMs, each containing 9 - 12 PU cores (each cooled by an internal water loop).
– One Storage Controller (SC) SCM, with a total of 960 MB L4 cache.
•Memory:
– A minimum of 512 GB and a maximum of 40 TB of memory (excluding 256 GB for hardware system area (HSA)) is available for use. For more information, see Table 2-3 on page 32.
– Up to 20 dual inline memory modules (DIMMs) are plugged in a CPC drawer that are 32 GB, 64 GB, 128 GB, 256 GB, or 512 GB.
•Fanouts
The CPC drawer provides up to 12 PCIe+ fanout adapters to connect to the PCIe+ I/O drawers, and Integrated Coupling Adapter Short Reach (ICA SR) coupling links:
– Two-port PCIe 16 gigabytes per second (GBps) I/O fanout, each port supports one domain in the 16-slot PCIe+ I/O drawers.
– ICA SR1.1 and ICA SR PCIe fanouts for coupling links (two links, 8 GBps each).
•Three or four Power Supply Units (PSUs), depending on the configuration (BPA or PDU), which provide power to the CPC drawer and are accessible from the rear.
Loss of one PSU leaves enough power to satisfy the power requirements of the entire drawer. The PSUs can be concurrently maintained.
•Two dual-function Flexible Support Processor (FSP) oscillator cards (OSCs), which provide redundant interfaces to the internal management network and provide clock synchronization to the Z CPCs.
•Five fans are installed at the front of the drawer to provide cooling airflow for the resources that are installed in the drawer except for the PU SCMs, which are water-cooled.
The CPC drawer communication topology is shown in Figure 2-4 on page 29. All CPC drawers are interconnected with high-speed communications links (A-Bus) through the SC chip L4 shared caches. Symmetric multiprocessor (SMP) cables are used to interconnect all the CPC drawers. The X-Bus provides connectivity between PUs within the logical clusters and the SC on the drawer.
Figure 2-4 z15 T01 CPC drawer communication topology
The design that is used to connect the PU and storage control allows the system to be operated and controlled by the IBM Processor Resource/Systems Manager (PR/SM) facility as a memory-coherent SMP system.
2.3.1 Single-chip modules
The CPC drawer has four PU SCMs and one SC SCM. Each PU SCM supports up to 12 active PU cores, and L1, L2, and L3 caches.
The SC SCM includes 960 MB shared eDRAM cache, interface logic to the four PU SCMs, and SMP fabric logic. The SC SCM is configured to provide L4 cache that is shared by all PU cores in the CPC drawer.
2.3.2 Processor unit
PU is the generic term for an IBM z/Architecture processor. Each PU is a superscalar processor with the following attributes:
•Up to six instructions can be decoded per clock cycle.
•Up to 10 instructions can be in execution per clock cycle.
•Instructions can be issued out of order. The PU uses a high-frequency, low-latency pipeline that provides robust performance across a wide range of workloads.
•Memory accesses might not be in the same instruction order (out-of-order operand fetching).
•Most instructions flow through a pipeline with varying numbers of steps for different types of instructions. Several instructions can be running at any moment, and are subject to the maximum number of decodes and completions per cycle.
PU cache
The on-chip cache for the PU (core) features the following design:
•Each PU core has an L1 cache (private) that is divided into a 128 KB cache for instructions and a 128 KB cache for data.
•Each PU core has a private L2 cache, with 4 MB D-cache (D for data) and 2 MB I-cache (I for instruction).
•Each PU SCM contains a 256 MB L3 cache that is shared by all PU cores in the SCM. The shared L3 cache uses eDRAM.
This on-chip cache implementation optimizes system performance for high-frequency processors, with cache improvements, new Translation/TLB2 design, pipeline optimizations, better branch prediction, new accelerators and architectures, and Secure Execution support.
The z15 cache structure is shown Figure 2-5.
Figure 2-5 z15 T01 cache structure
Drawer cache
In addition to the on-chip (L1, L2, L3) cache, each drawer provides 960 MB L4 cache, for a total of 4,800 MB L4 cache (z15 T01 Max190).
PU sparing
Hardware fault detection is embedded throughout the system design and is combined with comprehensive instruction-level retry and dynamic PU sparing. This function provides the reliability and availability that is required for true IBM Z integrity.
On-core cryptographic hardware
Dedicated on-chip cryptographic hardware for each PU core includes extended key and hash sizes for the Advanced Encryption Standard (AES) and Secure Hash Algorithm (SHA). For more information, see 1.2.5, “Cryptography” on page 17. This cryptographic hardware is available with any processor type, for example, CP, zIIP, and IFL.
On-chip functions
The compression accelerator replaces the IBM zEnterprise Data Compression (zEDC) Express feature that was on previous IBM Z platforms.
The sort accelerator uses the SORTL instruction to be used by DFSORT and the IBM Db2 Utilities for z/OS Suite to help reduce CPU usage and improve elapsed time for sort workloads.
Software support
The z15 PUs provide full compatibility with software for z/Architecture, and extend the Instruction Set Architecture (ISA) to enable enhanced functions and performance. The following hardware instructions that support more efficient code generation and execution are introduced in the z15:
•Central Processor Assist for Cryptographic Functions (CPACF)
•Compression call (CMPSC)
•CPU Measurement Facility (MF)
•Out-of-order execution
•Large page support
•IBM Virtual Flash Memory
•Instruction Execution Protection
•Guarded Storage Facility (GSF)
•Transactional Execution Facility
•Runtime Instrumentation Facility
•Single-instruction, multiple-data (SIMD)
•Secure Execution Facility for Linux
PU characterization
PUs are ordered in single increments. The internal system functions are based on the configuration that is ordered. They characterize each PU into one of various types during system initialization, which is often called a power-on reset (POR) operation.
Characterizing PUs dynamically without a POR is possible by using a process that is called Dynamic Processor Unit Reassignment. A PU that is not characterized cannot be used. Each PU can be designated by using one of the following characterizations:
•CP: These standard processors are used for general workloads.
•IFLs: Designates processors to be used specifically for running the Linux application programs.
•Unassigned Integrated Facilities for Linux (UIFL): Allows you to directly purchase an IFL feature that is marked as being deactivated upon installation, which avoids software charges until the IFL is brought online for use.
•zIIP: An “Off Load Processor” for workloads that are restricted to Db2 type applications. Also used for System Recovery Boost, which is a z15 exclusive feature. For more information, see “Reliability, availability, and serviceability” on page 91.
•Integrated Coupling Facility (ICF): Designates processors to be used specifically for coupling.
•System Assist Processor (SAP): Designates processors to be used specifically for assisting I/O operations.
•IFP: The IFP is standard and not defined by the customer (it is used for infrastructure management).
At least one CP must be purchased before a zIIP can be purchased. You can purchase up to two zIIPs for each purchased CP (assigned or unassigned) on the system. However, a logical partition (LPAR) definition can go beyond the 1:2 ratio. For example, on a system with two physical CPs, a maximum of four physical zIIPs can be installed. An LPAR definition for that system can contain up to two logical CPs and four logical zIIPs. Another possible configuration is one logical CP and three logical zIIPs.
Converting a PU from one type to any other type is possible by using the Dynamic Processor Unit Reassignment process. These conversions occur concurrently with the system operation.
|
Note: The addition of ICFs, IFLs, zIIPs, and SAP to the z15 does not change the system capacity setting or its millions of service units (MSU) rating.
|
2.3.3 Memory
Maximum physical memory size is directly related to the number of CPC drawers in the system. An IBM Z platform features more installed memory than was ordered because part of the installed memory is used to implement the redundant array of independent memory (RAIM) design. With the z15, up to 8 TB of memory per CPC drawer can be ordered and up to 40 TB for a five-CPC drawer system.
|
Important: z/OS V2R3 requires a minimum of 8 GB of memory (2 GB of memory when running under z/VM). z/OS can support up to 4 TB of memory in an LPAR.
|
The minimum and maximum memory sizes for each z15 feature are listed in Table 2-3.
Table 2-3 z15 Model T01 memory per feature
|
Feature name
|
CPC drawers
|
Memory
|
|
Max34 (Feature Code 0655)
|
1
|
512 GB - 8 TB
|
|
Max71 (Feature Code 0656)
|
2
|
512 GB - 16 TB
|
|
Max108 (Feature Code 0657)
|
3
|
512 GB - 24 TB
|
|
Max145 (Feature Code 0658)
|
4
|
512 GB - 32 TB
|
|
Max190 (Feature Code 0659)
|
5
|
512 GB - 40 TB
|
The HSA on the z15 has a fixed amount of memory (256 GB) that is managed separately from available memory. However, the maximum amount of orderable memory can vary from the theoretical number because of dependencies on the memory granularity. On z15 platforms, the granularity for memory is in 64, 128, 256, 512, 1024, and 2048 GB increments.
Physically, memory is organized in the following ways:
•A CPC drawer always contains a minimum of 480 GB to a maximum of 10 TB of installed memory, of which 8 TB maximum is usable by the operating system.
•A CPC drawer can have more installed memory than is enabled. The excess memory can be enabled by a Licensed Internal Code (LIC) load.
•Memory upgrades are first satisfied by using installed but unused memory capacity until it is exhausted. When no more unused memory is available from the installed cards, the cards must be upgraded to a higher capacity, or a CPC drawer with more memory must be installed.
When an LPAR is activated, PR/SM attempts to allocate PUs and the memory of an LPAR in a single CPC drawer. However, if this allocation is not possible, PR/SM uses memory resources in any CPC drawer. For example, if the allocated PUs span more than one CPC drawer, PR/SM attempts to allocate memory across that same set of CPC drawers (even if all required memory is available in only one of those CPC drawers).
No matter which CPC drawer the memory is installed in, an LPAR has access to that memory after it is allocated. Despite the CPC drawer structure, the z15 is still an SMP system because the PUs can access all of the available memory.
A memory upgrade is considered to be concurrent when it requires no change of the physical memory cards. A memory card change is disruptive when no use is made of Enhanced Drawer Availability (EDA). In a multiple-CPC drawer system, a single CPC drawer can be concurrently removed and reinstalled for a repair with EDA.
For model upgrades involving the addition of a CPC drawer, the minimum usable memory increment (256 GB) is added to the system. During an upgrade, adding a CPC drawer and physical memory in the new drawer are concurrent operations.
Concurrent memory upgrade
If physical memory is available, memory can be upgraded concurrently by using Licensed Internal Code Configuration Control (LICCC).
Redundant array of independent memory
RAIM technology makes the memory subsystem (in essence) a fully fault-tolerant N+1 design. The RAIM design automatically detects and recovers from failures of dynamic random access memory (DRAM), sockets, memory channels, or DIMMs.
The RAIM design is fully integrated in the z15, and was enhanced to include one Memory Controller Unit (MCU) per processor chip, with five memory channels and one DIMM per channel. A fifth channel in each MCU enables memory to be implemented as RAIM. This technology has significant reliability, availability, and serviceability (RAS) capabilities in the area of error correction. Bit, lane, DRAM, DIMM, socket, and complete memory channel failures (including many types of multiple failures) can be detected and corrected.
For more information about memory design and configuration options, see IBM z15 (8561) Technical Guide, SG24-8851.
2.3.4 Hardware system area
The HSA is a fixed-size, reserved area of memory that is separate from the customer-purchased memory. The HSA is used for several internal functions, but the bulk of it is used by channel subsystem (CSS) functions.
The fixed-size 256 GB HSA of z15 T01 is large enough to accommodate any LPAR definitions or changes, which eliminates most outage situations and the need for extensive planning.
A fixed, large HSA allows the dynamic I/O capability of the z15 to be enabled by default. It also enables the dynamic addition and removal of the following features:
•LPAR to new or existing CSS
•CSS (up to six can be defined in z15 T01)
•Subchannel set (up to four can be defined in z15 T01)
•A total of 85 LPARs
•Devices, up to the maximum number permitted, in each subchannel set
•Logical processors by type
•Cryptographic adapters
2.4 I/O system structure
The z15 supports the PCIe-based infrastructure for the PCIe+ I/O drawers. The PCIe I/O infrastructure consists of the Dual Port PCIe fanouts in the CPC drawers that support 16 GBps connectivity to the PCIe+ I/O drawer.
The z15 CPC drawer does not support 12x InfiniBand and 1x InfiniBand coupling.
|
Ordering of I/O features: Ordering I/O feature types determines the appropriate number of PCIe+ I/O drawers.
|
Figure 2-6 shows a high-level view of the I/O system structure for the z15 T01.
Figure 2-6 z15 T01 I/O system structure
The z15 T01 CPC drawer has 12 fanouts (numbered LG01 - LG12). The fanouts that are installed in these positions can be one of the following types:
•Dual port PCIe+ fanouts for PCIe+ I/O drawer connectivity
•ICA SR fanouts for coupling
•Filler plates to assist with airflow cooling
For coupling link connectivity (Parallel Sysplex and Server Time Protocol (STP) configuration), the z15 supports the following link types:
•ICA SR1.1 and ICA SR (installed in a CPC drawer)
•Coupling Express Long Reach (CE LR) (installed in a PCIe+ I/O drawer)
For systems with multiple CPC drawers, the locations of the PCIe+ fanouts are configured and plugged across all drawers for maximum availability. This configuration helps ensure that alternative paths maintain access to critical I/O devices, such as storage and networks (see Figure 2-7).
Figure 2-7 z15 T01 CPC drawer: Front view
The PCIe+ I/O drawer (see Figure 2-8), is a 19-inch single side drawer that is 8U high. I/O features are installed horizontally, with cooling air flow from front to rear. The drawer contains 16 adapter slots and two slots for PCIe switch cards.
Figure 2-8 PCIe+ I/O drawer: Rear and front view
The two I/O domains per drawer each contain up to eight I/O features that support the following types:
•FICON Express16SA, FICON Express16S+, FICON Express16S, or FICON Express8S
•OSA-Express7S, OSA-Express6S, or OSA-Express5S
•Crypto-Express7S, Crypto-Express6S, or Crypto-Express5S
•25 GbE RDMA over Converged Ethernet (RoCE) Express2.1, 10 GbE RoCE Express2.1, or 10 GbE RoCE Express
•zHyperLink Express1.1 and zHyperLink Express
•CE LR
For more information about the I/O feature that is available with the z15, see Chapter 4, “Supported features and functions” on page 57.
2.5 Power and cooling
The z15 T01 meets the American Society of Heating, Refrigerating, and Air-Conditioning Engineers (ASHRAE) Class A3 specifications. ASHRAE is an organization that is devoted to the advancement of indoor-environment-control technology in the heating, ventilation, and air conditioning industry.
2.5.1 Power options
The z15 T01 19-inch frames are available with the following power options:
•PDU
Use of PDU for z15 T01 can enable fewer frames, which allows for extra I/O slots and improves power efficiency to lower overall energy costs. It offers some standardization and ease of data center installation planning. PDU supports up to 12 PCIe+ I/O drawers.
•Bulk Power Assembly (BPA)
The BPA supports up to 11 PCIe+ I/O drawers. This option is required when ordered with an Internal Battery Feature (IBF), WCU, or Balanced Power.
|
Statement of Direction: IBM z15 is planned to be the last IBM Z server to offer an IBF. As customer data centers continue to improve power stability and uninterruptible power supply (UPS) coordination, IBM Z continues to innovate to help customers take advantage of common power efficiency and monitoring across their infrastructures. Extra support for data center power planning can be requested through your IBM Sales representative.
Bulk Power® Assembly (BPA) support removal: Based on the direction of the market, the IBM Z system following IBM z15™ is planned to be the last Z system to support BPA. Customers should plan to migrate from BPA to Intelligent Power Distribution Unit (iPDU).
|
The z15 T01 operates with one or two sets redundant power supplies. Each set has its own individual power cords or pair of power cords, depending on the number of Bulk Power Regulator (BPR) pairs that is installed. Power cords attach to a three-phase, 50/60 Hz, 200 - 480 V AC power source. The loss of just one power supply per set has no effect on system operation.
The optional Balanced Power Plan Ahead feature is available for future growth, which also assures adequate and balanced power for all possible configurations. With this feature, downtime for upgrading a system is eliminated because the initial installation includes the maximum power requirements in terms of BPRs and power cords.
2.5.2 Cooling options
The z15 T01 cooling system is available with two options: Radiator (air) cooling or water cooling. SCMs are always cooled with an internal water loop, no matter which cooling option that is chosen. The liquid in the internal water system can be cooled by using a radiator (for air-cooling option) or customer-supplied chilled water supply (for water-cooling option). PCIe+ I/O drawers, power enclosures, and CPC drawers are cooled by chilled air with blowers.
Conversion from air to water-cooled systems, and vice versa, is not available. The following options are available:
•Radiator (air) cooling
The air-cooling system in the z15 T01 is redesigned for better availability and lower cooling power consumption. The radiator design is a closed-loop water-cooling pump system for the SCMs in the CPC drawers. It is designed with N+2 pumps, blowers, controls, and sensors. The radiator unit is cooled by air.
•Water cooling
|
Water Cooling1: IBM z15 is planned to be the last IBM Z server to offer customer water cooling.
|
1 Statements by IBM regarding its plans, directions, and intent are subject to change or withdrawal without notice at the sole discretion of IBM. Information regarding potential future products is intended to outline general product direction and should not be relied on in making a purchasing decision. The information that is mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code, or functions.
The z15 T01 continues to offer the choice of using a building’s chilled water to cool the system by using WCU technology. The SCMs in the CPC drawer are cooled by an internal, closed, water-cooling loop. In the internal closed loop, water exchanges heat with building-chilled water (provided by the customer) through a cold plate.
In addition to the SCMs, the internal water loop circulates through two heat exchangers that are in the path of the exhaust air in the rear of the frames. These heat exchangers remove approximately 60 - 65% of the residual heat from the PCIe+ I/O drawers.
The z15 operates with two fully redundant WCUs in Frame-A and Frame-B on separate loops (when present). One WCU in each loop can support the entire load, and replacement of a WCU is fully concurrent. During a total loss of building-chilled water or if both WCUs fail per frame, the rear door heat exchangers cool the internal water-cooling loop.
The water-cooling option is available with the BPA power option only.
2.5.3 Power considerations
Consider the following points regarding power:
•One to four 42U 19-inch IBM frames are used (replacing the two 24-inch frame).
•Air flow is front to rear. All blowers are mounted on the front of the frame.
•All external power cabling is at the rear of the frames, no power cabling in front.
•Supports top or bottom exit power.
•A High-Voltage DC (HVDC) option is not available.
•No Emergency Power Off (EPO) switch is used.
Specific power requirements depend on the number of frames, the number of CPC drawers, the number and type of I/O units that are installed, and the power option (PDU or BPA).
For more information about the maximum power consumption tables for the various configurations and environments, see IBM 8561 Installation Manual for Physical Planning, GC28-7002.
For more information about the power and weight estimation tool, see IBM Resource Link®.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.