Central processor complex hardware components
This chapter introduces the z14 central processor complex (CPC) hardware components. It also describes the significant features and functions with their characteristics and options.
|
Note: Throughout this chapter, “z14” refers to IBM z14 Model M0x (Machine Type 3906) unless otherwise specified.
|
The objective of this chapter is to explain the z14 hardware building blocks and how these components interconnect from a physical point of view. This information is useful for planning purposes and can help in defining configurations that fit your requirements.
This chapter includes the following topics:
2.1 Frames and drawers
IBM Z frames are enclosures that are built to Electronic Industries Alliance (EIA) standards. The z14 server (Machine Type 3906) has two 42U EIA frames (A and Z), which are bolted together. They have positions for up to four processor drawers and up to five Peripheral Component Interconnect Express (PCIe) I/O drawers. The other components included in the frames are described in the following sections.
The z14 server and its predecessor, the z13, have the option of ordering the infrastructure to support the top exit of fiber optic cables (FICON, OSA, 12x InfiniBand, 1x InfiniBand, ICA, zHyperLink Express, Coupling Express LR, and RoCE) and copper cables for the 1000BASE-T Ethernet features. On the z14 server, the top exit capability is designed to provide an option for overhead power cabling.
The z14 server can be delivered as an air-cooled or water-cooled system. An air-cooled system with the optional top exit I/O and power feature with the maximum of four CPC drawers and five PCIe I/O drawers is shown in Figure 2-1.
Figure 2-1 z14 internal front view of an air-cooled CPC (Machine type 3906, models M04 or M05)
A water-cooled system without the optional top exit I/O and power feature with the maximum of four CPC drawers and five PCIe I/O drawers is shown in Figure 2-2.
Figure 2-2 z14 internal front view of a water-cooled CPC (Machine type 3906, models M04 or M05)
2.1.1 The A frame
As shown in Figure 2-1 on page 36 and Figure 2-2, the A frame includes the following major components (from top to bottom of the frame):
•Two Support Element (SE) 1U servers are installed at the top of the A frame.
•Two optional integrated battery features (IBFs), which provide the function of a local uninterrupted power source. The IBF enhances the robustness of the power design, which increases power line disturbance immunity.
•One PCIe I/O drawer. The inclusion of this PCIe I/O drawer depends on the I/O configuration of the z14 server.
•Two System Control Hubs (SCHs). The SCHs are the replacement of the Bulk Power Hubs that were used in IBM Z before z13 servers.
•Up to four CPC drawers. The number of the CPC drawers depends on the model number of the z14 server. At least one CPC drawer must be installed, and the extra slots are filled up from the bottom to the top.
•The cooling units for the closed loop water-cooling system of the CPC drawers in the bottom of the A Frame differ for air-cooled and water-cooled systems. Consider the following points:
– For an air-cooled z14 server (see Figure 2-1 on page 36), two pumps and two blowers (N+1 redundant design) are installed.
– For a water-cooled z14 server (see Figure 2-2), two Water Conditioning Units (WCUs) are installed. The WCUs are connected to a customer-operated (data center) chilled water supply.
2.1.2 Z Frame
As shown in Figure 2-1 on page 36 and Figure 2-2 on page 37, the Z frame includes the following major components (from top to bottom of the frame):
•Two or four optional IBFs. The number of batteries depends on the number of bulk power regulators that are installed. IBFs are always installed in pairs.
•Bulk power regulators (BPRs). The number of BPRs varies depending on the configuration of the z14 server. For more information about the required number of BPRs, see 2.8.1, “Power and cooling” on page 77.
•The Keyboard and Display tray (newer design for z14), which is in front of the I/O drawer slots, includes the keyboards and the displays that are connected to the Support Elements.
•Up to four PCIe I/O drawers, which install from the top position (location Z22B) to bottom location Z01B).
•An optional overhead power cable feature is shown in Figure 2-1 on page 36. When this feature is ordered, it is present on the Z frame. The top I/O exit cabling feature must also be installed in this case.
2.1.3 z14 cover (door) design
Figure 2-3 shows the standard cover set FC #0160, which includes the rear vectored covers and the new optional thin cover set FC #0161 for space-constrained installations.
Figure 2-3 Cover sets available with z14
2.1.4 Top exit I/O cabling
As with the z13 and zEC12, the z14 server supports the Top Exit I/O Cabling feature (FC 7942). This feature routes all coupling links and I/O cables, including a 1000BASE-T Ethernet cable from I/O drawers or PCIe I/O drawers through four frame extensions, out the top of the frame.
In the z14 server, the Top Exit I/O Cabling feature (FC 7942) is available for radiator-cooled (air-cooled) models and water-cooled models. For more information, see IBM 3906 Installation Manual for Physical Planning, GC28-6965, and 10.3, “Physical planning” on page 398.
2.1.5 PCIe I/O drawer
As shown in Figure 2-4, each PCIe I/O drawer (up to five) has 38 slots each to support the PCIe I/O infrastructure with a bandwidth of 16 GBps and includes the following features:
•A total of 32 I/O cards that are spread over 4 I/O domains.
•Four PCIe switch cards that provide connectivity to the PCIe fanouts that are installed in the processor drawer. Each domain has a dedicated support partition (four per system) to manage the native PCIe card types that are installed in that domain.
•Two Flexible Support Processor (FSP) cards that are used to control the drawer function.
•Redundant N+1 blowers and power supplies.
Figure 2-4 PCIe I/O drawer front and rear view
PCIe I/O infrastructure
The PCIe I/O infrastructure uses the PCIe fanout that is installed in the processor (CPC) drawers to connect to the PCIe I/O drawer, which can include the following features:
•New features FICON Expresss16+ two port card, long wavelength (LX) or short wavelength (SX), which contains two physical channel IDs (PCHIDs)
•FICON Express features (only for a carry-forward MES):
– FICON Express16S two port card, long wavelength (LX) or short wavelength (SX), which contains two PCHIDs
– FICON Express8S two port card, long wavelength (LX) or short wavelength (SX), which contains two PCHIDs
•Open Systems Adapter (OSA)-Express7S 25GbE Short Reach (SR) - New feature
•Open System Adapter (OSA)-Express6S:
– OSA-Express6S 10 Gb Ethernet one port card, Long Reach (LR) or Short Reach (SR), which contains one physical channel ID PCHID
– OSA-Express6S Gb Ethernet two port card, LX or SX, which contains one PCHID
– OSA-Express6S 1000BASE-T Ethernet, two port card, RJ-45, which contains one PCHID
•Open System Adapter (OSA)-Express5S and 4S features (only for a carry-forward MES):
– OSA-Express5S 10 Gb Ethernet (one port card, LR or SR, and one PCHID)
– OSA-Express5S Gb Ethernet (two port card, LX or SX, and one PCHID)
– OSA-Express5S and 4S 1000BASE-T Ethernet (two port card, RJ-45, and one CHPID)
•New feature Crypto Express6S. Each feature holds one PCI Express cryptographic adapter. Each adapter can be configured by the installation as a Secure IBM Common Cryptographic Architecture (CCA) coprocessor, as a Secure IBM Enterprise Public Key Cryptography Standards (PKCS) #11 (EP11) coprocessor, or as an accelerator.
•Crypto Express5S feature (only for a carry-forward MES). Each feature holds one PCI Express cryptographic adapter. Each adapter can be configured by the installation as a Secure IBM Common Cryptographic Architecture (CCA) coprocessor, as a Secure IBM Enterprise Public Key Cryptography Standards (PKCS) #11 (EP11) coprocessor, or as an accelerator.
•New feature zHyperLink Express is a two-port card that is directly connected to DASD storage that is designed to increase the scalability of IBM Z transaction processing and lower I/O latency. This feature is installed in the I/O drawer.
•New feature Coupling Express LR (CE LR) is a two-port feature that is used for long-distance coupling connectivity and uses a new coupling channel type, CL5. It also uses 10GbE RoCE technology and is designed to drive distances up to 10 km (6.21 miles) and support a link data rate of 10 Gigabits per second (Gbps). This feature is installed in the I/O drawer and each port supports up to four CHPIDs per PCHID and point-to-point connectivity only.
•New feature - 25GbE RoCE Express2 - Remote Direct Memory Access (RDMA) over Converged Ethernet. The next generation RoCE with improved virtualization scale, better performance, and RAS. It is a two-port card and supports up to 63 virtual functions per port and 126 VFs per PCHID (feature)
•10GbE RoCE Express2 - Remote Direct Memory Access (RDMA) over Converged Ethernet. The next generation RoCE with improved virtualization scale, better performance, and RAS. It is a two-port card and supports up to 63 virtual functions per port and 126 VFs per PCHID (feature).
•10GbE RoCE Express - Remote Direct Memory Access (RDMA) over Converged Ethernet (only for a carry-forward MES). It is a two-port card and support up to 31 virtual functions per adapter.
•zEnterprise Data Compression (zEDC) Express. The zEnterprise Data Compression Express feature occupies one I/O slot, but it does not have a CHPID type. Up to 15 partitions can share the feature concurrently.
2.2 CPC drawer
The z14 (3906) server continues the design of z13 by packaging processors in drawers. A z14 CPC drawer includes the following features:
•Up to seven single chip modules (SCMs)
•Memory
•Symmetric multiprocessor (SMP) connectivity
•Connectors to support PCIe I/O drawers (through PCIe fanout hubs)
•Coupling links to other CPCs
The CPC drawers are in the A frame. The z14 server can have up to four CPC drawers installed (the minimum is one drawer). A CPC drawer and its components are shown in Figure 2-5.
Figure 2-5 CPC drawer components (top view) of a model M05
The z14 CPC drawer is packaged slightly differently than z13 and contains the following components:
•Two Drawer Sizes with each containing two logical CP clusters:
– 5 PU SCMs + 1 SC SCM / drawer (41 PUs) – for models M01, M02, M03, M04
– 6 PU SCMs + 1 SC SCM / drawer (49 PUs) – for model M05
•PU SCM 14nm SOI technology, 17 layers of metal, core running at 5.2GHz: 10 PUs/SCM (7, 8, 9, 10 active core PNs).
•One System Controller (SC) SCM, with a 672 MB L4 cache.
•Five DDR3 dual in-line module (DIMM) slots per memory controller, for a total of up to 25 DIMMs per drawer.
•DIMMs plugged in to 15, 20 or 25 DIMM slots, providing 640 - 40960 GB of physical memory (includes RAIM) and 512 - 32576 GB of addressable memory in a four-drawer system.
•10 PCIe Gen3 x16 fanouts (16 GBps bandwidth) per CPC Drawer:
– PCIe Gen3 I/O fanout for PCIe I/O Drawer
– ICA SR, PCIe fanout
•Four GX++ slots for IFB fanouts (6 GBps bandwidth): HCA3-O, HCA3-O LR.
•Two flexible service processor (FSP) cards for system control.
•Two DC converter assemblies (DCAs) that provide power to the CPC drawer. The loss of one DCA leaves enough power to satisfy the drawer’s power requirements (n+1 redundancy). The DCAs can be concurrently removed and replaced (one at a time).
•Water-cooling manifold for PU chips.
•Three SMP connectors for inter-drawer connectivity.
Figure 2-6 shows the front view of a CPC drawer, with fanouts slots and connector for water cooling, and the rear view of drawer, with the DIMM slots and DCA connector.
Figure 2-6 Front and rear view of the CPC drawer
Figure 2-7 shows the front view of a fully populated processor drawer. Redundant FSP adapters (2) always are installed, and PCIe I/O fanouts are plugged in specific slots for best performance and availability. SMP interconnect cables are present when multiple drawers are present. Also present are the ICA SR and InfiniBand fanouts for coupling connectivity.
Figure 2-7 Fully populated processor drawer
Figure 2-8 shows the CPC drawer logical structure, component connections (including the CP SCMs), and the storage control (SC) SCMs.
Figure 2-8 CPC drawer logical structure
Memory is connected to the SCMs through memory control units (MCUs). Five MCUs can be placed in a drawer (one per PU SCM) that provide the interface to the controller on memory DIMM. A memory controller drives five DIMM slots.
The buses are organized in the following configurations:
•The PCIe I/O buses provide connectivity for PCIe fanouts and can sustain up to 16 GBps data traffic per bus direction.
•The X-bus provides interconnects between SC chip and PUs chips to each other, in the same node.
•The A-bus provides interconnects between SC chips in different drawers by using SMP cables.
•Processor support interfaces (PSIs) are used to communicate with FSP cards for system control.
2.2.1 CPC drawer interconnect topology
Figure 2-9 shows the point-to-point topology for CPC drawers and nodes communication. Each CPC drawer communicates directly to all of the other processors drawers in the CPC through two links.
Figure 2-9 Drawer to drawer communication
The CPC drawers are in the Frame A and are populated from bottom to top.
The order of CPC drawer installation and position in the Frame A is listed in Table 2-1.
Table 2-1 CPC drawer installation order and position in Frame A
|
CPC drawer
|
CPC drawer 1
|
CPC drawer 2
|
CPC drawer 3
|
CPC drawer 4
|
|
Installation order
|
First
|
Second
|
Third
|
Fourth
|
|
Position in Frame A
|
A15A
|
A19A
|
A23A
|
A27A
|
CPC drawer installation is concurrent, except for the upgrade to the model M05, which can be obtained only from manufacturing with the model M05 processor drawers that are installed. Concurrent drawer repair requires a minimum of two drawers.
2.2.2 Oscillator
The z14 server has two oscillator cards (OSCs): One primary and one backup. If the primary OSC fails, the secondary detects the failure, takes over transparently, and continues to provide the clock signal to the CPC. The two oscillators have Bayonet Neill-Concelman (BNC) connectors that provide pulse per second signal (PPS) synchronization to an external time source with PPS output.
The SEs provide the Simple Network Time Protocol (SNTP) client function. When Server Time Protocol (STP) is used, the time of an STP-only Coordinated Timing Network (CTN) can be synchronized to the time that is provided by a Network Time Protocol (NTP) server. This configuration allows time-of-day (TOD) synchronization in a heterogeneous platform environment.
The accuracy of an STP-only CTN is improved by using an NTP server with the PPS output signal as the External Time Source (ETS). NTP server devices with PPS output are available from several vendors that offer network timing solutions. A cable connection from the PPS port on the OSC to the PPS output of the NTP server is required when z14 uses STP and is configured in an STP-only CTN that uses NTP with PPS as the external time source. The z14 server cannot participate in a mixed CTN; it can participate only in an STP-only CTN.
STP tracks the highly stable and accurate PPS signal from the NTP server and maintains an accuracy of 10 µs to the ETS, as measured at the PPS input of the z14 server.
If STP uses an NTP server without PPS, a time accuracy of 100 ms to the ETS is maintained.
Although not part of the CPC drawer design, the OSCs cards are next to the CPC drawers and connected to the same backplane to which the drawers are connected. All four drawers connect to the OSC backplane.
Figure 2-10 shows the location of the two OSC cards with BNC connectors for PPS on the CPC, which is next to the drawer 2 and drawer 3 locations.
Figure 2-10 Oscillators cards
|
Tip: STP is available as FC 1021. It is implemented in the Licensed Internal Code (LIC), and allows multiple servers to maintain time synchronization with each other and synchronization to an ETS. For more information, see the following publications:
•Server Time Protocol Planning Guide, SG24-7280
•Server Time Protocol Implementation Guide, SG24-7281
•Server Time Protocol Recovery Guide, SG24-7380
|
2.2.3 System control
The various system elements are managed through the FSPs. An FSP is based on the IBM PowerPC® microprocessor technology. Each FSP card has two ports to connect to two internal Ethernet LANs, through system control hubs (SCH1 and SCH2). The FSPs communicate with the SEs and provide a subsystem interface (SSI) for controlling components.
Figure 2-11 shows a conceptual overview of the system control design.
Figure 2-11 Conceptual overview of system control elements
|
Note: The maximum configuration has four CPC drawers and five PCIe or I/O drawers for z14. The various supported FSP connections are referenced in Figure 2-11.
|
A typical FSP operation is to control a power supply. An SE sends a command to the FSP to start the power supply. The FSP (by using SSI connections) cycles the various components of the power supply, monitors the success of each step and the resulting voltages, and reports this status to the SE.
Most system elements are duplexed (n+1), and each element has at least one FSP. Two internal Ethernet LANs and two SEs, for redundancy, and crossover capability between the LANs, are available so that both SEs can operate on both LANs.
The Hardware Management Consoles (HMCs) and SEs are connected directly to one or two Ethernet Customer LANs. One or more HMCs can be used.
2.2.4 CPC drawer power
Each CPC drawer gets its power from two DCAs. The DCAs provide the required power for the drawer in an n+1 configuration. Loss of one DCA leaves enough power to meet power requirements for the entire drawer. The DCAs can be concurrently serviced and are accessed from the rear of the frame A.
Figure 2-12 shows the location of DCAs on the backplane of the A frame.
Figure 2-12 DCA power supplies
2.3 Single chip modules
The single chip module (SCM) is a multi-layer metal substrate module that holds one PU chip or an SC chip. Both CP and SC chip size is 696 mm2 (25.3 mm x 27.5 mm). Each CPC drawer has six (model M01-M04) or seven SCMs (model M05), five or six PU SCMs (6.1 billion transistors each), and one SC SCM (9.7 billion transistors).
The two types of SCMs (PU and SC) are shown in Figure 2-13.
Figure 2-13 Single chip modules (PU SCM and SC SCM)
PU and SC chips use CMOS 14 nm process, 17 layers of metal, and state-of-the-art Silicon-On-Insulator (SOI) technology.
The SCMs are plugged into a socket that is part of the CPC drawer packaging. The interconnectivity between the CPC drawers is accomplished through SMP connectors and cables. Three inter-drawer connections are available on each CPC drawer. This configuration allows a multidrawer system to act as a symmetric multiprocessor (SMP) system.
Figure 2-14 shows the logical cluster structure for a model M05, with the 6 PU SCMs, the SC SCM, and their connections. For models M01-M04, the PU SCM to the far left in Logical Cluster 1 is not present. The number of cores on each of the five or six populated PU SCMs can range 7 - 10 cores.
Figure 2-14 Logical drawer structure
2.3.1 Processor unit chip
The z14 PU chip (installed as a PU SCM) is an evolution of the z13 core design. It includes the following features:
•CMOS 14nm SOI technology
•Out-of-order instruction processing
•Pipeline enhancements, dynamic improved simultaneous multithreading (SMT), enhanced single-instruction multiple-data (SIMD), and redesigned, larger on-chip caches
Each PU chip includes up to 10 active cores that run at 5.2 GHz, which means that the cycle time is 0.192 ns. The PU chips come in four versions: 7, 8, 9, or 10 active cores. For models M01, M02, M03, and M04, the processor units in each drawer are implemented with 41 active cores per drawer. This configuration means that model M01 has 41, model M02 has 82, mode, M03 has 123, and model M04 has 164 active cores.
Model M05 has 49 active cores per drawer. This configuration provides 196 active cores on model M05. A schematic representation of the PU chip is shown in Figure 2-15.
Figure 2-15 PU SCM floor plan
The PU chip contains the following enhancements:
•Cache Improvements:
– New power efficient logical directory design
– 33% larger L1 I$ (128 KB), private
– 2x larger L2 D$ (4 MB), private
– 2x larger L3 cache with symbol ECC, shared
•New Translation/TLB2 design:
– Four concurrent translations
– Reduced latency
– Lookup that is integrated into L2 access pipe
– 2x CRSTE growth
– 1.5x PTE growth
– New 64 entry 2 GB TLB2
•Pipeline Optimizations:
– Improved instruction delivery
– Faster branch wake-up
– Reduced execution latency
– Improved OSC avoidance
– Optimized second-generation SMT2
•Better Branch Prediction:
– 33% Larger BTB1 & BTB2
– New Perceptron Predictor
– New Simple Call Return Stack
2.3.2 Processor unit (core)
Each processor unit (see Figure 2-16) or core is a superscalar and out-of-order processor that includes 10 execution units.
Figure 2-16 Core layout
Consider the following points:
•Fixed-point unit (FXU): The FXU handles fixed-point arithmetic.
•Load-store unit (LSU): The LSU contains the data cache. It is responsible for handling all types of operand accesses of all lengths, modes, and formats as defined in the z/Architecture.
•Instruction fetch and branch (IFB) (prediction) and Instruction cache and merge (ICM). The IFB and ICM sub units contain the instruction cache, branch prediction logic, instruction fetching controls, and buffers. Its relative size is the result of the elaborate branch prediction.
•Instruction decode unit (IDU): The IDU is fed from the IFU buffers, and is responsible for parsing and decoding of all z/Architecture operation codes.
•Translation unit (XU): The XU has a large translation lookaside buffer (TLB) and the Dynamic Address Translation (DAT) function that handles the dynamic translation of logical to physical addresses.
•Instruction sequence unit (ISU): This unit enables the out-of-order (OoO) pipeline. It tracks register names, OoO instruction dependency, and handling of instruction resource dispatch.
•Recovery unit (RU): The RU keeps a copy of the complete state of the system that includes all registers, collects hardware fault signals, and manages the hardware recovery actions.
•Dedicated Co-Processor (COP): The dedicated coprocessor is responsible for data compression and encryption functions for each core.
•Core pervasive unit (PC) for instrumentation, error collection.
•Vector and Floating point Units (VFU).
•Binary floating-point unit (BFU): The BFU handles all binary and hexadecimal floating-point and fixed-point multiplication operations.
•Decimal floating-point unit (DFU): The DU runs floating-point, decimal fixed-point, and fixed-point division operations.
•Vector execution unit (VXU).
•Level 2 cache (L2) Instructions and data (L2I/L2D).
2.3.3 PU characterization
In each CPC drawer, PUs can be characterized for client use. The characterized PUs can be used in general to run supported operating systems, such as z/OS, z/VM, and Linux on z Systems. They also can run specific workloads, such as Java, XML services, IPSec, and some Db2 workloads, or functions, such as Coupling Facility Control Code (CFCC). For more information about PU characterization, see 3.5, “Processor unit functions” on page 109.
The maximum number of characterized PUs depends on the z14 model. Some PUs are characterized by the system as standard system assist processors (SAPs) to run the I/O processing. By default, least two spare PUs per system are available that are used to assume the function of a failed PU. The remaining installed PUs can be characterized for client use. A z14 model nomenclature includes a number that represents the maximum number of PUs that can be characterized for client use, as listed in Table 2-2.
Table 2-2 Number of PUs per z14 model
|
Model
|
CPs
|
IFLs
|
Unassigned IFLs
|
zIIP
|
ICFs
|
IFPs
|
Standard SAPs
|
Additional SAPs
|
Spares
|
|
|
M01
|
0 - 33
|
0 - 33
|
0 - 32
|
0 - 22
|
0 - 33
|
1
|
5
|
0 - 4
|
2
|
|
|
M02
|
0 - 69
|
0 - 69
|
0 - 68
|
0 - 46
|
0 - 69
|
1
|
10
|
0 - 8
|
2
|
|
|
M03
|
0 - 105
|
0 - 105
|
0 - 104
|
0 - 70
|
0 - 105
|
1
|
15
|
0 - 12
|
2
|
|
|
M04
|
0 - 141
|
0 - 141
|
0 - 140
|
0 - 94
|
0 - 141
|
1
|
20
|
0 - 16
|
2
|
|
|
M05
|
0 - 170
|
0 - 170
|
0 - 169
|
0 - 112
|
0 - 170
|
1
|
23
|
0 - 16
|
2
|
|
2.3.4 System Controller chip
The System Controller (SC) chip uses the CMOS 14nm SOI technology, with 17 layers of metal. It measures 25.3 x 27.5 mm, and has 9.7 billion transistors. Each node of the CPC drawer has one SC chip.
Figure 2-17 on page 53 shows a schematic representation of the SC chip. Consider the following points:
•X-Bus (CP-CP and CP-SC): Significant changes to allow SC to fit more X-Bus connections
•A Bus (SC-SC off drawer): Minor changes to reflect protocol improvements and new system topology
•672 MB shared eDRAM L4 Cache
•L4 Directory is built with eDRAM
•New L4 Cache Management:
– Ratio of L3 to L4 cache capacity is increasing
– New on-drawer Cluster-to-Cluster (topology change) management
Figure 2-17 SC chip layout
2.3.5 Cache level structure
The cache structure comparison between z13 and z14 is shown in Figure 2-18.
Figure 2-18 Cache structure comparison: z14 versus z13
2.4 Memory
The maximum physical memory size is directly related to the number of CPC drawers in the system. Each CPC drawer can contain up to 8000 GB of physical memory, for a total of 8000 GB + (3 x 8192 GB) = 32576 GB, which is approximately 32 TB of installed (physical) memory per system.
The maximum and minimum memory sizes that you can order for each z14 model are listed in Table 2-3.
Table 2-3 z14 memory sizes
|
Model
|
Number of CPC drawers
|
Customer memory (GB)
|
|
M01
|
1
|
256 - 8000
|
|
M02
|
2
|
256 - 16192
|
|
M03
|
3
|
256 - 24384
|
|
M04
|
4
|
256 - 32576
|
|
M05
|
4
|
256 - 32576
|
The minimum physical installed memory is 512 GB per CPC drawer. The minimum initial amount of memory that can be ordered is 256 GB for all z14 models. The maximum customer memory size is based on the physical installed memory minus the RAIM and minus the hardware system area (HSA) memory, which has a fixed amount of 192 GB.
The memory granularity, which is based on the installed customer memory, is listed in Table 2-4.
Table 2-4 Memory ordering granularity
|
Granularity (GB)
|
Customer memory (GB)
|
|
64
|
256 - 576
|
|
128
|
704 - 1600
|
|
256
|
1856 - 3904
|
|
512
|
4416 - 32576
|
With the z14, the memory granularity varies from 64 GB (for customer memory sizes 256 - 576 GB) up to 512 GB (for CPCs having 4416 - 32576 GB of customer memory).
2.4.1 Memory subsystem topology
The z14 memory subsystem uses high-speed, differential-ended communications memory channels to link a host memory to the main memory storage devices.
Figure 2-19 shows an overview of the CPC drawer memory topology of a z14 server.
Figure 2-19 CPC drawer memory topology
Each CPC drawer includes 15, 20, or 25 DIMMs. DIMMs are connected to each PU chip through the memory control units (MCU). Each PU SCM has one MCU, which uses five channels (one for each DIMM and one for RAIM implementation) in a 4 +1 design. Each CPC drawer can have three, four, or five populated MCUs.
DIMMs are used in 32, 64, 128, 256, and 512 GB sizes with five DIMMs of the same size included in a memory feature. (160, 320, 640, 1280, and 2560 GB RAIM array size).
2.4.2 Redundant array of independent memory
The z14 server uses the RAIM technology. The RAIM design detects and recovers from failures of dynamic random access memory (DRAM), sockets, memory channels, or DIMMs.
The RAIM design requires the addition of one memory channel that is dedicated for reliability, availability, and serviceability (RAS), as shown in Figure 2-20.
Figure 2-20 RAIM configuration per logical cluster
The fifth channel in each MCU enables memory to be implemented as a Redundant Array of Independent Memory (RAIM). This technology has significant error detection and correction capabilities. Bit, lane, DRAM, DIMM, socket, and complete memory channel failures can be detected and corrected, including many types of multiple failures. Therefore, RAIM takes 20% of DIMM capacity (a non-RAIM option is not available).
The RAIM design provides the following layers of memory recovery:
•ECC with 90B/64B Reed Solomon code.
•DRAM failure, with marking technology in which two DRAMs can be marked and no half sparing is needed. A call for replacement occurs on the third DRAM failure.
•Lane failure with CRC retry, data-lane sparing, and clock-RAIM with lane sparing.
•DIMM failure with CRC retry, data-lane sparing, and clock-RAIM with lane sparing.
•DIMM controller ASIC failure.
•Channel failure.
2.4.3 Memory configurations
Memory sizes in each CPC drawer do not have to be similar. Different CPC drawers can contain different amounts of memory. The 15 drawer configurations that support memory are listed in Table 2-5. Total physical memory includes RAIM (20%) and HSA (192 GB).
Table 2-5 Physically installed memory drawer configurations
|
|
Physical
memory
|
32 GB
#DIMMs
|
64 GB
#DIMMs
|
128 GB
#DIMMs
|
256 GB
#DIMMs
|
512 GB
#DIMMs
|
-RAIM
GB
|
-HSA
GB
|
HDW
dialed
increments
|
|
1
|
640
|
10
|
5
|
0
|
0
|
0
|
512
|
320
|
|
|
2
|
960
|
0
|
15
|
0
|
0
|
0
|
768
|
576
|
256
|
|
3
|
1280
|
0
|
10
|
5
|
0
|
0
|
1024
|
832
|
256
|
|
4
|
1600
|
0
|
5
|
10
|
0
|
0
|
1280
|
1088
|
256
|
|
5
|
1920
|
0
|
0
|
15
|
0
|
0
|
1536
|
1344
|
256
|
|
6
|
2240
|
0
|
5
|
15
|
0
|
0
|
1792
|
1600
|
256
|
|
7
|
2560
|
0
|
0
|
20
|
0
|
0
|
2048
|
1856
|
256
|
|
8
|
3200
|
0
|
0
|
25
|
0
|
0
|
2560
|
2368
|
512
|
|
9
|
3840
|
0
|
0
|
20
|
5
|
0
|
3072
|
2880
|
512
|
|
10
|
4480
|
0
|
0
|
15
|
10
|
0
|
3584
|
3392
|
512
|
|
11
|
5120
|
0
|
0
|
10
|
15
|
0
|
4096
|
3904
|
512
|
|
12
|
6400
|
0
|
0
|
0
|
25
|
0
|
5120
|
4928
|
1024
|
|
13
|
7680
|
0
|
0
|
0
|
20
|
5
|
6144
|
5952
|
1024
|
|
14
|
8960
|
0
|
0
|
0
|
15
|
10
|
7168
|
6976
|
1024
|
|
15
|
10240
|
0
|
0
|
0
|
10
|
15
|
8192
|
8000
|
1024
|
•A CPC drawer always contains a minimum of 10 32 GB DIMMs and 5 64 GB DIMMS (640 GB) as listed in drawer configuration number 1 in Table 2-5.
•A CPC drawer can have more memory that is installed than enabled. The amount of memory that can be enabled by the client is the total physically installed memory minus the RAIM amount (20%) and minus the 192 GB of HSA memory.
•A CPC drawer can have available unused memory, which can be ordered as a memory upgrade and enabled by LIC without DIMM changes.
•DIMM changes require a disruptive power-on reset (POR) on z14 model M01. DIMM changes are always done concurrently on z14 models with multiple CPC drawers that use Enhanced Drawer Availability (EDA).
DIMM plugging for the configurations in each CPC drawer do not have to be similar. Each memory 5 slot DIMM bank must have the same DIMM size; however, a drawer can have a mix of DIMM banks. Table 2-6 on page 58 lists the memory population by DIMM bank for the 15 configurations that are listed in Table 2-5.
Table 2-6 Memory population by DIMM Bank (one CPC drawer)
|
|
MD01-
MD05 |
MD06-
MD10 |
MD11-
MD15 |
MD16-
MD20 |
MD21-
MD25 |
MD26-
MD30 |
Total
-RAIM
|
|
1
|
|
|
64
|
32
|
32
|
|
512
|
|
2
|
|
|
64
|
64
|
64
|
|
768
|
|
3
|
|
|
10
|
64
|
64
|
|
1024
|
|
4
|
|
|
128
|
128
|
64
|
|
1280
|
|
5
|
|
|
128
|
128
|
128
|
|
1536
|
|
6
|
|
64
|
128
|
128
|
128
|
|
1792
|
|
7
|
|
128
|
128
|
128
|
128
|
|
2048
|
|
8
|
|
128
|
128
|
128
|
128
|
128
|
2560
|
|
9
|
|
256
|
128
|
128
|
128
|
128
|
3072
|
|
10
|
|
256
|
128
|
128
|
256
|
128
|
3584
|
|
11
|
|
256
|
256
|
128
|
256
|
128
|
4096
|
|
12
|
|
256
|
256
|
256
|
256
|
256
|
5120
|
|
13
|
|
512
|
256
|
256
|
256
|
256
|
6144
|
|
14
|
|
512
|
256
|
256
|
512
|
256
|
7168
|
|
15
|
|
512
|
512
|
256
|
512
|
256
|
8192
|
•The support element View Hardware Configuration task can be used to determine the size and quantity of the memory plugged in each drawer. Figure 2-21 shows an example of the location and description of the installed memory modules.
Figure 2-21 View Hardware Configuration task on the Support Element
Figure 2-22 shows the CPC drawer and DIMM locations for a z14.
Figure 2-22 CPC drawer and DIMM locations
Table 2-7 lists the physical memory DIMM plugging configurations by feature code from manufacturing when the system is ordered. The drawer columns for the specific model contain the memory configuration number for the specific drawer. Use available unused memory that can be enabled by LIC, when required. Consider the following points:
•If more storage is ordered by using other feature codes, such as Virtual Flash Memory, Flexible Memory, or Preplanned memory, the extra storage is installed and plugged as necessary.
Table 2-7 System memory configurations
|
|
Inc
|
Memory Inc
GB
|
M01
|
|
M02
|
|
M03
|
|
M04 / M05
|
||||||||||
|
FC
|
DRW
1
|
Dial Max
|
DRW 1
|
DRW 2
|
Dial Max
|
DRW 1
|
DRW 2
|
DRW3
|
Dial Max
|
DRW 1
|
DRW 2
|
DRW 3
|
DRW 4
|
Dial Max
|
|||||
|
1660
|
64
|
256
|
1
|
320
|
|
1
|
1
|
832
|
|
1
|
1
|
1
|
1344
|
|
1
|
1
|
1
|
1
|
1856
|
|
1661
|
320
|
1
|
320
|
1
|
1
|
832
|
|
1
|
1
|
1
|
1344
|
1
|
1
|
1
|
1
|
1856
|
|||
|
1662
|
384
|
2
|
576
|
1
|
1
|
832
|
|
1
|
1
|
1
|
1344
|
1
|
1
|
1
|
1
|
1856
|
|||
|
1663
|
448
|
2
|
576
|
1
|
1
|
832
|
|
1
|
1
|
1
|
1344
|
1
|
1
|
1
|
1
|
1856
|
|||
|
1664
|
512
|
2
|
576
|
1
|
1
|
832
|
|
1
|
1
|
1
|
1344
|
1
|
1
|
1
|
1
|
1856
|
|||
|
1665
|
576
|
2
|
576
|
1
|
1
|
832
|
|
1
|
1
|
1
|
1344
|
1
|
1
|
1
|
1
|
1856
|
|||
|
1666
|
|
704
|
3
|
832
|
1
|
1
|
832
|
|
1
|
1
|
1
|
1344
|
1
|
1
|
1
|
1
|
1856
|
||
|
1667
|
|
832
|
3
|
832
|
1
|
1
|
832
|
|
1
|
1
|
1
|
1344
|
1
|
1
|
1
|
1
|
1856
|
||
|
1668
|
|
960
|
4
|
1088
|
1
|
2
|
1088
|
|
1
|
1
|
1
|
1344
|
1
|
1
|
1
|
1
|
1856
|
||
|
1669
|
|
1088
|
4
|
1088
|
1
|
2
|
1088
|
|
1
|
1
|
1
|
1344
|
1
|
1
|
1
|
1
|
1856
|
||
|
1670
|
|
1216
|
5
|
1344
|
1
|
2
|
1344
|
|
1
|
1
|
1
|
1344
|
1
|
1
|
1
|
1
|
1856
|
||
|
1671
|
|
1344
|
5
|
1344
|
|
2
|
2
|
1344
|
|
1
|
1
|
1
|
1344
|
|
1
|
1
|
1
|
1
|
1856
|
|
1672
|
|
1472
|
6
|
1600
|
2
|
3
|
1600
|
1
|
1
|
2
|
1600
|
1
|
1
|
1
|
1
|
1856
|
|||
|
1673
|
|
1600
|
6
|
1600
|
2
|
3
|
1600
|
1
|
1
|
2
|
1600
|
1
|
1
|
1
|
1
|
1856
|
|||
|
1674
|
|
1856
|
7
|
1856
|
3
|
3
|
1856
|
2
|
1
|
2
|
1856
|
1
|
1
|
1
|
1
|
1856
|
|||
|
1675
|
|
2112
|
8
|
2368
|
3
|
4
|
2112
|
2
|
2
|
2
|
2112
|
1
|
1
|
1
|
2
|
2112
|
|||
|
1676
|
|
2368
|
8
|
2368
|
4
|
4
|
2368
|
2
|
2
|
3
|
2368
|
2
|
1
|
1
|
2
|
2368
|
|||
|
1677
|
|
2624
|
9
|
2880
|
4
|
5
|
2624
|
3
|
2
|
3
|
2624
|
2
|
1
|
2
|
2
|
2624
|
|||
|
1678
|
|
2880
|
9
|
2880
|
5
|
5
|
2880
|
3
|
3
|
3
|
2880
|
2
|
2
|
2
|
2
|
2880
|
|||
|
1679
|
|
3136
|
10
|
3392
|
5
|
6
|
3136
|
3
|
3
|
4
|
3136
|
2
|
2
|
2
|
3
|
3136
|
|||
|
1680
|
|
3392
|
10
|
3392
|
6
|
6
|
3392
|
4
|
3
|
4
|
3392
|
3
|
2
|
2
|
3
|
3392
|
|||
|
1681
|
|
3648
|
11
|
3904
|
|
6
|
7
|
3648
|
|
4
|
4
|
4
|
3648
|
|
3
|
2
|
3
|
3
|
3648
|
|
1682
|
|
3904
|
11
|
3904
|
|
7
|
7
|
3904
|
|
4
|
4
|
5
|
3904
|
|
3
|
3
|
3
|
3
|
3904
|
|
1683
|
|
4416
|
12
|
4928
|
|
7
|
8
|
4416
|
|
5
|
5
|
5
|
4416
|
|
4
|
3
|
3
|
4
|
4416
|
|
1684
|
|
4928
|
12
|
4928
|
|
8
|
8
|
4928
|
|
6
|
5
|
6
|
4928
|
|
4
|
4
|
4
|
4
|
4928
|
|
1685
|
|
5440
|
13
|
5952
|
|
8
|
9
|
5440
|
|
6
|
6
|
7
|
5440
|
|
5
|
4
|
4
|
5
|
5440
|
|
1686
|
|
5952
|
13
|
5952
|
|
9
|
9
|
5952
|
|
7
|
7
|
7
|
5952
|
|
5
|
5
|
5
|
5
|
5952
|
|
1687
|
|
6464
|
14
|
6976
|
|
9
|
10
|
6464
|
|
7
|
7
|
8
|
6464
|
|
6
|
5
|
5
|
6
|
6464
|
|
1688
|
|
6976
|
14
|
6976
|
|
10
|
10
|
6976
|
|
8
|
7
|
8
|
6976
|
|
6
|
6
|
6
|
6
|
6976
|
|
1689
|
|
7488
|
15
|
8000
|
|
10
|
11
|
7488
|
|
8
|
8
|
8
|
7488
|
|
7
|
6
|
6
|
7
|
7488
|
|
1690
|
|
8000
|
15
|
8000
|
|
11
|
11
|
8000
|
|
8
|
8
|
9
|
8000
|
|
7
|
7
|
7
|
7
|
8000
|
|
1691
|
|
8512
|
|
|
|
11
|
12
|
9024
|
|
9
|
8
|
9
|
8512
|
|
7
|
7
|
7
|
8
|
8512
|
|
1692
|
|
9024
|
|
|
|
11
|
12
|
9024
|
|
9
|
9
|
9
|
9024
|
|
8
|
7
|
7
|
8
|
9024
|
|
1693
|
|
9536
|
|
|
|
12
|
12
|
10048
|
|
9
|
9
|
10
|
9536
|
|
8
|
7
|
8
|
8
|
9536
|
|
1694
|
|
10048
|
|
|
|
12
|
12
|
10048
|
|
10
|
9
|
10
|
10048
|
|
8
|
8
|
8
|
8
|
10048
|
|
1695
|
|
10560
|
|
|
|
12
|
13
|
11072
|
|
10
|
10
|
10
|
10560
|
|
8
|
8
|
8
|
9
|
10560
|
|
1696
|
|
11072
|
|
|
|
12
|
13
|
11072
|
|
10
|
10
|
11
|
11072
|
|
9
|
8
|
8
|
9
|
11072
|
|
1697
|
|
11584
|
|
|
|
13
|
13
|
12096
|
|
11
|
10
|
11
|
11584
|
|
9
|
8
|
9
|
9
|
11584
|
|
1698
|
|
12096
|
|
|
|
13
|
13
|
12096
|
|
11
|
11
|
11
|
12096
|
|
9
|
9
|
9
|
9
|
12096
|
|
1699
|
|
12608
|
|
|
|
13
|
14
|
13120
|
|
11
|
11
|
12
|
13120
|
|
9
|
9
|
9
|
10
|
12608
|
|
1700
|
|
13120
|
|
|
|
13
|
14
|
13120
|
|
11
|
11
|
12
|
13120
|
|
10
|
9
|
9
|
10
|
13120
|
|
1701
|
|
13632
|
|
|
|
14
|
14
|
14144
|
|
12
|
11
|
12
|
14144
|
|
10
|
9
|
10
|
10
|
13632
|
|
1702
|
|
14144
|
|
|
|
14
|
14
|
14144
|
|
12
|
11
|
12
|
14144
|
|
10
|
10
|
10
|
10
|
14144
|
|
1703
|
|
14656
|
|
|
|
14
|
15
|
15168
|
|
12
|
12
|
12
|
15168
|
|
10
|
10
|
10
|
11
|
14656
|
|
1704
|
|
15168
|
|
|
|
14
|
15
|
15168
|
|
12
|
12
|
12
|
15168
|
|
11
|
10
|
10
|
11
|
15168
|
|
1705
|
|
15680
|
|
|
|
15
|
15
|
16192
|
|
12
|
12
|
13
|
16192
|
|
11
|
10
|
11
|
11
|
15680
|
|
1706
|
|
16192
|
|
|
|
15
|
15
|
16192
|
|
12
|
12
|
13
|
16192
|
|
11
|
11
|
11
|
11
|
16192
|
|
1707
|
|
16704
|
|
|
|
|
|
13
|
12
|
13
|
17216
|
|
11
|
11
|
11
|
12
|
17216
|
||
|
1708
|
|
17216
|
|
|
|
|
|
13
|
12
|
13
|
17216
|
|
11
|
11
|
11
|
12
|
17216
|
||
|
1709
|
|
17728
|
|
|
|
|
|
13
|
13
|
13
|
18240
|
|
12
|
11
|
11
|
12
|
18240
|
||
|
1710
|
|
18240
|
|
|
|
|
|
13
|
13
|
13
|
18240
|
|
12
|
11
|
11
|
12
|
18240
|
||
|
1711
|
|
18752
|
|
|
|
|
|
13
|
13
|
14
|
19264
|
|
12
|
11
|
12
|
12
|
19264
|
||
|
1712
|
|
19264
|
|
|
|
|
|
13
|
13
|
14
|
19264
|
|
12
|
11
|
12
|
12
|
19264
|
||
|
1713
|
|
19776
|
|
|
|
|
|
14
|
13
|
14
|
20288
|
|
12
|
12
|
12
|
12
|
20288
|
||
|
1714
|
|
20288
|
|
|
|
|
|
14
|
13
|
14
|
20288
|
|
12
|
12
|
12
|
12
|
20288
|
||
|
1715
|
|
20800
|
|
|
|
|
|
14
|
14
|
14
|
21312
|
|
12
|
12
|
12
|
13
|
21312
|
||
|
1716
|
|
21312
|
|
|
|
|
|
|
|
14
|
14
|
14
|
21312
|
|
13
|
12
|
12
|
13
|
21312
|
|
1717
|
|
21824
|
|
|
|
|
|
|
|
14
|
14
|
15
|
22336
|
|
13
|
12
|
12
|
13
|
22336
|
|
1718
|
|
22336
|
|
|
|
|
|
|
|
14
|
14
|
15
|
22336
|
|
13
|
12
|
12
|
13
|
22336
|
|
1719
|
|
22848
|
|
|
|
|
|
|
|
15
|
14
|
15
|
23360
|
|
13
|
12
|
13
|
13
|
23360
|
|
1720
|
|
23360
|
|
|
|
|
|
|
|
15
|
14
|
15
|
23360
|
|
13
|
12
|
13
|
13
|
23360
|
|
1721
|
|
23872
|
|
|
|
|
|
|
|
15
|
15
|
15
|
24384
|
|
13
|
13
|
13
|
13
|
24384
|
|
1722
|
|
24384
|
|
|
|
|
|
|
|
15
|
15
|
15
|
24384
|
|
13
|
13
|
13
|
13
|
24384
|
|
1723
|
|
24896
|
|
|
|
|
|
|
|
|
|
|
|
|
13
|
13
|
13
|
14
|
25408
|
|
1724
|
|
25408
|
|
|
|
|
|
|
|
|
|
|
|
|
13
|
13
|
13
|
14
|
25408
|
|
1725
|
|
25920
|
|
|
|
|
|
|
|
|
|
|
|
|
14
|
13
|
13
|
14
|
26432
|
|
1726
|
|
26432
|
|
|
|
|
|
|
|
|
|
|
|
|
14
|
13
|
13
|
14
|
26432
|
|
1727
|
|
26944
|
|
|
|
|
|
|
|
|
|
|
|
|
14
|
13
|
14
|
14
|
27456
|
|
1728
|
|
27456
|
|
|
|
|
|
|
|
|
|
|
|
|
14
|
13
|
14
|
14
|
27456
|
|
1729
|
|
27968
|
|
|
|
|
|
|
|
|
|
|
|
|
14
|
14
|
14
|
14
|
28480
|
|
1730
|
|
28480
|
|
|
|
|
|
|
|
|
|
|
|
|
14
|
14
|
14
|
14
|
28480
|
|
1731
|
|
28992
|
|
|
|
|
|
|
|
|
|
|
|
|
14
|
14
|
14
|
15
|
29504
|
|
1732
|
|
29504
|
|
|
|
|
|
|
|
|
|
|
|
|
14
|
14
|
14
|
15
|
29504
|
|
1733
|
|
30016
|
|
|
|
|
|
|
|
|
|
|
|
|
15
|
14
|
14
|
15
|
30528
|
|
1734
|
|
30528
|
|
|
|
|
|
|
|
|
|
|
|
|
15
|
14
|
14
|
15
|
30528
|
|
1735
|
|
31040
|
|
|
|
|
|
|
|
|
|
|
|
|
15
|
14
|
15
|
15
|
31552
|
|
1736
|
|
31552
|
|
|
|
|
|
|
|
|
|
|
|
|
15
|
14
|
15
|
15
|
31552
|
|
1737
|
|
32064
|
|
|
|
|
|
|
|
|
|
|
|
|
15
|
15
|
15
|
15
|
32576
|
|
1738
|
|
32576
|
|
|
|
|
|
|
|
|
|
|
|
|
15
|
15
|
15
|
15
|
32576
|
2.4.4 Memory upgrades
Memory upgrades can be ordered and enabled by LIC, by upgrading the DIMM cards, by adding DIMM cards, or by adding a CPC drawer.
For a model upgrade that results in the addition of a CPC drawer, the minimum memory increment is added to the system. Each CPC drawer has a minimum physical memory size of 320 GB.
During a model upgrade, adding a CPC drawer is a concurrent operation. Adding physical memory to the added drawer is also concurrent. If all or part of the added memory is enabled for use, it might become available to an active LPAR if the partition includes defined reserved storage. (For more information, see 3.7.3, “Reserved storage” on page 134.) Alternatively, the added memory can be used by an already-defined LPAR that is activated after the memory addition.
|
Note: Memory downgrades are always disruptive. Model downgrades (removal of a CPC drawer) are not supported.
|
2.4.5 Drawer replacement and memory
With EDA, which is supported for z14 servers, sufficient resources must be available to accommodate resources that are rendered unavailable when a CPC drawer is removed for upgrade or repair. For more information, see 2.6.2, “Enhanced drawer availability” on page 69.
Removing a CPC drawer often results in removing active memory. With the flexible memory option, removing the affected memory and reallocating its use elsewhere in the system are possible. For more information, see 2.4.7, “Flexible Memory Option” on page 63. This process requires more available memory to compensate for the memory that is lost with the removal of the drawer.
2.4.6 Virtual Flash Memory
IBM Virtual Flash Memory (VFM, FC 0604) replaces the Flash Express features (#0402,#0403) that were available on the IBM zEC12 and IBM z13. No application changes are required to change from IBM Flash Express to VFM.
VFM is designed to help improve availability and handling of paging workload spikes when z/OS V2.1, V2.2, or V2.3, or on z/OS V1.131 is run. With this support, z/OS is designed to help improve system availability and responsiveness by using VFM across transitional workload events, such as market openings and diagnostic data collection. z/OS is also designed to help improve processor performance by supporting middleware use of pageable large (1 MB) pages.
VFM can also be used in coupling facility images to provide extended capacity and availability for workloads that are use IBM WebSphere MQ Shared Queues structures. The use of VFM can help availability by reducing latency from paging delays that can occur at the start of the workday or during other transitional periods. It is also designed to help eliminate delays that can occur when collecting diagnostic data during failures.
VFM can help organizations meet their most demanding service level agreements and compete more effectively. VFM is designed to be easy to configure, and provide rapid time to value.
2.4.7 Flexible Memory Option
With the Flexible Memory Option, more physical memory is supplied to support the activation of the actual purchased memory entitlement in a single CPC drawer that is out of service during activation (POR), or in a scheduled concurrent drawer upgrade (memory add) or drawer maintenance (n+1 repair) with the use of enhanced drawer availability.
When you order memory, you can request extra flexible memory. The extra physical memory, if required, is calculated by the configurator and priced accordingly.
Flexible memory is available on the M02, M03, M04, and M05 models only. The flexible memory sizes that are available for the z14 are listed in Table 2-8.
Table 2-8 z14 (customer usable) memory sizes
|
Model
|
Standard memory (GB)
|
Flexible memory (GB)
|
|
M01
|
320 - 8000
|
N/A
|
|
M02
|
320 - 16192
|
320 - 8000
|
|
M03
|
320 - 24384
|
320 - 16192
|
|
M04
|
320 - 32576
|
320 - 24384
|
|
M05
|
320 - 32576
|
320 - 24384
|
Consider the following points:
•Standard Memory: Provides minimum physical memory that is required to hold customer purchase memory plus 192 GB HSA.
•Flexible Memory: Provides more physical memory that is needed to support activation base customer memory and HSA on a multiple CPC drawer z14 with one drawer out of service.
|
Note: Although flexible memory can be purchased, it cannot be used for normal everyday use. For that reason, a different purchase price for flexible memory is offered to increase the overall availability of the system.
|
2.4.8 Pre-planned memory
Pre-planned memory provides the capability for concurrent and permanent memory upgrades by changing the Licensed Internal Code Configuration Control (LICCC) without the use of EDA. It also differs from the flexible memory option. The flexible memory option is meant to anticipate nondisruptive CPC drawer replacement. Therefore, the use of flexible memory is temporary, in contrast with plan-ahead memory.
When you are preparing for a future memory upgrade, memory can be pre-plugged, based on a target capacity. The pre-planned memory is available for all z14 models, and can be ordered with flexible memory on a multiple drawer z14 model. The pre-plugged memory can be made available through a LICCC update.
You can order this LICCC through the following channels:
•The IBM Resource Link® (a login is required). For more information, see the IBM Resource Link website.
•Your IBM representative
The installation and activation of any pre-planned memory requires the purchase of the required feature codes (FCs), as listed in Table 2-9.
Table 2-9 Feature codes for plan-ahead memory
|
Memory
|
z14 feature code
|
|
Pre-planned memory
Charged when physical memory is installed.
Used for tracking the quantity of physical increments of plan-ahead memory capacity.
|
•FC 1990 - 32 GB
•FC 1991 - 64 GB
|
|
Virtual Flash Memory (VFM) Pre-planned Memory
Charged when physical memory is installed.
Used for tracking the quantity of physical increments of plan-ahead VFM memory capacity.
|
FC 1999 - 64 GB
|
|
Pre-planned memory activation
Charged when plan-ahead memory is enabled.
Used for tracking the quantity of increments of plan-ahead memory that are being activated.
|
•FC 1898/19381
(32 GB memory capacity Increments) |
1 (Main memory <1 TB)/(main memory >1 TB).
The payment for plan-ahead memory is a two-phase process. One charge occurs when the plan-ahead memory is ordered. Another charge occurs when the prepaid memory is activated for use. For more information about the exact terms and conditions, contact your IBM representative.
Pre-planned memory is installed by ordering FC 1990 (32 GB) or FC 1991 (64 GB). The ordered amount of plan-ahead memory is charged at a reduced price compared to the normal price for memory. One FC 1990 is needed for each 32 GB of usable memory (40 GB RAIM), or one FC 1991 is needed for each 64 GB of usable memory (80 GB RAIM).
If VFM is present, it is included in the Flexible memory calculations. Normal plan ahead memory increments are used first with the normal “feature conversion” action. When normal plan ahead features are used, the VFM 1.5 TB plan ahead increment is deleted and normal 32 GB memory increments are added. No feature conversions are allowed from the 1.5 TB VFM plan ahead increment to regular memory, plan ahead memory, or VFM memory.
Installed pre-planned memory is activated by ordering FC 1989/1938, FC 1893/1939, or FC 1894/1940, which causes the other portion of the previously contracted charge price to be invoiced.
|
Reminder: Normal memory upgrades use the plan-ahead memory first.
|
2.5 Reliability, availability, and serviceability
IBM Z servers continue to deliver enterprise class RAS with IBM z14 servers. The main philosophy behind RAS is about preventing or tolerating (masking) outages. It is also about providing the necessary instrumentation (in hardware, LIC and microcode, and software) to capture or collect the relevant failure information to help identify an issue without requiring a reproduction of the event. These outages can be planned or unplanned. Planned and unplanned outages can include the following situations (examples are not related to the RAS features of IBM Z servers):
•A planned outage because of the addition of processor capacity
•A planned outage because of the addition of I/O cards
•An unplanned outage because of a failure of a power supply
•An unplanned outage because of a memory failure
The IBM Z hardware has decades of intense engineering behind it, which results in a robust and reliable platform. The hardware has many RAS features that are built into it. For more information, see Chapter 9, “Reliability, availability, and serviceability” on page 363.
2.5.1 RAS in the CPC memory subsystem
Patented error correction technology in the memory subsystem continues to provide the most robust error correction from IBM to date. Two full DRAM failures per rank can be spared and a third full DRAM failure can be corrected.
DIMM level failures, including components, such as the memory controller application-specific integrated circuit (ASIC), power regulators, clocks, and system board, can be corrected. Memory channel failures, such as signal lines, control lines, and drivers and receivers on the MCM, can be corrected.
Upstream and downstream data signals can be spared by using two spare wires on the upstream and downstream paths. One of these signals can be used to spare a clock signal line (one upstream and one downstream). The following improvements were also added in the z14 server:
•No cascading of memory DIMMs
•Independent channel recovery
•Double tabs for clock lanes
•Separate replay buffer per channel
•Hardware driven lane soft error rate (SER) and sparing
2.5.2 General z14 RAS features
The z14 server includes the following RAS features:
•The z14 server provides a true N+1 cooling function for the radiator-cooled (air-cooled) model and a true N+1 (fully redundant) cooling function for the water-cooled model.
•The power supplies for the z14 server are also based on the N+1 design. The second power supply can maintain operations and avoid an unplanned outage of the system.
•The IBM Z processors feature improved chip packaging (encapsulated chip connectors) and use SER hardened latches throughout the design.
•An N+2 point-of-load (POL) power conversion redundancy, which protects the processor from the loss of voltage because of POL failures.
•An N+2 redundancy on the environmental sensors (ambient temperature, relative humidity, air density2, and corrosion).
•Enhanced bus structure that uses integrated time domain reflectometry (TDR) technology.
•PCIe service enhancements:
– Mandatory end-to-end cyclic redundancy check (ECRC)
– Customer operation code separate from maintenance code
– Native PCIe firmware stack that is running on the integrated firmware processor (IFP) to manage isolation and recovery
IBM z14 servers continue to deliver robust server designs through exciting new technologies, hardening both new and classic redundancy.
For more information, see Chapter 9, “Reliability, availability, and serviceability” on page 363.
2.6 Connectivity
Connections to PCIe I/O drawers, I/O drawers, Parallel Sysplex InfiniBand (PSIFB) coupling, and Integrated Coupling Adapters (ICAs) are driven from the CPC drawer fanout cards. These fanouts are on the front of the CPC drawer.
Figure 2-23 shows the location of the fanouts for a four CPC drawer system. In all, 10 PCIe fanout slots and 4 IFB fanout slots are available per CPC drawer. Each CPC drawer has two FSPs for system control; the location code is LGXX.
Figure 2-23 Location of the PCIe and IFB fanouts
Up to 10 PCIe fanouts (LG03 - LG12) and four IFB fanouts (LG13 - LG16) can be installed in each CPC drawer.
A fanout can be repaired concurrently with the use of redundant I/O interconnect. For more information, see 2.6.1, “Redundant I/O interconnect” on page 68.
The following types of fanouts are available:
•PCIe Generation3 fanout card: This copper fanout provides connectivity to the PCIe switch cards in the PCIe I/O drawer.
•Integrated Coupling Adapter (ICA SR): This adapter provides coupling connectivity between two z14 or z13s servers.
•Host Channel Adapter (HCA3-O [12xIFB]): This optical fanout provides 12x InfiniBand coupling link connectivity up to 150 meters (492 feet) distance to z14, z13, z13s, zEC12, zBC12, z196, and z114 servers.
•Host Channel Adapter (HCA3-O LR [1xIFB]): This optical long range fanout provides 1x InfiniBand coupling link connectivity up to a 10 km (6.2 miles) unrepeated or 100 km (62 miles) when extended by using IBM Z qualified dense wavelength division multiplexing (DWDM) equipment) distance to z14, z13s, z13, zEC12, zBC12, z196, and z114 servers.
When you are configuring for availability, balance the channels, coupling links, and OSAs across drawers. In a system that is configured for maximum availability, alternative paths maintain access to critical I/O devices, such as disks and networks. The CHPID Mapping Tool can be used to assist with configuring a system for high availability.
Enhanced (CPC) drawer availability (EDA) allows a single CPC drawer in a multidrawer CPC to be removed and reinstalled concurrently for an upgrade or a repair. Removing a CPC drawer means that the connectivity to the I/O devices that are connected to that CPC drawer is lost. To prevent connectivity loss, the redundant I/O interconnect feature allows you to maintain connection to critical devices, except for ICA and PSIFB coupling, when a CPC drawer is removed.
2.6.1 Redundant I/O interconnect
Redundancy is provided for PCIe I/O interconnects.
The PCIe I/O drawer supports up to 32 PCIe features, which are organized in four hardware domains per drawer, as shown in Figure 2-24.
Figure 2-24 Redundant I/O interconnect for PCIe I/O drawer
Each domain is driven through a PCIe Gen3 Interconnect switch adapter. The two PCIe switch cards provide a backup path for each other through the passive connection in the PCIe I/O drawer backplane. During a PCIe fanout or cable failure, all 16 PCIe features in the two domains can be driven through a single PCIe switch card (see Figure 2-25).
Figure 2-25 Redundant I/O Interconnect
To support Redundant I/O Interconnect (RII) between front to back domain pairs 0, 1 and 2, 3, the two interconnects to each pair must be driven from two different PCIe fanouts. Normally, each PCIe interconnect in a pair supports the eight features in its domain. In backup operation mode, one PCIe interconnect supports all 16 features in the domain pair.
|
Note: The PCIe Gen3 Interconnect (switch) adapter must be installed in the PCIe Drawer to maintain the interconnect across I/O domains. If the adapter is removed, the I/O cards in that domain (up to eight) become unavailable.
|
2.6.2 Enhanced drawer availability
With EDA, the effect of CPC drawer replacement is minimized. In a multiple CPC drawer system, a single CPC drawer can be concurrently removed and reinstalled for an upgrade or repair. Removing a CPC drawer without affecting the workload requires sufficient resources in the remaining CPC drawer.
Before removing the CPC drawer, the contents of the PUs and memory of the drawer must be relocated. PUs must be available on the remaining CPC drawers to replace the deactivated drawer. Also, sufficient redundant memory must be available if no degradation of applications is allowed. To ensure that the CPC configuration supports removal of a CPC drawer with minimal effect on the workload, consider the flexible memory option. Any CPC drawer can be replaced, including the first CPC drawer that initially contains the HSA.
Removal of a CPC drawer also removes the CPC drawer connectivity to the I/O drawers, PCIe I/O drawers, and coupling links. The effect of the removal of the CPC drawer on the system is limited by the use of redundant I/O interconnect. (For more information, see 2.6.1, “Redundant I/O interconnect” on page 68.) However, all PSIFB or ICA-SR links on the removed CPC drawer must be configured offline.
If the enhanced drawer availability and flexible memory options are not used when a CPC drawer must be replaced, the memory in the failing drawer is also removed. This process might be necessary during an upgrade or a repair action. Until the removed CPC drawer is replaced, a power-on reset of the system with the remaining CPC drawers is supported. The CPC drawer can then be replaced and added back into the configuration concurrently.
2.6.3 CPC drawer upgrade
All fanouts that are used for I/O and coupling links are rebalanced concurrently as part of a CPC drawer addition to support better RAS characteristics.
2.7 Model configurations
When a z14 order is configured, PUs are characterized according to their intended usage. They can be ordered as any of the following items:
CP The processor is purchased and activated. CP supports the z/OS, z/VSE, z/VM, z/TPF, and Linux on z Systems operating systems. It can also run Coupling Facility Control Code.
Capacity marked CP A processor that is purchased for future use as a CP is marked as available capacity. It is offline and not available for use until an upgrade for the CP is installed. It does not affect software licenses or maintenance charges.
IFL The Integrated Facility for Linux (IFL) is a processor that is purchased and activated for use by z/VM for Linux guests and Linux on z Systems operating systems.
Unassigned IFL A processor that is purchased for future use as an IFL. It is offline and cannot be used until an upgrade for the IFL is installed. It does not affect software licenses or maintenance charges.
ICF An internal coupling facility (ICF) processor that is purchased and activated for use by the Coupling Facility Control Code.
zIIP An “Off Load Processor” for workload that is restricted to Db2 type applications.
Additional SAP An optional processor that is purchased and activated for use as an SAP.
A minimum of one PU that is characterized as a CP, IFL, or ICF is required per system. The maximum number of CPs, IFLs, and ICFs is 170. The maximum number of zIIPs is always up to twice the number of PUs that are characterized as CPs.
Not all PUs on a model must be characterized.
The following items are present in the z14 server, but they are not part of the PUs that clients purchase and require no characterization:
•SAP to be used by the channel subsystem. The number of predefined SAPs depends on the z14 model.
•One IFP, which is used in the support of designated features, such as zEDC, RoCE, RoCE-2, zHyperLink Express, Internal Shared Memory (ISM) SMC-D, and RCE.
•Two spare PUs, which can transparently assume any characterization during a permanent failure of another PU.
The z14 model nomenclature is based on the number of PUs that are available for client use in each configuration. The models are listed in Table 2-10.
Table 2-10 z14 processor configurations
|
Model
|
Drawers/
PUs
|
CPs
|
IFLs
uIFL
|
ICFs
|
zIIPs
|
Add. SAPs
|
Std. SAPs
|
Spares
|
IFP
|
|
M01
|
1/33
|
0 - 33
|
0 - 33
0 - 32
|
0 - 33
|
0 - 22
|
0 - 4
|
5
|
2
|
1
|
|
M02
|
2/69
|
0 - 69
|
0 - 69
0 - 68
|
0 - 69
|
0 - 46
|
0 - 8
|
10
|
2
|
1
|
|
M03
|
3/105
|
0 - 105
|
0 - 105
0 - 104
|
0 - 105
|
0 - 70
|
0 - 12
|
15
|
2
|
1
|
|
M04
|
4/141
|
0 - 141
|
0 - 141
0 - 140
|
0 - 141
|
0 - 94
|
0 - 16
|
20
|
2
|
1
|
|
M05
|
4/170
|
0 - 170
|
0 - 170
0 - 169
|
0 - 170
|
0 - 112
|
0 - 16
|
23
|
2
|
1
|
A capacity marker identifies the number of CPs that were purchased. This number of purchased CPs is higher than or equal to the number of CPs that is actively used. The capacity marker marks the availability of purchased but unused capacity that is intended to be used as CPs in the future. They often have this status for software-charging reasons. Unused CPs are not a factor when establishing the millions of service units (MSU) value that is used for charging monthly license charge (MLC) software, or when charged on a per-processor basis.
2.7.1 Upgrades
Concurrent upgrades of CPs, IFLs, ICFs, zIIPs, or SAPs are available for the z14 server. However, concurrent PU upgrades require that more PUs be installed but not activated.
Spare PUs are used to replace defective PUs. Two spare PUs always are on a z14 server. In the rare event of a PU failure, a spare PU is activated concurrently and transparently and is assigned the characteristics of the failing PU.
If an upgrade request cannot be accomplished within the configuration, a hardware upgrade is required. The upgrade enables the addition of one or more CPC drawers to accommodate the wanted capacity. Extra CPC drawers can be installed concurrently.
Although upgrades from one z14 model to another z14 model are concurrent (meaning that one or more CPC drawers can be added) one exception exists. Upgrades from any z14 server (model M01, M02, M03, or M04) to a model M05 is not supported. M05 model is factory only.
The possible upgrades within the z14 configuration range are listed in Table 2-11.
Table 2-11 z14 to z14 upgrade paths
|
To 3906
From 3906
|
Model M01
|
Model M02
|
Model M03
|
Model M04
|
Model M05
|
|
Model M01
|
-
|
Yes
|
Yes
|
Yes
|
No
|
|
Model M02
|
-
|
-
|
Yes
|
Yes
|
No
|
|
Model M03
|
-
|
-
|
-
|
Yes
|
No
|
|
Model M04
|
-
|
-
|
-
|
-
|
No
|
You can also upgrade a IBM zEnterprise zEC12 (2827) or a IBM z13 (2964) to a z14 server and preserve the CPC serial number (S/N). The I/O cards can also be carried forward (with certain restrictions) to the z14 server.
|
Important: Upgrades from z Enterprise EC12 (zEC12) and IBM z13 are always disruptive.
|
Upgrade paths from any z Enterprise EC12 (zEC12) to any z14 server are supported, as listed in Table 2-12.
Table 2-12 IBM zEnterprise zEC12 to z14 upgrade paths
|
To 3906
From 2827
|
Model M01
|
Model M02
|
Model M03
|
Model M04
|
Model M05
|
|
Model H20
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
|
Model H43
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
|
Model H66
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
|
Model H89
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
|
Model HA1
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
Upgrades from any IBM z13 to any z14 server are supported, as listed in Table 2-13.
Table 2-13 z13 to z14 upgrade paths
|
To 3906
From 2964
|
Model M01
|
Model M02
|
Model M03
|
Model M04
|
Model M05
|
|
Model N30
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
|
Model N63
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
|
Model N96
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
|
Model NC9
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
|
Model NE1
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
2.7.2 Concurrent PU conversions
Assigned CPs, assigned IFLs, and unassigned IFLs, ICFs, zIIPs, and SAPs can be converted to other assigned or unassigned feature codes.
Most conversions are nondisruptive. In exceptional cases, the conversion might be disruptive; for example, when a model z14 with 30 CPs is converted to an all IFL system. In addition, an LPAR might be disrupted when PUs must be freed before they can be converted. Conversion information is listed in Table 2-14.
Table 2-14 Concurrent PU conversions
|
To
From
|
CP
|
IFL
|
Unassigned IFL
|
ICF
|
zIIP
|
SAP
|
|
CP
|
-
|
Yes
|
Yes
|
Yes
|
Yes
|
Yes
|
|
IFL
|
Yes
|
-
|
Yes
|
Yes
|
Yes
|
Yes
|
|
Unassigned IFL
|
Yes
|
Yes
|
-
|
Yes
|
Yes
|
Yes
|
|
ICF
|
Yes
|
Yes
|
Yes
|
-
|
Yes
|
Yes
|
|
zIIP
|
Yes
|
Yes
|
Yes
|
Yes
|
-
|
Yes
|
|
SAP
|
Yes
|
Yes
|
Yes
|
Yes
|
Yes
|
-
|
2.7.3 Model capacity identifier
To recognize how many PUs are characterized as CPs, the Store System Information (STSI) instruction returns a Model Capacity Identifier (MCI). The MCI determines the number and speed of characterized CPs. Characterization of a PU as an IFL, ICF, or zIIP is not reflected in the output of the STSI instruction because characterization has no effect on software charging. For more information about STSI output, see “Processor identification” on page 357.
The following distinct model capacity identifier ranges are recognized (one for full capacity and three for granular capacity):
•For full-capacity engines, model capacity identifiers 701 - 7H0 are used. They express capacity settings for 1 - 170 characterized CPs.
•Three model capacity identifier ranges offer a unique level of granular capacity at the low end. They are available when no more than 33 CPs are characterized. These three subcapacity settings are applied to up to 33 CPs, which combined offer 90 more capacity settings. For more information, see “Granular capacity”.
Granular capacity
The z14 server offers 99 capacity settings at the low end of the processor. Only 33 CPs can have granular capacity. When subcapacity settings are used, other PUs beyond 33 can be characterized only as specialty engines.
The three defined ranges of subcapacity settings have model capacity identifiers numbered 401- 433, 501 - 533, and 601 - 633.
|
Consideration: Within a z14 server, all CPs have the same capacity identifier. Specialty engines (IFLs, zIIPs, and ICFs) operate at full speed.
|
List of model capacity identifiers
Regardless of the number of CPC drawers, a configuration with one characterized CP is possible, as listed in Table 2-15. For example, model 7H0 might have only one PU that is characterized as a CP.
Table 2-15 Model capacity identifiers
|
z14
|
Model capacity identifier
|
|
Model M01
|
701 - 733, 601 - 633, 501 - 533, and 401 - 433
|
|
Model M02
|
701 - 769, 601 - 633, 501 - 533, and 401 - 433
|
|
Model M03
|
701 - 7A5, 601 - 633, 501 - 533, and 401 - 433
|
|
Model M04
|
701 - 7E5, 601 - 633, 501 - 533, and 401 - 433
|
|
Model M05
|
701 - 7H0, 601 - 633, 501 - 533, and 401 - 433
|
|
Important: On z14 servers, model capacity identifier 400 is used ICFs only configurations, and 400 or 401 for IFLs only model.
|
2.7.4 Model capacity identifier and MSU value
All model capacity identifiers have a related MSU value. The MSU values are used to determine the software license charge for MLC software. Tables with MSU values are available at the Mainframe Exhibits for IBM Servers website.
2.7.5 Capacity Backup
Capacity Backup (CBU) delivers temporary backup capacity in addition to the capacity that an installation might already have available in numbers of assigned CPs, IFLs, ICFs, zIIPs, and optional SAPs. CBU has the following types:
•CBU for CP
•CBU for IFL
•CBU for ICF
•CBU for zIIP
•Optional SAPs
When CBU for CP is added within the same capacity setting range (indicated by the model capacity indicator) as the currently assigned PUs, the total number of active PUs (the sum of all assigned CPs, IFLs, ICFs, zIIPs, and optional SAPs) plus the number of CBUs cannot exceed the total number of PUs available in the system.
When CBU for CP capacity is acquired by switching from one capacity setting to another, no more CBUs can be requested than the total number of PUs available for that capacity setting.
CBU and granular capacity
When CBU for CP is ordered, it replaces lost capacity for disaster recovery. Specialty engines (ICFs, IFLs, and zIIPs) always run at full capacity, and when running as a CBU to replace lost capacity for disaster recovery.
When you order CBU, specify the maximum number of CPs, ICFs, IFLs, zIIPs, and SAPs to be activated for disaster recovery. If disaster strikes, you decide how many of each of the contracted CBUs of any type to activate. The CBU rights are registered in one or more records in the CPC. Up to eight records can be active, which can contain various CBU activation variations that apply to the installation.
You can test the CBU. The number of CBU test activations that you can run for no extra fee in each CBU record is now determined by the number of years that are purchased with the CBU record. For example, a three-year CBU record has three test activations, as compared to a one-year CBU record that has one test activation.
You can increase the number of tests up to a maximum of 15 for each CBU record. The real activation of CBU lasts up to 90 days with a grace period of two days to prevent sudden deactivation when the 90-day period expires. The contract duration can be set 1 - 5 years.
The CBU record describes the following properties that are related to the CBU:
•Number of CP CBUs allowed to be activated
•Number of IFL CBUs allowed to be activated
•Number of ICF CBUs allowed to be activated
•Number of zIIP CBUs allowed to be activated
•Number of SAP CBUs allowed to be activated
•Number of additional CBU tests that are allowed for this CBU record
•Number of total CBU years ordered (duration of the contract)
•Expiration date of the CBU contract
The record content of the CBU configuration is documented in IBM configurator output, which is shown in Example 2-1. In the example, one CBU record is made for a five-year CBU contract without more CBU tests for the activation of one CP CBU.
Example 2-1 Simple CBU record and related configuration features
On Demand Capacity Selections:
NEW00001 - CBU - CP(1) - Years(5) - Tests(5)
Resulting feature numbers in configuration:
6817 Total CBU Years Ordered 5
6818 CBU Records Ordered 1
6820 Single CBU CP-Year 5 
In Example 2-2, a second CBU record is added to the configuration for two CP CBUs, two IFL CBUs, and two zIIP CBUs, with five more tests and a five-year CBU contract. The result is that a total number of 10 years of CBU ordered: Five years in the first record and five years in the second record. The two CBU records are independent and can be activated individually. Five more CBU tests were requested. Because a total of five years are contracted for a total of three CP CBUs (two IFL CBUs and two zIIP CBUs), they are shown as 15, 10, 10, and 10 CBU years for their respective types.
Example 2-2 Second CBU record and resulting configuration features
NEW00001 - CBU - Replenishment is required to reactivate
Expiration(06/21/2017)
NEW00002 - CBU - CP(2) - IFL(2) - zIIP(2)
Total Tests(10) - Years(5)
Resulting cumulative feature numbers in configuration:
6805 5 Additional CBU Tests 5
6817 Total CBU Years Ordered 10
6818 CBU Records Ordered 2
6820 Single CBU CP-Year 15
6822 Single CBU IFL-Year 10
6828 Single CBU zIIP-Year 10
CBU for CP rules
Consider the following guidelines when you are planning for CBU for CP capacity:
•The total CBU CP capacity features are equal to the number of added CPs plus the number of permanent CPs that change the capacity level. For example, if two CBU CPs are added to the current model 503, and the capacity level does not change, the 503 becomes 505:
(503 + 2 = 505)
If the capacity level changes from a 503 to a 606, the number of extra CPs (three) is added to the three CPs of the 503, which results in a total number of CBU CP capacity features of six:
(3 + 3 = 6)
•The CBU cannot decrease the number of CPs.
•The CBU cannot lower the capacity setting.
|
Remember: CBU for CPs, IFLs, ICFs, zIIPs, and SAPs can be activated together with On/Off Capacity on Demand temporary upgrades. Both facilities can be on a single system, and can be activated simultaneously.
|
CBU for specialty engines
Specialty engines (ICFs, IFLs, and zIIPs) run at full capacity for all capacity settings. This fact also applies to CBU for specialty engines. The minimum and maximum (min - max) numbers of all types of CBUs that can be activated on each of the models are listed in Table 2-16. The CBU record can contain larger numbers of CBUs than can fit in the current model.
Table 2-16 Capacity Backup matrix
|
Model
|
Total PUs available
|
CBU
CPs
min - max
|
CBU
IFLs
min - max
|
CBU
ICFs
min - max
|
CBU
zIIPs
min - max
|
CBU
SAPs
min - max
|
|
Model M01
|
33
|
0 - 170
|
0 - 170
|
0 - 170
|
0 - 114
|
0 - 16
|
|
Model M02
|
69
|
0 - 170
|
0 - 170
|
0 - 170
|
0 - 114
|
0 - 16
|
|
Model M03
|
105
|
0 - 170
|
0 - 170
|
0 - 170
|
0 - 114
|
0 - 16
|
|
Model M04
|
141
|
0 - 170
|
0 - 170
|
0 - 170
|
0 - 114
|
0 - 16
|
|
Model M05
|
170
|
0 - 170
|
0 - 170
|
0 - 170
|
0 - 114
|
0 - 16
|
Unassigned IFLs are ignored because they are considered spares and are available for use as CBU. When an unassigned IFL is converted to an assigned IFL, or when more PUs are characterized as IFLs, the number of CBUs of any type that can be activated is decreased.
2.7.6 On/Off Capacity on Demand and CPs
On/Off Capacity on Demand (CoD) provides temporary capacity for all types of characterized PUs. Relative to granular capacity, On/Off CoD for CPs is treated similarly to the way that CBU is handled.
On/Off CoD and granular capacity
When temporary capacity that is requested by On/Off CoD for CPs matches the model capacity identifier range of the permanent CP feature, the total number of active CPs equals the sum of the number of permanent CPs plus the number of temporary CPs ordered. For example, when a model capacity identifier 504 has two CP5s added temporarily, it becomes a model capacity identifier 506.
When the addition of temporary capacity that is requested by On/Off CoD for CPs results in a cross-over from one capacity identifier range to another, the total number of CPs active when the temporary CPs are activated is equal to the number of temporary CPs ordered. For example, when a CPC with model capacity identifier 504 specifies six CP6 temporary CPs through On/Off CoD, the result is a CPC with model capacity identifier 606. A cross-over does not necessarily mean that the CP count for the extra temporary capacity increases. The same 504 might temporarily be upgraded to a CPC with model capacity identifier 704. In this case, the number of CPs does not increase, but more temporary capacity is achieved.
On/Off CoD guidelines
When you request temporary capacity, consider the following guidelines:
•Temporary capacity must be greater than permanent capacity.
•Temporary capacity cannot be more than double the purchased capacity.
•On/Off CoD cannot decrease the number of engines on the CPC.
•Adding more engines than are currently installed is not possible.
For more information about temporary capacity increases, see Chapter 8, “System upgrades” on page 315.
2.8 Power and cooling
The z14 power and cooling system is a continuation of IBM z13 with the addition of improvements in availability, temperature tolerance, and vectored (directional) air output. The air-cooled z14 server now has a radiator unit (RU) N+1 design for the pumps and blowers. With the new rear cover design for vectored air output, you can choose whether the output air goes up or down.
The water-cooled system is still an option for the z14 server. The Top Exit Power feature is available for the z14 server. Combined with the Top Exit I/O Cabling feature, it gives you more options when you are planning your computer room cabling. For more information about the z14 Top Exit features, see 10.3, “Physical planning” on page 398.
2.8.1 Power and cooling
The system operates with two fully redundant power supplies. One is in the front side of the Z frame, and the other is in the rear side of the Z frame. Each power supply has one or two power cords. The number of power cords that are required depends on the system configuration. The total loss of one power supply has no effect on system operation.
Systems that specify two power cords can be brought up with one power cord and continue to run.
The larger systems that have a minimum of four BPR pairs that are installed must have four power cords installed. Systems that specify four power cords can be started with two power cords on the same side with sufficient power to keep the system running.
Power cords attach to a three-phase, 50/60 Hz, 200 - 480 V AC power source, or a 380 - 520 V DC power source.
A Balanced Power Plan Ahead feature is available for future growth, which helps to ensure adequate and balanced power for all possible configurations. With this feature, system downtime for upgrading a server is eliminated by including the maximum power requirements in terms of BPRs and power cords to your installation.
For ancillary equipment, such as the Hardware Management Console, its display, and its switch, more single phase outlets are required.
The power requirements depend on the cooling facility that is installed, and on the number of CPC drawers and I/O units that are installed. For more information about the requirements that are related to the number of installed I/O units, see 10.1.2, “Power requirements and consumption” on page 391.
2.8.2 High Voltage Direct Current power feature
The High Voltage Direct Current power feature is an option for z14 servers. It allows z14 servers to directly use the high voltage (HV) DC distribution, as shown in Figure 2-26. A direct HV DC data center power design can improve data center energy efficiency by removing the need for a DC-to-AC inversion step.
Figure 2-26 AC versus DC distribution
The z14 bulk power supplies were modified to support HV DC, so the only difference in the shipped hardware to implement this option is the DC power cords. Because HV DC is a new technology, multiple proposed standards are available.
z14 servers support ground referenced and dual polarity HV DC supplies, such as +/-190 V, +/-260 V, and +380 V. Beyond the data center uninterruptible power supply and power distribution energy savings, a z14 server that runs on HV DC power draws 1 - 3% less input power. HV DC does not change the number of power cords that a system requires.
2.8.3 Internal Battery Feature
The Internal Battery Feature (IBF) is an option on the z14 server. It is shown in Figure 2-1 on page 36 for air-cooled (radiator) models and in Figure 2-2 on page 37 for water-cooled models. The IBF provides a local uninterrupted power source.
The IBF further enhances the robustness of the power design, which increases power line disturbance immunity. It provides battery power to preserve processor data during a loss of power on all power feeds from the computer room. The IBF can hold power briefly during a brownout, or for orderly shutdown for a longer outage. For information about the hold times, which depend on the I/O configuration and amount of CPC drawers, see 10.1.4, “Internal Battery Feature” on page 396.
2.8.4 Power estimation tool
The power estimation tool for the z14 server allows you to enter your precise server configuration to obtain an estimate of power consumption. Log in to the Resource link with your user ID. Click Planning → Tools → Power Estimation Tools. Specify the quantity for the features that are installed in your system. This tool estimates the power consumption for the specified configuration. The tool does not verify that the specified configuration can be physically built.
|
Tip: The exact power consumption for your system varies. The object of the tool is to estimate the power requirements to aid you in planning for your system installation. Actual power consumption after installation can be confirmed by using the HMC Monitors Dashboard task.
|
2.8.5 Cooling
The PU SCMs are cooled by a cold plate that is connected to the internal water-cooling loop. The SC SCMs are air-cooled. In an air-cooled system, the radiator unit dissipates the heat from the internal water loop with air. The radiator unit provides improved availability with N+1 pumps and blowers. The WCUs are fully redundant in an N+1 arrangement.
Air-cooled models
In z14 servers, the CPC drawer, SC SCMs, PCIe I/O drawers, I/O drawers, and power enclosures are all cooled by forced air with blowers that are controlled by the Move Device Assembly (MDA).
The PU SCMs in the CPC drawers are cooled by water. The internal closed water loop removes heat from PU SCMs by circulating water between the radiator heat exchanger and the cold plate that is mounted on the PU SCMs. For more information, see 2.8.6, “Radiator Unit” on page 80.
Although the PU SCMs are cooled by water, the heat is exhausted into the room from the radiator heat exchanger by forced air with blowers. At the system level, z14 servers are still air-cooled systems.
Water-cooled models
z14 servers are available as water-cooled systems. With WCU technology, z14 servers can transfer most of the heat that they generate into the building’s chilled water, which effectively reduces the heat output to the computer room.
Unlike the radiator in air-cooled models, a WCU has two water loops: An internal closed water loop and an external (chilled) water loop. The external water loop connects to the client-supplied building’s chilled water. The internal water loop circulates between the WCU heat exchanger and the PU SCMs cold plates. The loop takes heat away from the PU SCMs and transfers it to the external water loop in the WCU’s heat exchanger. For more information, see 2.8.7, “Water-cooling unit” on page 82.
In addition to the PU SCMs, the internal water loop circulates through two heat exchangers that are in the path of the exhaust air in the rear of the frames. These heat exchangers remove approximately 60% - 65% of the residual heat from the I/O drawers, PCIe I/O drawers, the air-cooled logic in the CPC drawers, and the power enclosures. Almost two-thirds of the total heat that is generated can be removed from the room by the chilled water.
The selection of air-cooled models or water-cooled models is done when ordering, and the appropriate equipment is factory-installed. An MES (conversion) from an air-cooled model to a water-cooled model and vice versa is not allowed.
2.8.6 Radiator Unit
The Radiator Unit provides cooling to PU SCMs with closed loop water. No connection to an external chilled water supply is required. For the z14 server, the internal circulating water is conditioned water that is added to the radiator unit during system installation with the Fill and Drain Tool (FC 3380).
Fill and Drain Tool (FDT)
The Fill and Drain Tool (FDT) is included with new z14 servers, but if you have an FDT from a IBM z13 (FC 3380) or a zEC12 (FC 3378) in the data center, you can order an upgrade kit (FC 3379) to have the same equipment as in the FC 3380. It also can be used for the z14, IBM z13, and zEC12 servers. The FDT is used to provide the internal water at the installation and for maintenance, and to remove it at discontinuance.
The FDT is shown in Figure 2-27.
Figure 2-27 Fill and Drain Tool (FDT)
The radiator unit connects to all four CPC drawers (if installed) and cools all PU SCMs simultaneously. The cooling capability is a redundant N+2 design, so a single working pump and blower can support the entire load. The replacement of one pump or blower can be done concurrently and has no performance effect.
The water pumps, manifold assembly, radiator assembly (which includes the heat exchanger), and blowers are the main components of the z14 RU, as shown in Figure 2-28.
Figure 2-28 Radiator unit
The closed water loop in the radiator unit is shown in Figure 2-29. The warm water that is exiting from the PU SCMs cold plates enters pumps through a common manifold and is pumped through a heat exchanger where heat is extracted by the air flowing across the heat exchanger fins. The cooled water is then recirculated back into the PU SCMs cold plates.
Figure 2-29 Radiator cooling system
2.8.7 Water-cooling unit
z14 servers continue to provide the ability to cool systems with building-provide chilled water by employing the WCU technology. The PU SCMs in the CPC drawers are cooled by internal closed loop water. The internal closed loop water exchanges heat with building-provided chilled water in the WCU heat exchanger. The source of the building’s chilled water is provided by the client.
A WCU is shown in Figure 2-30.
Figure 2-30 Water-cooling unit
The water in the closed loop within the system exchanges heat with the continuous supply of building-provided chilled water. This water circulates between the PU SCMs cold plates and a heat exchanger within the WCU. Heat from the PU SCMs is transferred to the cold plates, where it is in turn transferred to the circulating system water (closed loop). The system water then dissipates its heat to the building-provided chilled water within the WCU’s heat exchanger. The PU SCMs are cooled efficiently in this manner.
This principle is shown in Figure 2-31.
Figure 2-31 WCU water loop
z14 servers operate with two fully redundant WCUs. These water-cooling units have each their own facility feed and return water connections. If water is interrupted to one of the units, the other unit picks up the entire load, and the server continues to operate without interruption. You must provide independent redundant water loops to the water-cooling units to obtain full redundancy.
The internal circulating water is conditioned water that is added to the radiator unit during system installation with the Fill and Drain Tool (FC 3380). The FDT is included with new z14 servers. However, if you have an FDT from a zEC12 (FC 3378) in the data center, you can order an upgrade kit (FC 3379) to have the same equipment as in the FC 3380, and it can be used for the zEC12, z13, and z14 servers. The FDT is used to provide the internal water at the installation and for maintenance, and to remove it at discontinuance. The FDT is shown in Figure 2-27 on page 80.
Exhaust Air Heat Exchanger
In z14 servers, all water-cooled models have two Exhaust Air Heat Exchanger units that are installed on the rear of the A and Z frames, as shown in Figure 2-32. These units remove heat from the internal system water loop and internal air exits the server into the hot air exhaust aisle.
Figure 2-32 Water-cooled model: Rear view
In addition to the PU SCMs cold plates, the internal water loop circulates through these two heat exchangers. These exchangers are in the path of the exhaust air in the rear of the frames. These heat exchangers remove approximately 65% of the residual heat from the I/O drawers, PCIe I/O drawer, the air-cooled logic in the CPC drawer, and the power enclosures. The goal is for two-thirds of the total heat that is generated to be removed from the room by the chilled water.
If one client water supply or one WCU fails, the remaining feed maintains PU SMCs cooling. The WCUs and the associated drive card are concurrently replaceable. In addition, the heat exchangers can be disconnected and removed from the system concurrently.
Considerations before you order
The water-cooling option is preferable because it can substantially lower the total power consumption of z14 servers and the total cost of ownership for the CPC. This savings is greater for the larger models of z14 servers, as listed in Table 2-17.
Table 2-17 Heat to water based on temperature
|
Temperature
|
Three CPC drawer typical configuration
|
Four CPC drawer typical configuration
|
Four CPC drawer maximum power configuration
|
|
Water-cooled system power in normal room/hot room
|
|||
|
|
12.9 kW/14.1 kW
|
17.4 kW/19.0 kW
|
24.7 kW/26.3 kW
|
|
Inlet air temperature
|
Heat to water and as % of total system heat load
|
||
|
18°C (64°F)
|
7.3 kW (57%)
|
9.8 kW (56%)
|
12.6 kW (51%)
|
|
23°C (73°F)
|
9.5 kW (74%)
|
12.6 kW (72%)
|
15.6 kW (63%)
|
|
27°C (80.6°F)
|
11.5 kW (89%)
|
14.8 kW (85%)
|
18.0 kW (73%)
|
|
32°C (89.6°F)
(hot room) |
14.8 kW (105%)
|
18.2 kW (96%)
|
21.6 kW (82%)
|
The water-cooling option cannot be installed in the field. Therefore, you must carefully consider the present and future computer room and CPC configuration options before you decide which cooling option to order. For more information, see 10.1.3, “Cooling requirements” on page 393.
2.9 Summary
All aspects of the z14 structure are listed in Table 2-18.
Table 2-18 System structure summary
|
Description
|
Model M01
|
Model M02
|
Model M03
|
Model M04
|
Model M05
|
|
Number of CPC drawers
|
1
|
2
|
3
|
4
|
4
|
|
Number of SCMs
|
6
|
12
|
18
|
24
|
28
|
|
Total number of PUs
|
41
|
82
|
123
|
164
|
196
|
|
Maximum number of characterized PUs
|
33
|
69
|
105
|
141
|
170
|
|
Number of CPs
|
0 - 33
|
0 - 69
|
0 - 105
|
0 - 141
|
0 - 170
|
|
Number of IFLs
|
0 - 33
|
0 - 69
|
0 - 105
|
0 - 141
|
0 - 170
|
|
Number of ICFs
|
0 - 33
|
0 - 69
|
0 - 105
|
0 - 141
|
0 - 170
|
|
Number of zIIPs
|
0 - 22
|
0 - 46
|
0 - 70
|
0 - 94
|
0 - 112
|
|
Standard SAPs
|
5
|
10
|
15
|
20
|
23
|
|
Additional SAPs
|
0 - 4
|
0 - 8
|
0 - 12
|
0 - 16
|
0 - 16
|
|
Number of IFP
|
1
|
1
|
1
|
1
|
1
|
|
Standard spare PUs
|
2
|
2
|
2
|
2
|
2
|
|
Enabled memory sizes GB
|
320 - 8000
|
320 - 16192
|
320 - 24384
|
320 - 32576
|
320 - 32576
|
|
Flexible memory sizes GB
|
N/A
|
320 - 8000
|
320 - 16192
|
320 - 24384
|
320 - 24384
|
|
L1 cache per PU (I/D)
|
128/128 KB
|
128/128 KB
|
128/128 KB
|
128/128 KB
|
128/128 KB
|
|
L2 cache per PU
|
2/4 MB (I/D)
|
2/4 MB (I/D)
|
2/4 MB (I/D)
|
2/4 MB (I/D)
|
2/4 MB (I/D)
|
|
L3 shared cache per PU chip
|
128 MB
|
128 MB
|
128 MB
|
128 MB
|
128 MB
|
|
L4 shared cache per node
|
672 MB
|
672 MB
|
672 MB
|
672 MB
|
672 MB
|
|
Cycle time (ns)
|
0.192
|
0.192
|
0.192
|
0.192
|
0.192
|
|
Clock frequency
|
5.2 GHz
|
5.2 GHz
|
5.2 GHz
|
5.2 GHz
|
5.2 GHz
|
|
Maximum number of PCIe fanouts
|
10
|
20
|
30
|
40
|
40
|
|
Maximum number of IFB fanouts
|
4
|
8
|
12
|
16
|
16
|
|
I/O interface per IFB cable
|
6 GBps
|
6 GBps
|
6 GBps
|
6 GBps
|
6 GBps
|
|
I/O interface per PCIe cable
|
16 GBps
|
16 GBps
|
16 GBps
|
16 GBps
|
16 GBps
|
|
Number of support elements
|
2
|
2
|
2
|
2
|
2
|
|
External AC power
|
3-phase
|
3-phase
|
3-phase
|
3-phase
|
3-phase
|
|
Optional external DC
|
520 V/380 V
|
520 V/380 V
|
520 V/380 V
|
520 V/380 V
|
520 V/380 V
|
|
Internal Battery Feature
|
Optional
|
Optional
|
Optional
|
Optional
|
Optional
|
1 z/OS V1.13 has additional requirements. See the Software Requirements section.
2 The air density sensor measures air pressure and is used to control blower speed.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.