Channel Subsystem
This chapter presents an overview of the Channel Subsystem.
This chapter includes the following sections:
|
Remember: This book is third in a series. The first two are IBM zEnterprise EC12 Technical Introduction, SG24-8050, and IBM zEnterprise EC12 Technical Guide, SG24-8049.
|
1.1 Channel subsystem (CSS)
The channels in the CSS allow the transfer of data between main storage and I/O devices or other servers under the control of a channel program. Through these channel connections, the CSS enables communication from server memory to peripheral devices.
The CSS allows channel I/O operations to continue independently of other operations within the server. This function allows other functions to resume after an I/O operation is initiated. The following key entities comprise the CSS:
•Channel
The communication path from the Channel Subsystem to the connected control units and I/O devices. The channel subsystem communicates with I/O devices by using channel paths between the channel subsystem and control units.
•Physical channel ID (PCHID)
A PCHID reflects the physical location of a channel-type interface. A PCHID number is based on the I/O drawer or cage location, the channel feature slot number, and the port number of the channel feature. A CHPID does not directly correspond to a hardware channel port, but is assigned to a PCHID in Hardware Configuration Definition (HCD) or input/output configuration program (IOCP).
•Channel Path Identifier (CHPID)
A value that is assigned to each channel path of the system that uniquely identifies that path. A total of 256 CHPIDs are supported by one CSS. On a IBM zEnterprise EC12 system, a CHPID number is assigned to a physical location by the user by using HCD or IOCP.
•Subchannel
Provides the logical appearance of a device to the program, and contains the information that is required for sustaining a single I/O operation. A subchannel is assigned for each device that is defined to the logical partition.
•Control unit
Provides the logical capabilities necessary to operate and control an I/O device. It adapts the characteristics of each device so that it can respond to the standard form of control that is provided by the CSS. A control unit can be housed separately, or it can be physically and logically integrated with the I/O device, the channel subsystem, or within the server itself.
•Input/output (I/O) device
Provides external storage, which is a means of communication between data processing systems, or a means of communication between a system and its environment. In the simplest case, an I/O device is attached to one control unit and is accessible through one channel path.
Figure 1-1 shows the relationship between the CSS, the channels, the control units, and the I/O devices.
Figure 1-1 Channel Subsystem relationship to channels, control units, and I/O devices
1.1.1 Multiple channel subsystems
The multiple CSS concept that is implemented in the System z servers is designed to offer considerable increase in processing power, memory size, and I/O connectivity over previous servers.
Table 1-1 shows a summary of CSS features available on various System z models. A third subchannel set (SS2) was introduced in the IBM zEnterprise 196 server. With SS2, you can define extra 64-K subchannels in each CSS. For more information, see 1.3, “Subchannel sets” on page 6 for details.
Table 1-1 CSS features per System z model
|
Item
|
zEC12
|
z196
|
IBM System z10® EC
|
|
Number of CSSs
|
4 per server
|
4 per server
|
4 per server
|
|
Devices in Subchannel set-0
|
63.75 K per CSS
255 K per server
|
63.75 K per CSS
255 K per server
|
63.75 K per CSS
255 K per server
|
|
Devices in Subchannel set-1
|
64 K-1 per CSS
256 K-4 per server
|
64 K-1 per CSS
256 K-4 per server
|
64 K-1 per CSS
256 K-4 per server
|
|
Devices in Subchannel set-2
|
64 K-1 per CSS
256 K-4 per server
|
64 K-1 per CSS
256 K-4 per server
|
N/A
|
|
Partitions
|
15 per CSS
60 per server
|
15 per CSS
60 per server
|
15 per CSS
60 per server
|
|
CHPIDs
|
256 per CSS
1024 per server
|
256 per CSS
1024 per server
|
256 per CSS
1024 per server
|
On a System z server, all channel subsystem images are defined within a single I/O configuration data set (IOCDS). The IOCDS is loaded into the server’s hardware system area (HSA) and initialized during a power-on reset.
Figure 1-2 shows a logical view of these relationships. The zEC12 supports four Channel Subsystems per server. CSSs are numbered from 0 to 3. These designations are sometimes referred to as the “CSS Image ID” or “CSSID” (CSSID 0, 1, 2, or 3 for zEC12, z196 and z10 EC).
Each channel subsystem can have from 1 to 256 channels, and can in turn be configured with 1 to 15 logical partitions. There is a maximum of 60 logical partitions per CEC.
Figure 1-2 Logical view of multiple CSSs in a zEC12 server
1.1.2 Multiple image facility (MIF)
MIF enables resource sharing across logical partitions within a single CSS. When a channel is shared across logical partitions in multiple CSSs, it is known as “spanning.” See Table 1-2 on page 10 for more information.
1.2 Logical partitions
A logical partition (LPAR) supports the running of an operating system, such as z/OS, and provides central processors (CPs), memory, subchannels, and access to channels. The zEC12 server does not support basic mode. Only LPAR mode can be defined.
The following definitions apply to System z servers:
•Logical partition name
Defined by the user through HCD or IOCP. It is the name in the RESOURCE statement in the IOCP. The logical partition names must be unique across all CSSs in the server.
•Logical partition identifier
A number in the range of 00 to 3F. It is assigned in the image profile through the Support Element or the Hardware Management Console. It is unique across the server, and can also be referred to as the User Logical Partition ID (UPID).
Generally, establish a numbering convention for the logical partition identifiers. The example uses the CSS number concatenated to the MIF Image ID, which means logical partition ID “3A” is in CSS “3” with MIFID “A”. This convention fits within the allowed range of logical partition IDs and conveys useful information.
•MIF ID
Defined by using HCD or IOCP in the RESOURCE statement. It is in the range of x'1’ to x'F' and is unique within a CSS, but not unique across multiple CSSs.
Multiple CSSs can specify the same MIF ID. The MIF ID is also known as Image ID (IID).
Figure 1-3 shows the relationship between CSSs, logical partitions, and associated MIF IDs. TST1, PROD1, and PROD2 are defined to CSS 0. TST2, PROD3, and PROD4 are defined to CSS 1. TST3 is defined to CSS 2. TST4 and PROD5 are defined to CSS 3. Notice that PROD2 and PROD5 have the same MIFID, although their logical partition IDs are different. Also, the HCD and Hardware Configuration Manager (HCM) panel refers to the definition of a MIF ID in each CSSID.
Figure 1-3 CSS and logical partition definition of a zEC12 server
I/O operations for a logical partition are identified as originating from a Channel Image (CI). The Channel Image is defined as CI = CSSID + MIFID.
For IBM FICON® I/O operations, the addressing is CI (+ CHPID) + S_ID + D_ID + CUI + UA, where the terms are defined as:
S_ID Source ID
D_ID Destination ID
CUI Control Unit ID
UA Unit Address
The FICON control unit logical addressing for channel-to-channel connections is CSSID.MIFID.
1.3 Subchannel sets
In IBM System zArchitecture, each I/O device is represented by a separate set of controls for each logical partition, called a subchannel. The subchannel is used by the operating system to pass an I/O request from the System Control Program (SCP) to the channel subsystem. To a program, the subchannel is displayed as a device. In the channel subsystem, the primary control block for an I/O is the Subchannel, which represented by a control block that is called the Unit Control Word (UCW). UCWs are part of the HSA.
The number of devices that can be addressed by a logical partition can be a limitation for some installations. The concept of MSS provides relief for this problem.
Usually, a subchannel represents an addressable I/O device. Therefore, a disk control unit with 30 drives uses 30 subchannels for base addresses. An addressable device is associated with a device number and the device number is commonly, but incorrectly, known as the device address.
Subchannel numbers are limited to four hexadecimal digits by hardware and software architectures. Four hexadecimal digits provides up to 64-K addresses, which are known as a set. IBM reserves 256 subchannels, leaving 63.75-K subchannels for general use.
For I/O constraint relief, three subchannel sets are available per CSS. This configuration allows access to a greater number of logical volumes. It also allows improved device connectivity for parallel access volumes (PAVs), peer-to-peer remote copy (PPRC) secondaries, and IBM FlashCopy® devices. The zEC12 allows you to IPL from subchannel set 1 (SS1) or subchannel set 2 (SS2), in addition to subchannel set 0.
The third subchannel set was introduced and implemented in each logical channel subsystem by the IBM zEnterprise 196 server. Subchannel set 0 (SS0) provides a total of 63.75 K subchannels, Subchannel set 1 (SS1) and Subchannel set 2 (SS2) provide to the installation the full range of 64 K-1 addresses.
Figure 1-4 shows the channel subsystems (CSS) and the associated subchannel sets.
Figure 1-4 Multiple subchannels sets
Each CSS has three subchannel sets. The three subchannel sets enable a total of 63.75-K subchannels in set 0, 64 K-1 subchannels in set 1, and an extra 64 k-1 subchannels in set 2.
The current implementation in z/OS allows the use of Subchannel set 0 (SS0) to define any type of device allowed today as base, alias, and secondary devices. The second and third subchannel sets (SS1 and SS2) allow you to define disk alias devices (of both primary and secondary devices), and Metro Mirror secondary devices only.
The appropriate subchannel set number must be included in IOCP definitions or in the HCD definitions that produce the IOCDS. The subchannel set number defaults to zero.
With the availability of HyperPAV, the requirement for PAV devices is greatly reduced. HyperPAV allows an alias address to be used to access any base on the same control unit image per I/O base. It also allows different HyperPAV hosts to use one alias to access different basis, which reduces the number of alias addresses required. HyperPAV is designed to enable applications to achieve equal or better performance than is possible with the original PAV feature alone, while also using the same or fewer z/OS resources. HyperPAV is an optional feature on the IBM System Storage® DS8000® series.
Figure 1-6 on page 11 and Example 1-1 on page 12 illustrate the use and definition of multiple subchannel sets.
1.4 Channels
The channel subsystem communicates with I/O devices through channel paths between the channel subsystem and control units. A channel is the communication path from the channel subsystem to the connected control units and I/O devices.
1.4.1 Channel Path ID
A CHPID is a value that is assigned to each channel path of the system that uniquely identifies that path. A total of 256 CHPIDs are supported by each CSS. A CHPID number is assigned to a PCHID by the user using HCD or IOCP.
1.4.2 Physical Channel ID
A PCHID reflects the physical location of a channel-type interface. A PCHID number is based on the I/O cage, I/O Drawer, or PCIe I/O Drawer location, the channel feature slot number, and the port number of the channel feature. A CHPID no longer directly corresponds to a hardware channel port, but is assigned to a PCHID by using HCD or IOCP.
You can address 256 CHPIDs within a single channel subsystem. This address space provides a maximum of 1024 CHPIDs when four CSSs are defined. Each CHPID within a CSS is associated with a single channel. The physical channel, which uniquely identifies a connector jack on a channel feature, is known by its PCHID number.
PCHIDs identify the physical ports on cards that are in I/O cages, I/O drawers, or PCIe I/O Drawers. They follow the numbering scheme that is defined for the processor model. PCHID values are shown in Table 1-7 on page 17 and Table 1-8 on page 17 for I/O cages, in Table 1-10 on page 19 for I/O drawers, and in Table 1-11 on page 20 for PCIe I/O Drawers.
1.4.3 Adapter ID
The Adapter ID (AID) assigned to each InfiniBand host channel adapter (HCA) used for coupling initially reflects the physical location of the HCA.
The AID specifies the adapter identifier that is associated with the host channel adapter (HCA) on which this channel path is defined. It is determined from the PCHID report when a host channel adapter is ordered. An AID is assigned for HCA2-O or HCA3-O adapters, and is required for all Coupling over InfiniBand (CIB) channel path definitions. The AID is a two-digit hexadecimal number between x”00” and x”1F”. A maximum of 16 CIB channel paths can be defined for the same AID across both ports.
For more information about InfiniBand, see Getting Started with InfiniBand on System z10 and System z9, SG24-7539.
1.4.4 Control unit
A control unit (CU) provides the capabilities necessary to operate and control an I/O device. The control unit acts as an interface between the CSS and the I/O device, masking the uniqueness of the I/O device from the CSS. A control unit can be housed separately, or it can be physically and logically integrated with the I/O device, the CSS, or within the server itself.
1.4.5 I/O device
An input/output (I/O) device is the end point in the “conduit” between a server and a peripheral device. Although the channel does not communicate directly with I/O devices (it communicates with control units), they are mentioned because subchannels as displayed as I/O devices to programs.
An I/O device has the characteristics of the peripheral device that it represents. It can provide external storage, a means of communication between data-processing systems, or a means of communication between a system and its environment. In the simplest case, an I/O device is attached to one control unit and is accessible through one channel path.
1.4.6 Channel path sharing
There are now two possibilities for channel path sharing:
•MIF
This configuration enables channel sharing among logical partitions that run in one channel subsystem.
•Spanning
This configuration extends the MIF concept of sharing channels across logical partitions in a single CSS to sharing channels across logical partitions and multiple channel subsystems.
Spanning is the ability for the channel to be configured to multiple channel subsystems. When defined that way, the channels can be transparently shared by any or all of the configured logical partitions, regardless of the channel subsystem to which they are configured.
MIF-shared channels
IBM Processor Resource/Systems Manager™ (PR/SM™) allows sharing of resources across logical partitions. MIF enables channel sharing among logical partitions, but sharing is limited to partitions defined to one channel subsystem.
It is important to understand qualifiers that apply to a logical partition definition. The following definitions for zEC12, z196, and System z10 EC are described in 1.2, “Logical partitions” on page 5:
•Logical partition name
•Logical partition identifier
•MIF ID
Spanned channels
Spanning is the ability of channels to be configured to multiple CSSs, and be transparently shared by any or all of the configured logical partitions that are configured in these CSSs.
Figure 1-5 shows an example with two CSSs that use spanned channels. There are spanned external channels (with associated PCHIDs) and spanned internal channels (no associated PCHIDs).
Figure 1-5 MIF-shared and spanned channels
Certain channels that are commonly used on earlier servers cannot be spanned. For example, on a z10 EC or z196, IBM ESCON® channels can be MIF-shared within one CSS, but cannot be spanned across multiple CSSs. Table 1-2 shows channels that can be shared and spanned by CHPID type on a zEC12 server.
Table 1-2 Spanned and shared channels on a zEC12 server
|
Channel type
|
CHPID definition
|
MIF-shared channels
|
Spanned channels
|
|
|
FICON Express8s
|
External
|
FC and FCP
|
Yes
|
Yes
|
|
FICON Express8a
|
External
|
FC and FCP
|
Yes
|
Yes
|
|
FICON Express41
|
External
|
FC and FCP
|
Yes
|
Yes
|
|
OSA-Express3a
|
External
|
OSD, OSE, OSC, OSN, OSM, and OSX (depending on feature code)
|
Yes
|
Yes
|
|
OSA-Express4s
|
External
|
OSD, OSE, OSC, OSN, OSM, and OSX (depending on feature code)
|
Yes
|
Yes
|
|
ISC-3a
|
External
|
CFP
|
Yes
|
Yes
|
|
PSIFB
|
External
|
CIB
|
Yes
|
Yes
|
|
IC
|
Internal
|
ICP
|
Yes
|
Yes
|
|
IBM HiperSockets™
|
Internal
|
IQD
|
Yes
|
Yes
|
1 Available only if carried forward during an upgrade.
Each channel type is described in more detail in Chapter 2, “Configuration Planning” on page 27.
Figure 1-6 shows a server with four CSSs using a third subchannel set:
•The logical CHPID numbers 50, 54, 58, and 5C are shared channels between the logical partitions. These CHPIDs are defined as spanned channel for each physical FICON (TYPE=FC) channels, PCHID numbers 241, 250, 1A2, and 5D1.
•All logical partitions in each CSS are using the subchannel sets SS0, SS1, and SS2 to address the same DASD. They use SS0 to address the base addresses D000-D070, SS1 to address the alias addresses D071-D09F, and SS2 to address the alias addresses D0A0-D0FF.
Example 1-1 on page 12 shows the IOCDS definitions for four CCSs and three subchannel sets.
Figure 1-6 Configuration example with CSSs, MSSs, and PCHIDs
The IOCP statements in Example 1-1 are not intended to represent a real server, but to illustrate the new elements involved. In the past, dummy LPARs needed to be defined in the IOCP to enable the user to add them dynamically later. Now IOCP automatically reserves all available CSSs and LPs not explicitly defined, up to four CSSs with 15 LPs each.
Example 1-1 IOCP definition example
ID MSG2='SYS6.IODF78 - 2012-06-20 15:48',SYSTEM=(2827,1), *
LSYSTEM=SCZP401, *
TOK=('SCZP401',008000093BD52817154857200112172F00000000,*
00000000,'12-06-20','15:48:57','SYS6','IODF78')
RESOURCE PARTITION=((CSS(0),(A0A,A),(A0B,B),(A0C,C),(A0D,D),(A*
0E,E),(A0F,F),(A01,1),(A02,2),(A03,3),(A04,4),(A05,5),(A*
06,6),(A07,7),(A08,8),(A09,9)),(CSS(1),(A1B,B),(A1D,D),(*
A1E,E),(A1F,F),(A11,1),(A12,2)))
CHPID PATH=(CSS(0,1,2,3),50),SHARED, *
PARTITION=((CSS(2),(A21),(=))), *
NOTPART=((CSS(0),(A0B,A0C,A0D,A0E,A0F),(=)),(CSS(1),(A1B*
,A1D,A1E,A1F),(=))),SWITCH=61,PCHID=524,TYPE=FC
CHPID PATH=(CSS(0,1,2,3),54),SHARED, *
PARTITION=((CSS(2),(A21),(=))), *
NOTPART=((CSS(0),(A0B,A0C,A0D,A0E,A0F),(=)),(CSS(1),(A1B*
,A1D,A1E,A1F),(=))),SWITCH=62,PCHID=554,TYPE=FC
CHPID PATH=(CSS(0,1,2,3),58),SHARED, *
PARTITION=((CSS(2),(A21),(=))), *
NOTPART=((CSS(0),(A0B,A0C,A0D,A0E,A0F),(=)),(CSS(1),(A1B*
,A1D,A1E,A1F),(=))),SWITCH=63,PCHID=514,TYPE=FC
CHPID PATH=(CSS(0,1,2,3),5C),SHARED, *
PARTITION=((CSS(2),(A21),(=))), *
NOTPART=((CSS(0),(A0B,A0C,A0D,A0E,A0F),(=)),(CSS(1),(A1B*
,A1D,A1E,A1F),(=))),SWITCH=64,PCHID=564,TYPE=FC
CNTLUNIT CUNUMBR=D000, *
PATH=((CSS(0),50,54,58,5C),(CSS(1),50,54,58,5C),(CSS(2),*
50,54,58,5C),(CSS(3),50,54,58,5C)),UNITADD=((00,256)), *
LINK=((CSS(0),10,10,24,24),(CSS(1),10,10,24,24),(CSS(2),*
10,10,24,24),(CSS(3),10,10,24,24)),CUADD=0,UNIT=2107
IODEVICE ADDRESS=(D000,113),CUNUMBR=(D000),STADET=Y,UNIT=3390B
IODEVICE ADDRESS=(D071,143),CUNUMBR=(D000),STADET=Y,SCHSET=1, *
UNIT=3390A
The SCHSET parameter in the IODEVICE statement defines the subchannel set used for a specific address range. If no subchannel set is defined, it defaults to subchannel set 0. The following devices are defined in the IOCDS:
•Bases D000-D070 → CSS0-3 → SS0
•Aliases D071-D09F → CSS0-3 → SS1
1.4.7 Channel program
A channel program is a set of channel command words (CCWs). Channel programs are built by the requester of the I/O and then control is passed to the next phase of the I/O, which is run by the System Assist Processor.
1.4.8 System Assist Processor
A System Assist Processor is a special-purpose PU responsible for handling I/O. System Assist Processors are sometimes referred to as I/O processors (IOPs).
1.4.9 Hardware system area (HSA)
The HSA is an area of memory in the processor main storage that is used by the hardware. It is established during power-on reset (POR) using the configuration information from the IOCDS. The zEC12 provides a fixed amount of 32 GB that is reserved for HSA storage size and is fenced off from client purchased memory. The HSA contains the subchannels (UCWs). Table 1-3 show the HSA size for the zEC12, z196, z114, z10 EC, and z10 BC servers.
Table 1-3 Fixed HSA sizes
|
System z Server
|
Fixed HSA Size
|
|
IBM zEnterprise EC12
|
32 GB
|
|
IBM zEnterprise 196
|
16 GB
|
|
IBM zEnterprise 114
|
8 GB
|
|
IBM System z10 Enterprise Class
|
16 GB
|
|
IBM System z10 Business Class
|
8 GB
|
1.4.10 Fanout card and adapters
A fanout card is designed to provide the path for data between memory and I/O by using feature cards and cables. zEC12 supports two different internal I/O infrastructures for the internal connection. zEC12 uses InfiniBand-based infrastructure for the internal connection to I/O cages and I/O drawers. It uses PCIe-based infrastructure for PCIe I/O drawers in which the cards for the connection to peripheral devices and networks.
The zEC12 server can have one to four books installed in the central processor complex (CPC). The InfiniBand and PCIe fanouts are in the front of each book. Each book has eight fanout slots. They are named D1 to DA, top to bottom. Slots D3 and D4 are not used for fanouts. Six types of fanout cards are supported by zEC12. Each slot holds one of the following six fanouts:
•Host Channel Adapter (HCA2-C, only when carried forward)
This copper fanout provides connectivity to the IFB-MP card in the I/O cage and I/O drawer.
•PCIe Fanout
This copper fanout provides connectivity to the PCIe switch card in the PCIe I/O drawer.
•Host Channel Adapter (HCA2-O (12xIFB), only when carried forward)
This optical fanout provides 12x InfiniBand coupling link connectivity up to 150 meters distance to an HCA2-O card on a zEC12, z196, z114, and System z10, or an HCA3-O on a zEC12, z196, and z114.
•Host Channel Adapter (HCA2-O LR (1xIFB), only when carried forward)
This optical long range fanout provides 1x InfiniBand coupling link connectivity up to 10-km unrepeated distance to an HCA2-O LR card on a zEC12, z196, z114, and System z10 servers, or an HCA3-O LR card on a zEC12, z196, and z114.
•Host Channel Adapter (HCA3-O (12xIFB)):
This optical fanout provides 12x InfiniBand coupling link connectivity up to 150 meters distance to an HCA2-O on a zEC12, z196, z114, and System z10, or an HCA3-O on a zEC12, z196, z114.
•Host Channel Adapter (HCA3-O LR (1xIFB)):
This optical long range fanout provides 1x InfiniBand coupling link connectivity up to 10-km unrepeated distance to an HCA2-O LR on a zEC12, z196, z114, and System z10 servers, or an HCA3-O LR on a zEC12, z196, z114.
|
Restriction: HCA2-O, HCA2-O LR, and HCA2-C can be carried forward only on an upgrade to a zEC12.
|
The HCA3-O LR (1xIFB) fanout comes with four ports, and each other fanout comes with two ports.
|
Restriction: A Memory Bus Adapter (MBA) used for copper cable ICB-4 links is not supported on a zEC12 or z196 server.
|
The HCA3-O, HCA3-O LR, HCA2-O, and HCA2-O LR fanouts that are used for coupling links have an assigned AID. This AID must be used for definitions in IOCDS to have a relation between the physical adapter location and the CHPID number. For AID number assignment, see “Adapter ID and port assignment” on page 107.
1.4.11 I/O cage, I/O drawer, PCIe I/O drawer
For a new build of a zEC12, a maximum of five PCIe drawers can be installed supporting up to 160 PCIe I/O features.
A mixture of I/O cages, I/O drawers, and PCIe I/O drawers are only available on upgrades to a zEC12. The number and mix of I/O cages, I/O drawers, and PCIe I/O drawers depends on the amount of I/O features that are carried forward by an upgrade.
Some I/O features are supported only by I/O cages and I/O drawers:
•FICON Express4
•FICON Express8
•OSA Express3
•ISC-3
•Crypto Express3
Depending on the amount of I/O cages and I/O drawers, PCIe I/O drawers can be added to support PCIe I/O features:
•FICON Express8S
•OSA Express4S
•Crypto Express4S
•Flash Express
Table 1-4 gives an overview of the number of I/O cages, I/O drawers, and PCIe drawers that can be present in a zEC12. The zEC12 supports the following I/O cage, I/O drawer, and PCIe I/O drawer combination:
•One I/O cage and a combination of four I/O drawer and PCIe I/O drawers
•Combination of six I/O drawers and PCIe I/O drawers
•Five PCIe I/O drawers
|
Restriction: With any I/O drawer and PCIe I/O drawer combination, there cannot be more than two I/O drawers or five PCIe I/O drawers.
|
Table 1-4 I/O cage and drawer summary
|
Description
|
New Build
|
Carry Forward
|
MES Add
|
|
I/O Cage
|
0
|
0-1
|
0
|
|
I/O Drawer
|
0
|
0-2
|
0
|
|
PCIe I/O Drawer
|
0-5
|
0-5
|
0-5
|
The number of I/O cages, I/O drawers, or PCIe I/O drawers can be driven by the total number of cards in a configuration, or by the cage or drawer power budget.
A maximum of 44 I/O features can be carried forward. Table 1-5 lists the number and mix of I/O cages and I/O drawers. The configuration depends on the number of older I/O features.
|
Restriction: On new build zEC12s, only PCIe I/O drawers are supported.
|
Table 1-5 Number and mix of I/O cages and I/O drawers
|
Number of I/O cards carried forward on upgrades
|
Number of I/O cages
|
Number of I/O drawers
|
|
0
|
0
|
0
|
|
1-8
|
0
|
1
|
|
9-16
|
0
|
2
|
|
17-28
|
1
|
0
|
|
29-36
|
1
|
1
|
|
37-44
|
1
|
2
|
1.4.12 I/O cage domains, slots, and PCHIDs
Each I/O cage supports up to seven I/O domains and a total of 28 I/O cards. Each I/O domain supports four I/O cards. Figure 1-7 shows the physical layout of the I/O cage, with the I/O slots and the I/O domains.
Figure 1-7 z196 I/O cage
The assignment of slots to domains in each I/O cage is identified in Table 1-6.
Table 1-6 I/O domain assignment in I/O cage
|
I/O domain
|
I/O slots in domain
|
|
0 (A)
|
1, 3, 6, and 8
|
|
1 (B)
|
2, 4, 7, and 9
|
|
2 (C)
|
10. 12, 15, and 17
|
|
3 (D)
|
11, 13, 16, and 18
|
|
4 (E)
|
19, 21, 24, and 26
|
|
5 (F)
|
20, 22, 25, and 27
|
|
6 (G)
|
29, 30, 31, and 32
|
|
Exception: The Power Sequence Controller (PSC) feature is not supported on zEC12
|
I/O slots 5, 14, 23, and 28 contain the IFB/MP cards that are used for IFB connections. Assuming a fully loaded I/O cage, there are two IFBs in each of these slots.
One IFB/MP card in slot 28 serves only I/O domain 6 in slots 29 - 32. Two IFB connections are still provided to allow redundant I/O interconnect.
The address of each PCHID is determined by the physical location of the card in the I/O cage. Each slot in an I/O cage supports up to 16 PCHID addresses. Table 1-7 lists the PCHID addresses used for front locations in each I/O cage.
Table 1-7 PCHIDs address range per I/O cage and I/O slot (front)
|
I/O slot #
|
I/O Domain
|
PCHID range
Cage 1 / Bottom A
|
|
1
|
0
|
100-10F
|
|
2
|
1
|
110-11F
|
|
3
|
0
|
120-12F
|
|
4
|
1
|
130-13F
|
|
5
|
IFB-MP
|
|
|
6
|
0
|
140-14F
|
|
7
|
1
|
150-15F
|
|
8
|
0
|
160-16F
|
|
9
|
1
|
170-17F
|
|
10
|
2
|
180-18F
|
|
11
|
3
|
190-19F
|
|
12
|
2
|
1A0-1AF
|
|
13
|
3
|
1B0-1BF
|
|
14
|
IFB-MP
|
|
|
15
|
2
|
1C0-1CF
|
|
16
|
3
|
1D0-1DF
|
|
17
|
2
|
1E0-1DF
|
|
18
|
3
|
1F0-1FF
|
Table 1-8 lists the PCHID addresses used for the rear locations in each I/O cage.
Table 1-8 PCHIDs address range per I/O cage and I/O slot (rear)
|
I/O slot #
|
I/O Domain
|
PCHID range
Cage 1 / Bottom A
|
|
19
|
4
|
200-20F
|
|
20
|
5
|
210-21F
|
|
21
|
4
|
220-22F
|
|
22
|
5
|
230-23F
|
|
23
|
IFB-MP
|
|
|
24
|
4
|
240-24F
|
|
25
|
5
|
250-25F
|
|
26
|
4
|
260-26F
|
|
27
|
5
|
270-27F
|
|
28
|
IFB-MP
|
|
|
29
|
6
|
280-28F
|
|
30
|
6
|
290-29F
|
|
31
|
6
|
2A0-2AF
|
|
32
|
6
|
2B0-2BF
|
|
33
|
DCAs
|
|
|
34
|
||
|
35
|
DCAs
|
|
|
36
|
||
1.4.13 I/O drawer domains, slots, and PCHIDs
Each I/O drawer supports two I/O domains and a total of eight I/O cards. Each I/O domain supports four I/O cards. Figure 1-8 shows the physical layout of the I/O drawer, with the I/O slots and the I/O domains.
Figure 1-8 zEC12 I/O drawer
The assignment of slots to domains in each I/O drawer is identified in Table 1-9.
Table 1-9 I/O domain assignment in the I/O drawer
|
I/O domains
|
I/O slots in domain
|
|
A
|
2, 5, 8, and 10
|
|
B
|
3, 4, 7, and 11
|
I/O slot 9 contains the IFB-MP card that is used for IFB connections. Two IFB connections are always provided to allow redundant I/O interconnect (Figure 1-8 on page 18).
The PCHID number range for each I/O card is determined by the physical location of the I/O card in an I/O drawer. Table 1-10 lists the PCHID number range for each I/O slot in each I/O drawer.
Table 1-10 PCHID assignments for I/O drawers
|
I/O Drawer Slots
|
PCHID range
|
|
|
Drawer 1
Z22B
|
Drawer 2
Z15B
|
|
|
2
|
580 - 58F
|
500 - 50F
|
|
3
|
590 - 59F
|
510 - 51F
|
|
4
|
5A0 - 5AF
|
520 - 52F
|
|
5
|
5B0 - 5BF
|
530 - 53F
|
|
7
|
5C0 - 5CF
|
540 - 54F
|
|
8
|
5D0 - 5DF
|
550 - 55F
|
|
10
|
5E0 - 5EF
|
560 - 56F
|
|
11
|
5F0 - 5FF
|
570 - 57F
|
1.4.14 PCIe drawer domains, slots, and PCHIDs
Each PCIe I/O drawer supports up to 4 I/O domains and a total of 32 I/O cards. Each I/O domain supports eight I/O cards. The PCIe I/O drawer domains and their related I/O slots are shown in Figure 1-9.
Figure 1-9 I/O domains of PCIe I/O drawer
The PCIe I/O Drawer supports up to 32 I/O cards. They are organized in four hardware domains per drawer. Each I/O domain supports up to eight I/O cards (FICON, OSA, Crypto, and Flash Express). Each domain is driven through a PCIe switch card. Two PCIe switch cards always provide a backup path for each other through the passive connection in the PCIe I/O Drawer backplane. During a PCIe fanout card or cable failure, all 16 I/O cards in the two domains can be driven through a single PCIe switch card.
Table 1-11 lists the I/O domains and I/O slots per domain.
Table 1-11 I/O domains of PCIe I/O drawer
|
Domain
|
I/O slot in domain
|
|
0
|
01, 02, 03, 04, 06, 07, 08, 09
|
|
1
|
30, 31, 32, 33, 35, 36, 37, 38
|
|
2
|
11, 12, 13, 14, 16, 17, 18, 19
|
|
3
|
20, 21, 22, 23, 25, 26, 27, 28
|
Table 1-12 lists the PCHID assignments for slots in the PCIe I/O drawers. Only the active ports on an installed card are assigned a PCHID. The remainder is unused.
Table 1-12 PCHID assignments for PCIe I/O drawers
|
PCIe I/O
drawer
slot
|
PCHID range
|
||||
|
Drawer 1
Z22B
|
Drawer 2
Z15B
|
Drawer 3
Z08B
|
Drawer 4
Z01B
|
Drawer 5
A08B (RCU)
A15B (WCU)
|
|
|
1
|
580 - 583
|
500 - 503
|
380- 383 3
|
300 - 303
|
180 - 183
|
|
2
|
584 - 587
|
504 - 507
|
384 - 387 3
|
304 - 307
|
184 - 187
|
|
3
|
588 - 58B
|
508 - 50B
|
388 - 38B
|
308 - 30B 1
|
188 - 18B
|
|
4
|
58C - 58F
|
50C - 50F
|
38C - 38F
|
30C - 30F
|
18C - 18F
|
|
6
|
590 - 593
|
510 - 513
|
390 - 393
|
310 - 313
|
190 - 193
|
|
7
|
594 - 597
|
514 - 517
|
394 - 397
|
314 - 317
|
194 - 197
|
|
8
|
598 - 59B
|
518 - 51B
|
398 - 39B
|
318 - 31B
|
198 - 19B
|
|
9
|
59C - 59F
|
51C - 51F
|
39C - 39F
|
31C - 31F
|
19C - 19F
|
|
11
|
5A0 - 5A3
|
520 - 523
|
3A0 - 3A3
|
320 - 323 1
|
A0 - 1A3
|
|
12
|
5A4 - 5A7
|
524 - 527
|
3A4 - 3A7
|
324 - 327
|
1A4 - 1A7
|
|
13
|
5A8 - 5AB
|
528 - 52B
|
3A8 - 3AB
|
328 - 32B
|
1A8 - 1AB
|
|
14
|
5AC - 5AF
|
52C - 52F
|
3AC - 3AF
|
32C - 32F
|
1AC - 1AF
|
|
16
|
5B0 - 5B3
|
530 - 533
|
3B0 - 3B3
|
330 - 333
|
1B0 - 1B3
|
|
17
|
5B4 - 5B7
|
534 - 537
|
3B4 - 3B7
|
334 - 337
|
1B4 - 1B7
|
|
18
|
5B8 - 5BB
|
538 - 53B
|
3B8 - 3BB
|
338 - 33B
|
1B8 - 1BB
|
|
19
|
5BC - 5BF
|
53C - 53F
|
3BC - 3BF
|
33C - 33F
|
1BC - 1BF
|
|
20
|
5C0 - 5C3
|
540 - 543
|
3C0 - 3C3
|
340 - 343
|
1C0 - 1C3
|
|
21
|
5C4 - 5C7
|
544 - 547 3
|
C4 - 3C7
|
344 - 347
|
1C4 - 1C7
|
|
22
|
5C8 - 5CB
|
548 - 54B
|
3C8 - 3CB
|
348 - 34B
|
1C8 - 1CB
|
|
23
|
5CC - 5CF
|
54C - 54F
|
3CC - 3CF
|
34C - 34F
|
1CC - 1CF
|
|
25
|
5D0 - 5D3
|
550 - 553
|
3D0 - 3D3
|
350 - 353
|
1D0 - 1D3
|
|
26
|
5D4 - 5D7
|
554 - 557
|
3D4 - 3D7
|
354 - 357
|
1D4 - 1D7
|
|
27
|
5D8 - 5DB
|
558 - 55B
|
3D8 - 3DB
|
358 - 35B
|
1D8 - 1DB
|
|
28
|
5DC - 5DF
|
55C - 55F
|
3DC - 3DF
|
35C - 35F
|
1DC - 1DF
|
|
30
|
5E0 - 5E3
|
560 - 563
|
3E0 - 3E3
|
360 - 363
|
1E0 - 1E3
|
|
31
|
5E4 - 5E7
|
564 - 567
|
3E4 - 3E7 3
|
64 - 367
|
1E4 - 1E7
|
|
32
|
5E8 - 5EB
|
568 - 56B
|
3E8 - 3EB
|
368 - 36B
|
1E8 - 1EB
|
|
33
|
5EC - 5EF
|
56C - 56F
|
3EC - 3EF
|
36C - 36F
|
1EC - 1EF
|
|
35
|
5F0 - 5F3
|
570 - 573
|
3F0 - 3F3
|
370 - 373
|
1F0 - 1F3
|
|
36
|
5F4 - 5F7
|
574 - 577
|
3F4 - 3F7
|
374 - 377
|
1F4 - 1F7
|
|
37
|
5F8 - 5FB
|
578 - 57B
|
3F8 - 3FB
|
378 - 37B
|
1F8 - 1FB
|
|
38
|
5FC - 5FF
|
57C - 57F
|
3FC - 3FF
|
37C - 37F
|
1FC - 1FF
|
1.5 Defining multiple CSSs and the third subchannel set
This section explains the new IODF definition for the zEC12 processor. For more information, see Hardware Configuration Definition Planning, GA22-7525 and Input/Output Configuration Program User’s Guide for ICP IOCP, SB10-7037.
When the definition of a new processor of type 2827 is added by HCD, the maximum number of logical partitions is automatically generated in input/output definition file (IODF) as shown in Figure 1-10.
|
Goto Filter Backup Query Help
------------------------------------------------------------------------------
Processor List Row 1 of 6 More: >
Command ===> _______________________________________________ Scroll ===> PAGE
Select one or more processors, then press Enter. To add, use F11.
/ Proc. ID Type + Model + Mode+ Serial-# + Description
_ ISGSYN 2064 1C7 LPAR __________ ________________________________
_ ISGS11 2064 1C7 LPAR __________ ________________________________
_ SCZP101 2094 S18 LPAR 02991E2094
_ SCZP201 2097 E26 LPAR 01DE502097
_ SCZP401 2827 H43 LPAR 0B3BD52827 zEC12
******************************* Bottom of data ********************************
+------------------------------------------------------------------+
| Definition of processor SCZP401 has been extended to its maximum |
| configuration. |
+------------------------------------------------------------------+
|
Figure 1-10 HCD: Adding a zEC12 processor
Figure 1-11 on page 23 shows the channel subsystem list after a new 2827 processor is defined. The zEC12 supports the maximum of four CSSs in a processor and three subchannel sets for each CSS. Note that three subchannel sets are defined in each channel subsystem, which is the maximum number that is supported, as follows:
•The maximum number of devices is 65280 per CSS for Subchannel set 0.
•The maximum number of devices is 65535 per CSS for Subchannel set 1.
•The maximum number of devices is 65535 per CSS for Subchannel set 2.
|
Channel Subsystem List Row 1 of 4
Command ===> _______________________________________________ Scroll ===> CSR
Select one or more channel subsystems, then press Enter. To add, use F11.
Processor ID . . . : SCZP401
CSS Devices in SS0 Devices in SS1 Devices in SS2
/ ID Maximum + Actual Maximum + Actual Maximum + Actual
_ 0 65280 7699 65535 11559 65535 0
_ 1 65280 7643 65535 11559 65535 0
_ 2 65280 7319 65535 11559 65535 0
_ 3 65280 7319 65535 11559 65535 0
|
Figure 1-11 Channel subsystem list after you define a new zEC12 processor
Each CSS can have a maximum of 15 logical partitions defined, so a zEC12 system can have up to 60 logical partitions, including reserved logical partitions.
Figure 1-12 shows the partition list in a channel subsystem. Initially, all 15 LPARs supported by a CSS are assigned as “reserved”. This means that all numbers (between x'1' and x'F') of the partition in each channel subsystem are already defined. This setting is indicated by an asterisk (*) in the partition name field. The asterisk ensures that the CSS treats this logical partition as a reserved logical partition. This configuration allows for the dynamic activation of an LPAR.
|
----------------------------- Partition List ----------------------------
Goto Backup Query Help
-----------------------------------------------------------------------
Row 1 of 15
Command ===> ________________________________________ Scroll ===> CSR
Select one or more partitions, then press Enter. To add, use F11.
Processor ID . . . . : SCZP401
Configuration mode . : LPAR
Channel Subsystem ID : 0
/ Partition Name Number Usage + Description
_ A0A A OS ITSOZVM3
_ A0B B OS CHPID holder
_ A0C C CF WTSCPLX8 CF8A
_ A0D D CF WTSCPLX8 CF8B
_ A0E E CF TESTPLEX CF7A
_ A0F F CF WTSCPLX1 CF02
_ A01 1 OS TESTPLEX SC74
|
Figure 1-12 Partition list after you define a new zEC12 processor
The partition usage is also defined as CF/OS (that is, partition number 4). This configuration means that these reserved logical partitions can be used as operating system, Coupling Facility, or Linux logical partitions.
During the initial configuration setup (the definition of the new zEC12 processor) in HCD, change only the definitions for the number of partitions with which you want to work.
The reserved logical partitions can be seen only as definitions in HCD or HCM, or in the IOCP deck that is produced by HCD. The Resource Partition statement in IOCP shows the reserved partitions with * in the name field. Space is allocated in the HSA. The reserved logical partition is not visible yet on the Contents of CPC in the Hardware Management Console (HMC).
In the IOCP deck that is produced by HCD, the RESOURCE PARTITION statement shows the reserved partitions with * in the name field, as shown in Figure 1-13. The sample extract of an IOCP shows that there are “reserved” partitions in CSS1, CSS2, and CSS3.
|
ID MSG2='SYS6.IODF78 - 2012-06-20 15:48',SYSTEM=(2827,1), *
LSYSTEM=SCZP401, *
TOK=('SCZP401',008000093BD52817154857200112172F00000000,*
00000000,'12-06-20','15:48:57','SYS6','IODF78')
RESOURCE PARTITION=((CSS(0),(A0A,A),(A0B,B),(A0C,C),(A0D,D),(A*
0E,E),(A0F,F),(A01,1),(A02,2),(A03,3),(A04,4),(A05,5),(A*
06,6),(A07,7),(A08,8),(A09,9)),(CSS(1),(A1B,B),(A1D,D),(*
A1E,E),(A1F,F),(A11,1),(A12,2),(A13,3),(A15,5),(A16,6),(*
A17,7),(A18,8),(A19,9),(*,4),(*,A),(*,C)),(CSS(2),(A2B,B*
),(A21,1),(*,2),(*,3),(*,4),(*,5),(*,6),(*,7),(*,8),(*,9*
),(*,A),(*,C),(*,D),(*,E),(*,F)),(CSS(3),(A31,1),(*,2),(*
*,3),(*,4),(*,5),(*,6),(*,7),(*,8),(*,9),(*,A),(*,B),(*,*
C),(*,D),(*,E),(*,F)))
|
Figure 1-13 Sample extract of IOCP deck
|
Consideration: IOCP ignores the MAXDEV keyword on the RESOURCE statement for a z10, z196, and zEC12. Instead, it always reserves the maximum number of subchannel sets and subchannels.
|
For more information about how to manage reserved logical partitions, see Chapter 10, “Reserved logical partitions” on page 531.
1.6 Activation
No single component or single part of the CPC completely defines the channel subsystem on the zEC12. It is more like a construct that comprises many CPC resources, both hardware and IBM Licensed Internal Code (LIC). These resources work together to support I/O operations across the CPC. These operations include I/O queuing, de-queuing, priority management, and identification of all I/O operations that are performed by logical partitions.
1.6.1 Activation procedure
During initialization of the server, CSS rules are enforced. Initialization of the server includes definition and activation of the logical partitions. The activation steps are as follows:
1. Reset the profile to activate the zEC12 CPC:
a. Build the CSS HSA contents based on the I/O configuration definition in the selected IOCDS.
b. Initialize all defined channel types.
c. Initialize FICON links.
2. Activate the required logical partitions (image profile):
a. Initialize the logical partition per the logical partition image profile.
b. Assign storage to the logical partition. The storage is never shared with other logical partitions.
c. For FICON channels, establish logical paths.
3. Load the initial program/load (Load Profile or Manual Load).
Perform an I/O system reset for the logical partition for all defined channel paths. The operating system starts the required I/O operations.
1.6.2 Dynamic addition or deletion of a logical partition
All logical partitions within a CSS are reserved by HCP/IOCP. The HSA for the z196 and zEC12 server is now a separately reserved area of memory outside of client purchased memory. You no longer must reserve space for future definitions in the IOCDS or do a POR to support these HSA activities.
When the definition of a new processor with 2827 type is added by HCD, the maximum number of logical partitions, channel subsystems, and subchannel sets is automatically generated in the IODF. A detailed example of reserved logical partitions is provided in Chapter 10, “Reserved logical partitions” on page 531.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.