This appendix introduces the native Peripheral Component Interconnect Express (PCIe) features management on IBM Z platform (IBM z14, z13 and z13s)¹. The appendix also includes concepts of the integrated firmware processor (IFP) and resource groups (RG).

This appendix includes the following topics:

The native PCIe adapter is a new category of features that was introduced in zEC12. IBM z14 supports the following native PCIE features:

These adapters are installed into a PCIe I/O drawer with the I/O features and include a physical channel ID (PCHID) that is assigned according to its physical location.

For all the feature adapters that are installed in an I/O drawer, management functions in the form of device drivers and diagnostic tools are always implemented to support virtualization of the adapter, service, and maintenance.

Traditionally, these management functions are integrated on the adapter with specific hardware design. For the newly introduced native PCIe adapters, these functions are moved out of the adapter and are now handled by an IFP.

For traditional I/O adapters, such as the Open Systems Adapter (OSA) and Fibre Channel connection (FICON) cards, the application-specific integrated circuit (ASIC) chip on the adapter always downloads the device drivers and diagnostic tools from the Support Element (SE) and run the management functions on the adapter. In the new design, no ASIC chip is used for management function on the native PCIe feature adapters.

For the RoCE Express, Coupling Express Long Reach, and zEDC, device drivers and diagnostic tools are now running on the IFP and use four RGs. Management functions, including virtualization, servicing and recovery, diagnostics, failover, firmware updates against an adapter, and other functions, are still implemented.

The IFP is a processor unit (PU) exclusively used to manage native PCIe feature adapters that are installed in the PCIe I/O drawer. On previous systems, this processor was not used and was called a reserved processor. It is allocated from the system PU pool and is not counted in the PUs available for characterization.

If a native PCIe feature is installed in the system, the system allocates and initializes an IFP during its power-on reset (POR) phase. Although the IFP is allocated to one of the physical PUs, it is not visible to the users. In an error or failover scenario, PU sparing also happens for an IFP, with the same rules as other PUs.

The IFP allocates four resource groups for running the management functions of native PCIe feature adapters. A native PCIe feature adapter is managed by one of the resource groups according to in which I/O domain this adapter is located.

As shown in Figure C-1, each I/O domain in a PCIe I/O drawer of a z14 server is logically attached to one resource group. The native PCIe I/O feature adapters are managed by their respective RG for device drivers and diagnostic tools functions.

Up to five PCIe I/O drawers are supported on z14 servers. The same type of native PCIe features is always assigned to different I/O domains in different resource groups (and different PCIe I/O drawers if the configuration includes them) to eliminate the possibility of a single point of failure.

As of this writing, an I/O domain of the PCIe I/O drawer can support a total of two native PCIe feature adapters with the PCIe feature cards (FICON, OSA, and Crypto).

The IFP and resource groups perform the following management tasks on the native PCIe adapters:

The firmware of native PCIe adapters and resource groups is part of the system’s microcode and can be updated by a Microcode Change Level (MCL) upgrade. MCL upgrades on adapters or on the code of the resource groups require the specific adapter or all native PCIe adapters that are managed by the specific resource group (depending on the type of u-code that it applies) to be offline during activation of the MCL.

However, to maintain availability, MCLs can be applied to only one resource group at a time. While one resource group is offline, the second resource group and all adapters in it remain active. An MCL application for a native PCIe adapter or resource group is not possible if an error condition exists within the other resource group.

If an error occurs in one of the resource groups or features that are assigned to one of the resource groups, the IFP manages error recovery and collects error data. The error data is sent by the IFP to the SE, which then provides a message on the SE and the Hardware Management Console (HMC). If an error requires maintenance, a call home to the IBM Support system is started by the HMC.

Any maintenance of a native PCIe feature is managed by the IFP, including testing or replacing a feature card. Before a feature is configured offline, the IFP ensures that the same type of feature is available in the same or the other resource group (if applicable).

The following maximum number of native PCIe adapters can be installed in a z14 server:

Considering availability, install adapters of the same type in slots of different I/O domains, drawers, fanouts, and resource groups. The next sections provide more information about achieving a highly available configuration.

During the ordering process of the native PCIe adapters, such as the zEDC Express and RoCE Express features, features of the same type are evenly spread across two resource groups (resource group 1 and resource group 2) for availability and serviceability.

A sample PCHID report of a z13 configuration with four zEDC Express features and four 10GbE RoCE Express features is shown in Figure C-2 on page 473. The following information is listed for each adapter:

The native PCIe features are not part of the traditional channel subsystem (CSS). Although they do not include a channel-path identifier (CHPID) assigned, they include a PCHID that is assigned according to their physical location in the PCIe I/O drawer.

To define the native PCIe adapters in the HCD or HMC, a new I/O configuration program (IOCP) FUNCTION statement is introduced that includes several feature-specific parameters.

The IOCP example that is shown in Example C-3 defines zEDC Express and 10GbE RoCE Express2 features to LPARs LP14 and LP15.

The FUNCTION identifier (FID) is a hexadecimal number 000 - FFF that you use to assign a PCHID to the FUNCTION to identify the specific hardware feature in the PCIe I/O drawer. Because the FUNCTION is not related to a channel subsystem, all LPARs on a central processor complex (CPC) can be defined to it. However, a FUNCTION cannot be shared between LPARs. It is only dedicated or reconfigurable by using the PART parameter. The TYPE parameter is required for z14.

If you want several LPARs to use a zEDC Express feature, you must use a Virtual Function (VF) number. A VF number is a number 1 - nnn, where nnn is the maximum number of LPARs that the feature supports. The maximum is 15 for the zEDC Express feature, 31 for the 10GbE RoCE Express (FC 0411), 126 for the RoCE Express2 feature, and 254 for the zHyperLink Express feature³.

The physical network ID (PNETID) is required to set up the Shared Memory Communications over Remote Direct Memory Access (SMC-R) communication between two RoCE Express/Express2 features. Each FUNCTION definition supports up to four PNETIDs.