Defining Coupling Facility links
This chapter describes the coupling connectivity options that are available on z14 ZR1. Coupling links support Parallel Sysplex and STP.
This chapter includes the following topics:
9.1 Coupling connectivity options on z14 ZR1
A Parallel Sysplex (sysplex) is a collection of z/OS systems that cooperate to process workloads by using certain hardware and software products. The hardware and software components that make up a sysplex cooperate to provide higher availability, coordinated systems management, and improved growth potential over a conventional computer system of comparable processing power.
The Server Time Protocol (STP) facility (FC 1021) is required to synchronize the time-of-day (TOD) clocks for systems in a sysplex that run on different CPCs.
For more information about STP configuration, see Chapter 8, “Preparing for Sysplex and configuring Server Time Protocol” on page 159.
A coupling facility (CF) enables parallel processing and improved data sharing for authorized programs that are running in the sysplex. The cross-system extended services (XES) component of z/OS enables applications and subsystems to take advantage of the coupling facility.
In a Parallel Sysplex, the central processor complexes (CPCs) are connected through a coupling facility by using coupling links.
9.1.1 Coupling connectivity for Parallel Sysplex on z14 ZR1
Coupling connectivity for Parallel Sysplex on z14 ZR1 use Coupling Express Long Range (CE LR) and Integrated Coupling Adapter Short Reach (ICA SR). The ICA SR feature is designed to support distances of up to 150 m (492 feet). The CE LR feature supports distances up to 10 km (6.2 miles) unrepeated between systems, and up to 100 km (62 miles) with a qualified Dense Wavelength Division Multiplexer (DWDM). The available options for coupling links on z14 ZR1 are shown in Figure 9-1.
Figure 9-1 z14 ZR1 coupling connectivity
|
Note: IBM z14 ZR1 does not support HCA3-O fanout for 12x IFB (#0171) and HCA3-O LR fanout for 1x IFB (#0170).
|
9.1.2 Internal coupling
Internal coupling (IC) links are used for internal communication between LPARs on the same system that is running coupling facilities (CF) and z/OS images. The connection is emulated in Licensed Internal Code (LIC) and provides fast and secure memory-to-memory communications between LPARs within a single system. No physical cabling is required.
9.1.3 Integrated Coupling Adapter Short Reach
First introduced with the IBM z13, the ICA SR is a two-port fanout that is used for short distance coupling connectivity and uses the coupling channel type CS5. The ICA SR also uses PCIe Gen3 technology, with x16 lanes that are bifurcated into x8 lanes for coupling.
The ICA SR supports cable length of up to 150 m (492.1 feet) and supports a link data rate of 8 Gbps1. It also supports up to four CHPIDs per port and seven subchannels (devices) per CHPID. The coupling links can be defined as shared between images within a CSS. They also can be spanned across multiple CSSs in an IBM Z CPC.
9.1.4 Coupling Express Long Range
The CE LR is a two-port PCIe native adapter that is used for long-distance coupling connectivity and uses a new coupling channel type: CL5. The CE LR feature uses PCIe Gen3 technology and is hosted in a PCIe I/O drawer. CE LR is not supported in a switched environment (point-to-point links only).
The feature supports communication at unrepeated distances up to 10 km (6.2 miles) by using 9μm single mode fiber optic cables and repeated distances up to 100 km (62 miles) by using IBM Z qualified DWDM vendor equipment. It supports up to four CHPIDs per port and 32 subchannels (devices) per CHPID. CE LR supports a link data rate of up to 10 Gbps2. The coupling links can be defined as shared between images within a CSS or spanned across multiple CSSs in an IBM Z CPC.
9.1.5 Preparing to define coupling facility links
A good point to start your preparation for implementing any kind of coupling link is an accurate and current documentation that clearly illustrates all connections that are needed to the new or upgraded CPC.
When installing coupling links, ensure that you ordered enough ports to provide your configuration with physical feature redundancy. Your sysplex should be configured for the highest possible availability.
Sysplex failure independence is a function of a z/OS to CF relationship. For example, all connectors to a structure on a stand-alone CF are failure independent. However, with an ICF, all connections from z/OS images on the same footprint are failure-dependent.
For more information, see Coupling Facility Configuration Options, ZSW01971, which can be found at the Parallel Sysplex on IBM Z web page.
Evaluate whether any channel features in the current configuration are not supported on the z14 ZR1. The configuration is reviewed for any channel types that cannot be carried forward nor connected to the z14 ZR1 server.
Per the earlier Statement for Direction (see 9.1.1, “Coupling connectivity for Parallel Sysplex on z14 ZR1” on page 200), IBM z14 ZR1 server does not support InfiniBand coupling connectivity. Therefore, when deploying a z14 ZR1 CPC, the only coupling links that are supported are the CE LR for long range, and the ICA SR for short reach coupling connectivity.
Another important point to be considered is to ensure that all processors that it will be connected by using coupling links follow the following restrictions: Only IBM Z servers of the current generation (N) or from the previous generation (N-1) can coexist in the same sysplex or CTN if they are assigned to an STP role and use coupling facility structures. The z14 ZR1 can connect to only z14, z13, and z13s if they are equipped with CE LR or ICA SR coupling links.
|
Note: z14 ZR1 supports coupling connectivity between N and N-1 server generations only. An N-2 server can be attached to an N-1 server for STP timing purposes only.
|
If coupling links are to be connected cross sites through DWDM, it is necessary to verify whether the DWDM equipment to be used supports the respective couplink link technology and is qualified for Server Time Protocol.
|
Note: Do not use DWDM equipment that is not qualified to transport STP information.
|
IBM does not provide channel cables as features on the z14 ZR1. Therefore, a complete analysis must be made of the I/O connectors that are used on systems that are being upgraded to z14 ZR1 to ensure that the appropriate fiber cabling is installed.
An equivalent study should be part of your preparation to install a new z14 ZR1 so that all cabling is delivered to the data center before the installation date.
All required cables for the z14 ZR1 must be identified and placed on order. Labeling of all cables it is required for the installation. At a minimum, the labels should identify the PCHID number.
If you received the configuration and PCHID reports from IBM, define your coupling links to fit your planned configuration to your new or upgraded CPC.
|
Note: Deactivate any coupling link on other connected systems before an upgrade, or you might experience configuration errors.
|
9.2 Coupling Express Long Range
This section describes implementing couplink links by using the Coupling Express Long reach feature.
9.2.1 CE LR: Implementation
The CE LR coupling link (CHPID type CL5) uses PCIe Gen3 technology and is hosted in a PCIe I/O drawer.
The definition of this CFlink is an example that it is part of the activity Define CF/STP link that is shown in Figure 1-3 on page 5.
A coupling link between a z14 ZR1 CPC and a z14 CPC that uses CL5 CHPIDs is defined in this section as an example (see Figure 9-2).
Figure 9-2 CF link connection that is CL5
CE LR is defined in IOCDS similar to PSIFB. Although this feature is a PCIe feature, a PCHID is used instead of an AID to identify the physical card. Example 9-1 shows a sample extract of the IOCP to define the new CHPID Type CL5 on the z14 that connects to a z14 ZR1.
Example 9-1 IOCP definitions for CHPID Type CL5 on z14
ID .. *
.. *
SYSTEM=(3906,1),LSYSTEM=CETUS, *
TOK=('CETUS',008001117A883907095804670118074F00000000,00*
000000,'18-03-15','09:58:04','........','........')
RESOURCE PARTITION=((CSS(0),(CETUS0A,A),.. *
),(CETUS0D,D),(CETUS0E,E),(CETUS0F,F),(CETUS01,1),(CETUS*
.. *
(CSS(2),(CETUS2A,A),(CETUS2B,B),(CETUS2C,C),(CETUS2D,D),*
(CETUS2E,E),.. *
CHPID PATH=(CSS(2),E9),SHARED,PARTITION=((CETUS2E),(=)), *
CPATH=(CSS(1),E9),CSYSTEM=MUSCA,PORT=2,PCHID=13C, *
TYPE=CL5
Example 9-2 on page 204 shows a sample extract of the corresponding IOCP definition for a connecting CHPID Type CL5 on a z14 ZR1.
Example 9-2 IOCP definitions for CHPID Type CL5 on z14
ID .. *
.. *
SYSTEM=(3907,1),LSYSTEM=MUSCA, *
TOK=('MUSCA',008001117A883907095804670118074F00000000,00*
000000,'18-03-15','09:58:04','........','........')
RESOURCE PARTITION=((CSS(0),(MUSCA0A,A),.. *
USCA07,7),(MUSCA08,8),(MUSCA09,9)),(CSS(1),(MUSCA1A,A),(*
MUSCA1B,B),(MUSCA1C,C),(MUSCA1D,D),(MUSCA1E,E),(MUSCA1F,*
F),(MUSCA11,1),.. *
CHPID PATH=(CSS(1),E9),SHARED,PARTITION=((MUSCA11),(=)), *
CPATH=(CSS(2),E9),CSYSTEM=CETUS,PORT=2,PCHID=124, *
TYPE=CL5
CNTLUNIT CUNUMBR=FFFD,PATH=((CSS(1),E5,E9)),UNIT=CFP
IODEVICE ADDRESS=(FFD4,008),CUNUMBR=(FFFD),UNIT=CFP
IODEVICE ADDRESS=(FFDC,008),CUNUMBR=(FFFD),UNIT=CFP
|
Note: When you connect CF sender and CF receiver channel paths, or CF peer channel paths, HCD proposes coupling facility control unit and device numbers that must be defined for a CF sender channel. (CF receiver channels do not require control units and devices to be defined.)
|
For more information about how to define CHPID Type CL5 in HCD, see 14.2.4, “Defining CL5 CHPIDs” on page 355 in this book.
9.2.2 CE LR: Managing the configuration
After activating the new configuration with the new CE LR CF links and after all the cables are connected, verify whether the CHPIDs are online and operating by using z/OS or the Support Element (SE) panels:
•Checking status by using z/OS commands:
– For example, to check the status of CHPID E9, you run a D M=CHP(E9) command, as shown in Example 9-3.
Example 9-3 Display status of CHPID E9
D M=CHP(E9)
IEE174I 16.04.03 DISPLAY M 738
CHPID E9: TYPE=34, DESC=COUPLING OVER ROCE, ONLINE
COUPLING FACILITY 003907.IBM.02.000000007A88
PARTITION: 1F CPCID: 00
NAMED CF77 CONTROL UNIT ID: FFFD
PATH PHYSICAL LOGICAL CHANNEL TYPE CAID PORT
E9 / 0507 ONLINE ONLINE CL5 10GbE-RoCE 013C 02
COUPLING FACILITY SUBCHANNEL STATUS
TOTAL: 48 IN USE: 48 NOT USING: 0 NOT USABLE: 0
OPERATIONAL DEVICES / SUBCHANNELS:
FFA4 / 3C40 FFA5 / 3C41 FFA6 / 3C42 FFA7 / 3C43
FFA8 / 3C44 FFA9 / 3C45 FFAA / 3C46 FFAB / 3C47
FFBC / 3C48 FFBD / 3C49 FFBE / 3C4A FFBF / 3C4B
FFC0 / 3C4C FFC1 / 3C4D FFC2 / 3C4E FFC3 / 3C4F
FFC4 / 3C50 FFC5 / 3C51 FFC6 / 3C52 FFC7 / 3C53
FFC8 / 3C54 FFC9 / 3C55 FFCA / 3C56 FFCB / 3C57
FFCC / 3C58 FFCD / 3C59 FFCE / 3C5A FFCF / 3C5B
FFD0 / 3C5C FFD1 / 3C5D FFD2 / 3C5E FFD3 / 3C5F
FFD4 / 3C60 FFD5 / 3C61 FFD6 / 3C62 FFD7 / 3C63
FFD8 / 3C64 FFD9 / 3C65 FFDA / 3C66 FFDB / 3C67
FFDC / 3C68 FFDD / 3C69 FFDE / 3C6A FFDF / 3C6B
FFE0 / 3C6C FFE1 / 3C6D FFE2 / 3C6E FFE3 / 3C6F
•Checking status by using SE panels:
a. From the HMC, select the CEC (under Systems Management) where the CHPID/PCHID you want to verify is stored, and click Single Object Operations (under Recovery task options).
Figure 9-3 SE Systems Management and channels
c. Look for the PCHID of which you are interested in checking the status. The result resembles the result that is shown in Figure 9-4.
Figure 9-4 Verifying channel CL5 by using CEC view
d. For more information about the PCHID, click the PCHID to show the details, as shown in Figure 9-5 on page 206.
Figure 9-5 CL5 PCHID details
You can also check the status from the LPAR view by selecting the LPAR that you want to check the CHPID status. Then, select the channels option under that LPAR. Now you can search for the CHPID and check the status, as shown in Figure 9-6.
Figure 9-6 SE Verify channel LPAR view
9.3 Integrated Coupling Adapter Short Range
This section describes implementing couplink links by using the Integrated Coupling Adapter Short Reach (ICA SR) feature.
9.3.1 ICA SR: Implementation
The CHPID type (CS5) was introduced with the z13 and uses a PCIe-Gen3 fanout feature that is named Integrated Coupling Adapter FC 0172.
The definition of this CF link is an example that it is part of the activity Define CF/STP link, as shown in Figure 1-3 on page 5.
A coupling link connection between a z14 ZR1 and a z14 that uses CS5 CHPIDs is shown in Figure 9-7.
Figure 9-7 CF link CS5 connection from z14 ZR1 to z14
The ICA SR is defined in IOCDS similar to PSIFB, by using an AID to identify the physical card. Example 9-4 shows a sample of the IOCP that is defining the CS5 CHPID.
Example 9-4 IOCP definitions for CHPID Type CS5 on z14
ID .. *
.. *
SYSTEM=(3906,1),LSYSTEM=CETUS, *
TOK=('CETUS',008001117A883907095804670118074F00000000,00*
000000,'18-03-15','09:58:04','........','........')
RESOURCE PARTITION=((CSS(0),(CETUS0A,A),.. *
(CSS(2),(CETUS2A,A),(CETUS2B,B),(CETUS2C,C),(CETUS2D,D),*
(CETUS2E,E),.. *
CHPID PATH=(CSS(2),E0),SHARED,PARTITION=((CETUS2E),(=)), *
CPATH=(CSS(1),E5),CSYSTEM=MUSCA,AID=20,PORT=1,TYPE=CS5
Example 9-5 shows a sample extract of the corresponding IOCP definition for a connecting CHPID Type CS5 on a z14 ZR1.
Example 9-5 IOCP definitions for CHPID Type CS5 on z14
ID .. *
.. *
SYSTEM=(3907,1),LSYSTEM=MUSCA, *
TOK=('MUSCA',008001117A883907095804670118074F00000000,00*
000000,'18-03-15','09:58:04','........','........')
RESOURCE PARTITION=((CSS(0),(MUSCA0A,A),.. *
USCA07,7),(MUSCA08,8),(MUSCA09,9)),(CSS(1),(MUSCA1A,A),(*
MUSCA1B,B),(MUSCA1C,C),(MUSCA1D,D),(MUSCA1E,E),(MUSCA1F,*
F),(MUSCA11,1),.. *
CHPID PATH=(CSS(1),E5),SHARED,PARTITION=((MUSCA11),(=)), *
CPATH=(CSS(2),E0),CSYSTEM=CETUS,AID=16,PORT=2,TYPE=CS5
CNTLUNIT CUNUMBR=FFFD,PATH=((CSS(1),E5,E9)),UNIT=CFP
IODEVICE ADDRESS=(FFD4,008),CUNUMBR=(FFFD),UNIT=CFP
IODEVICE ADDRESS=(FFDC,008),CUNUMBR=(FFFD),UNIT=CFP
For more information about how to define CHPID Type CS5 in HCD, see 14.2.3, “Defining a Coupling Facility link with CS5 CHPIDs” on page 352.
9.3.2 ICA SR: Managing the configuration
After activating the new configuration with the ICA SR CF links and all cables are connected, verify whether the CHPIDs are online and operating by using z/OS or the SE panels:
•Checking status by using the z/OS command:
Example 9-6 Display status of CHPID E1
D M=CHP(E1)
IEE174I 16.20.22 DISPLAY M 758
CHPID E1: TYPE=33, DESC=COUPLING OVER PCIE, ONLINE
COUPLING FACILITY 003907.IBM.02.000000007A88
PARTITION: 1F CPCID: 00
NAMED CF77 CONTROL UNIT ID: FFFD
PATH PHYSICAL LOGICAL CHANNEL TYPE CAID PORT
E1 / 0503 ONLINE ONLINE CS5 8X-PCIE3 0020 02
COUPLING FACILITY SUBCHANNEL STATUS
TOTAL: 48 IN USE: 48 NOT USING: 0 NOT USABLE: 0
OPERATIONAL DEVICES / SUBCHANNELS:
FFA4 / 3C40 FFA5 / 3C41 FFA6 / 3C42 FFA7 / 3C43
FFA8 / 3C44 FFA9 / 3C45 FFAA / 3C46 FFAB / 3C47
FFBC / 3C48 FFBD / 3C49 FFBE / 3C4A FFBF / 3C4B
FFC0 / 3C4C FFC1 / 3C4D FFC2 / 3C4E FFC3 / 3C4F
FFC4 / 3C50 FFC5 / 3C51 FFC6 / 3C52 FFC7 / 3C53
FFC8 / 3C54 FFC9 / 3C55 FFCA / 3C56 FFCB / 3C57
FFCC / 3C58 FFCD / 3C59 FFCE / 3C5A FFCF / 3C5B
FFD0 / 3C5C FFD1 / 3C5D FFD2 / 3C5E FFD3 / 3C5F
FFD4 / 3C60 FFD5 / 3C61 FFD6 / 3C62 FFD7 / 3C63
FFD8 / 3C64 FFD9 / 3C65 FFDA / 3C66 FFDB / 3C67
FFDC / 3C68 FFDD / 3C69 FFDE / 3C6A FFDF / 3C6B
FFE0 / 3C6C FFE1 / 3C6D FFE2 / 3C6E FFE3 / 3C6F
•Checking status by using SE panels:
a. From the HMC, select the CEC (under Systems Management) where is the CHPID/PCHID you want to verify, and click Single Object Operations (under Recovery task options).
Figure 9-8 SE Systems Management and channels
c. Look for the PCHID for which you are interested in checking the status. The result resembles the result that is shown in Figure 9-9.
Figure 9-9 Verifying channel CS5 using CEC view
d. For more information about the PCHID, click the PCHID to show a result (see Figure 9-10).
Figure 9-10 CS5 PCHID details
Another option is to check the status from the LPAR view by selecting the LPAR for which you want to check the CHPID status, and select the Channels option under that LPAR. Now, you can search for the CHPID and check the status, as shown in Figure 9-11.
Figure 9-11 SE Verify channel LPAR view
9.4 Defining an STP timing-only link by using ICA SR
This section describes how to configure timing-links (for STP messages) over ICA SR connectivity.
9.4.1 STP timing only: Implementation
In this section, a CHPID type (CS5) connection is used to show an example of STP timing-only link definition.
The definition of the STP timing-only link is part of the Define CF/STP link activity that is shown in Figure 1-3 on page 5.
A coupling link connection between a z14 ZR1 and a z14 that uses CS5 CHPIDs is used as an example of how to define an STP timing-only links (see Figure 9-12).
Figure 9-12 STP timing only link connection from z14 ZR1 to z14 (over CS5)
Timing-only link ICA SR is defined in IOCDS similar to ICA SR. The only difference is the control unit type, which is STP in this case. Example 9-7 shows a sample IOCP that defines the CS5 CHPID for timing-only links.
Example 9-7 IOCP defining STP timing only link on a z14 using CS5
ID .. *
.. *
SYSTEM=(3906,1),LSYSTEM=CETUS, *
TOK=('CETUS',00800112E0F73906094704240118061F00000000,00*
000000,'18-03-02','09:47:04','SYS9','IODF78')
RESOURCE PARTITION=((CSS(0),(CETUS0A,A),.. *
(CSS(2),(CETUS2A,A),(CETUS2B,B),(CETUS2C,C),(CETUS2D,D),*
(CETUS2E,E),(CETUS2F,F),(CETUS21,1),.. *
CHPID PATH=(CSS(2),E0),SHARED,PARTITION=((CETUS21),(=)), *
CPATH=(CSS(1),E5),CSYSTEM=MUSCA,AID=20,PORT=1,TYPE=CS5
CHPID PATH=(CSS(2),E1),SHARED,PARTITION=((CETUS21),(=)), *
CPATH=(CSS(1),E0),CSYSTEM=MUSCA,AID=20,PORT=2,TYPE=CS5
CNTLUNIT CUNUMBR=FFFD,PATH=((CSS(2),E9,ED,E1,E4,E5,E0)), *
UNIT=STP
The sequence of steps to define the STP timing only link connection between a z14 ZR1 and a z14 that uses CS5 CHPIDs by using HCD is the same as for defining the CF links to a connection by using CS5 CHPIDs. Complete the procedure as described in 9.3.2, “ICA SR: Managing the configuration” on page 208. Consider the following points:
•On the CF links connection step, after including the data that is related to the CPC CETUS side of the connection, enter YES on the Timing-only link option of the Connect to CF Channel Path panel (see Figure 9-13) and press Enter.
Figure 9-13 STP timing only link from z14 ZR1 to z13 by using CS5
•After accepting or overriding the Control Unit and Device numbers for both processors, HCD returns to the CF Channel Path Connectivity List panel (see Figure 9-14). You can see that only the STP timing links are now connected.
Figure 9-14 Only STP Timing links connected
9.4.2 STP timing-only links: Managing the configuration
After activating the new configuration with the ICA SR links that are defined as STP timing-only links and after all cables are connected, verify whether the CHPIDs are online and operating by using z/OS or the SE panels.
The same process that you used for ICA SR links is followed to check the status (online and operating) of the CS5 CHPIDs that were defined to work as STP timing-only links.
Complete the steps that are described in 9.3.2, “ICA SR: Managing the configuration” on page 208 by using the respective CHPID/PCHID of the links you are interested in checking the status.
9.5 CF LPAR setup and CFCC Level 22
In this section, we remind you of some configuration aspects that are related to a Coupling Facility LPAR, such as THIN INTERRUPT, MODE, memory considerations because of the new CF Level, and the changes in CFCC Level 22.
As shown in Figure 1-3 on page 5, the following aspects must be considered for the activity Setup CF LPAR:
•THIN INTERRUPT
The use of this enhancement can improve the responsiveness of shared ICFs or CPs that are running CFCC in a test or development environment.
However, production environments often use dedicated ICFs or dedicated CPs.
•MODE
The default setting for a 3907 Coupling Facility Partition is Volatile. Ensure that plans are in place to set this setting to Non-volatile, if required.
•Memory considerations
Memory planning must consider the CFCC memory and structure size increases that are associated with a new level of the Coupling Facility Control Code (CFCC).
LPARS that are running the CFCC code include increased storage requirements:
– CF Level 21 (or earlier) to CF Level 22
Also, as in previous CF Levels, ensure that the CF LPAR includes at least 512 MB of storage for CFCC code.
9.5.1 Coupling Facility Level 22
The new CFCC Level 22 introduces changes and improvements in the following areas:
•Coupling Facility processor scalability
Coupling Facility work management and dispatcher changes to allow improved efficiency as processors are added to scale up the capacity of a CF image:
– Non-prioritized (FIFO-based) work queues
– Master/subordinate system-managed duplexing protocol Processor scalability for CF images with five or more dedicated processors
– Use of improved instruction set that was introduced with z14
These changes automatically apply to all CF images on z14 ZR1
•XCF/XES List Notification Enhancements:
– CF list structures support three notification mechanisms to inform users about the status of shared objects in the CF:
• List: This information is used by many users, including XCF Signaling
• Key-range: Used predominantly by IBM WebSphere® MQ shared queues
• Sublist notification: Used predominantly by IMS shared queues
– CF users requested the following enhancements to these notification mechanisms:
• Immediate/delayed round-robin notification for list and key-range notifications (requested by WebSphere MQ)
• Aggressive list and key-range notifications (requested by IBM WebSphere MQ)
• List full/not-full notifications (requested by XCF Signaling)
1 The link data rates do not represent the performance of the links. The actual performance depends on many factors, including latency through the adapters, cable lengths, and the type of workload.
2 The link data rates do not represent the performance of the links. The actual performance depends on many factors, including latency through the adapters, cable lengths, and the type of workload.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.