Coupling links and common time
This chapter describes the connectivity options that support IBM Parallel Sysplex clustering technology and common time on IBM Z platforms. It covers the following topics:
9.1 IBM Z cluster technology
Clustering technology brings the power of parallel processing to business-critical applications. A Parallel Sysplex cluster consists of up to 32 IBM z/OS images, connected to one or more coupling facilities (CFs), using high-speed specialized links, called coupling links, for communication and time-keeping. The coupling facilities at the heart of the cluster enable high-speed, record-level read/write data sharing among the images in a cluster.
The primary purpose of coupling links is to support communications between z/OS and coupling facilities. The CF images provide critical locking/serialization, data caching and coherency, and messaging and queueing capabilities, which allow the systems in the sysplex to cooperate and share data.
When configured properly, a cluster has no single point of failure and can provide users with near-continuous application availability over planned and unplanned outages.
Figure 9-1 shows a Parallel Sysplex configuration with the Server Time Protocol (STP) feature. STP provides time synchronization for multiple servers and coupling facilities. The use of Network Time Protocol (NTP) servers as an external time source is supported by STP.
|
Note: A time synchronization mechanism, STP is a mandatory hardware requirement for a Parallel Sysplex environment that consists of more than one IBM Z platform.
|
Figure 9-1 Parallel Sysplex using stand-alone CFs and STP
STP is a server-wide facility that is implemented in the Licensed Internal Code (LIC) presenting a single view of time to IBM Processor Resource/Systems Manager™ (IBM PR/SM™). It is a message-based protocol in which timekeeping information is passed over coupling links between CPCs.
The CPCs are configured with coupling peer mode links, such as the Integrated Coupling Adapter (ICA SR), Coupling Express Long Reach (CE LR), InfiniBand coupling links, InterSystem Channel peer (ISC-3) link, and the Internal Coupling peer (IC) link. All of these links except the IC link are supported by STP.
With these components, coupling performance is balanced with the performance of z/OS images that are running on IBM Z platforms in a common time cluster.
A CF runs the Coupling Facility Control Code (CFCC) that is loaded into main storage at the time the CF image is activated. CFCC can run on a stand-alone CF CPC or in a logical partition.
9.1.1 Coupling links and STP
STP is a message-based protocol in which STP timekeeping information is passed over externally defined coupling links. Coupling links that can be used to transport STP messages are the Coupling Express LR, ICA SR coupling links, InfiniBand (IFB) coupling links, and InterSystem Channel-3 (ISC-3) links in peer mode.
|
Reminder: ISC-3 is not supported on the IBM z13 and later platforms. The zEC12 and zBC12 were the last platforms to support ISC-3 features.
Also, IBM z14 Model ZR1 does not support InfiniBand coupling links (PSIFB).
|
Timing-only coupling links
For a CPC that is not part of a Parallel Sysplex, but required to be time-synchronized, coupling links must be configured for the CPC to be part of a Coordinated Timing Network (CTN). Hardware configuration definition (HCD) has been enhanced to support timing-only links to be defined when a CF does not exist at either end of the coupling link. The timing-only links can be of type CL5 for CE LR, CS5 for ICA SR coupling links, CIB for InfiniBand coupling links, or CFP for ISC-3 in peer mode. The control unit type is STP, and no devices are defined to it.
|
Note: CF messages cannot be transferred over timing-only links.
|
Coupling link redundancy for STP
Between any two CPCs that are intended to exchange STP messages, each CPC must be configured with at least two coupling links for redundancy reasons. This configuration prevents the loss of one link, causing the loss of STP communication between the CPCs. As mentioned, if a CPC does not have a CF logical partition, timing-only links can be used to provide STP connectivity.
The maximum number of attached CPCs that are supported by any STP-configured CPC in a CTN is equal to the maximum number of coupling links that are supported by the server in the configuration.
For more information, see the following publications:
•Server Time Protocol Planning Guide, SG24-7280
•Server Time Protocol Implementation Guide, SG24-7281
•Server Time Protocol Recovery Guide, SG24-7380
9.1.2 Multi-site sysplex considerations
If a sysplex is configured across two or more sites, you must plan to extend the Coupling Express LR, 1x IFB, or ISC-3 links beyond the distance that is supported without repeaters. IBM supports Wavelength Division Multiplexing (WDM) products that are qualified by IBM for use in multisite sysplex solutions, such as IBM Geographically Dispersed Parallel Sysplex (GDPS).
If any messages are to be transmitted across multiple sites through WDM products, ensure that only qualified WDM products and supported configurations are used.
For a list of WDM vendor products that are qualified for coupling links (transporting CF or STP messages) in a multi-site sysplex, see 10.4.2, “IBM Z qualified WDM vendor products” on page 179.
Also, see Resource Link for a list of qualified WDM products (requires a Resource Link ID).
9.2 Connectivity options
From a hardware perspective, when configuring a cluster, connectivity is a primary area of focus, as are other basic hardware components, such as the CF, Server Time Protocol, and coupling links. In an IBM Parallel Sysplex, the objective is any-to-any connectivity and nondisruptive planned or unplanned outages. All channels and directors must be included in the design. The primary reasons for this inclusion are to ensure cross-connectivity to all devices in the cluster and to potentially eliminate the effects of any outages.
For availability purposes, use at least two coupling links to connect each server to each CF in a Parallel Sysplex cluster. The performance, availability, and distance requirements of a Parallel Sysplex environment are the key factors that identify the appropriate connectivity option for any particular configuration.
9.2.1 Coupling link options
To ensure that sysplex environments continue to scale with today's necessary efficiency as CPC performance grows, coupling link design is continuously enhanced.
|
Attention: Parallel Sysplex technology supports connectivity between systems that differ by up to two generations (n-2). For example, an IBM z14 can participate in an IBM Parallel Sysplex cluster with z13, z13s, zEC12, and zBC12 platforms.
However, the IBM z14 ZR1 does not support InfiniBand connectivity, as such, an IBM z14 ZR1 does support connectivity to only one generation back (n-1): z14 ZR1 can connect to z14, z13, and z13s, using Integrated Coupling Adapter Short Reach (ICA SR) and Coupling Express Long Reach (CE LR) features.
|
Supported on the IBM z13 and newer Z platforms, the ICA SR, CS5 coupling links are available. CS5 coupling links support connectivity between any z14, z14 ZR1, z13, or z13s to another z14, z14 ZR1, z13, or z13s
Supported on the IBM z131 and newer Z platforms, the Coupling Express LR, CL5 coupling links are available. CL5 coupling links support connectivity between any z14, z14 ZR1, z13, or z13s to another z14, z14 ZR1, z13, or z13s
On the z14, z13, z13s, and zEnterprise CPCs (zEC12, zBC12) platforms IFB2 coupling links are available. IFB coupling links support peer mode connectivity between z14, z13, zEC12, zBC12, to any z14, z13, zEC12, or zBC12 (12x IFB and 1x IFB)
As shown in Figure 9-2, the following supported connections are included:
•From IBM Z general-purpose platforms to IBM Z CF, whereby the CF can be hosted in a CF-only system or run in logical partition mode with other operating systems partitions on an IBM Z system.
•Within an IBM Z platform, an IC link uses memory-to-memory data transfers between logical partitions (LPARs).
Figure 9-2 Parallel Sysplex connectivity options
Table 9-1 lists the coupling link support for each IBM Z platform. Restrictions on the maximum numbers can apply, depending on the configuration. Always check with your IBM support team.
Table 9-1 Coupling link support
|
Feature
|
Maximum supported links1
|
|||||||
|
z14
|
z14 ZR12
FC 0636/7/8/9
|
z13
|
z13s
|
zEC12
|
zBC12
|
|||
|
N20
|
N10
|
H13
|
H06
|
|||||
|
IC
|
32
|
32
|
32
|
32
|
32
|
32
|
32
|
32
|
|
ISC-3
|
N/A
|
N/A
|
N/A
|
N/A
|
N/A
|
483
|
||
|
ICA SRa
|
80*
|
4/8/16/16
|
40*
|
16
|
8
|
N/A
|
N/A
|
N/A
|
|
CE LR
|
64
|
32
|
64
|
32
|
32
|
N/A
|
N/A
|
N/A
|
|
HCA3-O LR (1x IFB)a
|
64*
|
N/A
|
64*
|
32
|
32
|
64*
|
32
|
16
|
|
HCA3-O (12x IFB)a
|
32*
|
N/A
|
32*
|
16
|
8
|
32*
|
16
|
8
|
|
HCA2-O LR (1x IFB)
|
N/A
|
N/A
|
N/A
|
N/A
|
N/A
|
32d
|
12c
|
8c
|
|
HCA2-O (12x IFB)
|
N/A
|
N/A
|
N/A
|
N/A
|
N/A
|
32d
|
16c
|
8c
|
1 The maximum supported links vary depending on the IBM Z model (as such, the numbers are marked with an asterisk (*)).
2 z14 M/T 3907 has only one model: ZR1. The number of PCIe fanouts (in the CPC drawer, hosting ICA SR features) is determined by the feature: FC #0636 has one PU SCM and can drive up to two PCIe I/O features in the CPC drawer, FC #0637 - has two PU SCMs and can drive up to four PCIe I/O features, while FC 0638 and FC #0639 have four PU SCMs and can drive up to eight PCIe I/O features.
3 Only available if carried forward on upgrades.
4 32 with one I/O drawer, and 48 with two I/O drawers (RPQ 8P2733).
|
Notes: Consider the following points:
•With HMC V2.14.1, enabling and disabling InfiniBand ports can be done from the HMC as sysprog user.
•ISC-3 is not supported on IBM z13 or newer models.
•IBM z14 Model ZR1 does not support InfiniBand coupling links (PSIFB).
|
For more information about distance support for coupling links, see Chapter 10, “Extended distance solutions” on page 169.
9.2.2 Internal Coupling link
IC links are Licensed Internal Code-defined links to connect a CF to a z/OS logical partition in the same CPC. These links are available on all IBM Z platforms. The IC link is an IBM Z coupling connectivity option that enables high-speed, efficient communication between a CF partition and one or more z/OS logical partitions that are running on the same CPC. The IC is a linkless connection (implemented in LIC) and does not require any hardware or cabling.
An IC link is a fast coupling link that uses memory-to-memory data transfers. IC links do not have PCHID numbers, but do require CHPIDs.
IC links have the following attributes:
•They provide the fastest connectivity that is significantly faster than any external link alternatives.
•They result in better coupling efficiency than with external links, effectively reducing the CPC cost that is associated with Parallel Sysplex technology.
•They can be used in test or production configurations, reduce the cost of moving into Parallel Sysplex technology, and enhance performance and reliability.
•They can be defined as spanned channels across multiple CSSes.
•They are available at no extra cost (no feature code). Employing ICFs with IC links results in considerable cost savings when configuring a cluster.
IC links are enabled by defining CHPID type ICP. A maximum of 32 IC links can be defined on an IBM Z platform.
9.2.3 InterSystem Channel-3 (ISC-3)
ISC-3 provides the connectivity that is required for data sharing between the CF and the systems. These links also provide connectivity for STP messaging between any zEnterprise (zEC12 and zBC12) system to any zEnterprise system.
|
Note: zEnterprise 196 and zEnterprise 114 were the last systems to offer ordering of ISC-3 connections. ISC-3 is not supported on IBM z13 or later CPCs.
|
ISC-3 links support point-to-point connections (directly connecting CFs and systems) and require a unique channel definition at each end of the link.
The ISC-3 feature consists of a mother card (ISC-M feature code 0217) and one or two ISC-3 daughter cards (ISC-D feature code 0218). Each ISC-3 daughter card has two independent ports with one CHPID associated with each active port. A port on an ISC-3 daughter card is activated by using Licensed Internal Code - Control Code (LIC-CC) with feature code 0219.
Each ISC-3 port operates in peer mode. The ISC-3 port mode is defined by the CHPID type (CFP) by using the HCD and the IOCP. ISC-3 feature ports in peer mode provide connection from general-purpose models to coupling facilities (Figure 9-2 on page 157).
An ISC-3 port is defined in peer mode when its associated CHPID is defined as CHPID type CFP. Each feature port supports connection to another CPCs ISC-3 port through a 9-micron single mode fiber optic cable that is terminated with an LC connector. An ISC-3 port in peer mode operates at 2 Gbps and supports a maximum unrepeated distance of 10 km. RPQ 8P2197 (InterSystem Channel 3, or ISC-3, Long Distance option) provides an ISC-3 daughter card with two active ports that clock at 1 Gbps and support a maximum unrepeated distance of 20 km.
Under certain conditions, RPQ 8P2263 might be required along with RPQ 8P2197. Check with your IBM representative.
For more information about unrepeated distance support, see Table 10-1 on page 170. ISC-3 in peer mode supports STP.
9.2.4 Integrated Coupling Adapter Short Range
The ICA SR was introduced with the IBM z13. It allows z14, z14 ZR1, z13, or z13s connectivity to another z14, z14 ZR1, z13, or z13s. The ICA SR (FC 0172) is a two-port, short-distance coupling feature that uses coupling channel type: CS5. The ICA SR uses PCIe Gen3 technology, with x16 lanes that are bifurcated into x8 lanes for coupling.
The ICA SR is designed to drive distances up to 150 m and supports a link data rate of 8 GBps. It is designed to support up to four CHPIDs per port and seven subchannels (devices) per CHPID. The maximum number of ICA SR features is 40 for the z14, 20 for the z13, and eight for the z14 ZR1 and z13s.
The ICA SR feature configuration for each Z platform is as follows:
•Up to 10 ICA SR features and up to 20 ICA SR ports are supported in a z14 CPC drawer. A maximum of 80 ports are supported in the z14.
•Up to eight ICA SR features (16 ports) are supported in a z14 ZR1 (Feature dependent, see Table 9-1 on page 157).
•Up to 10 ICA SR features and up to 20 ICA SR ports are supported in a z13 CPC drawer. A maximum of 40 ports are supported in a z13.
•Up to 8 ICA SR features and a maximum of 16 ICA SR ports are supported in a z13s.
The ICA SR cannot be connected to an IFB coupling feature such as HCA3-O or HCA3-O LR InfiniBand coupling features. For distances up to 100 m, OM3 fiber can be used, and for distances up to 150 m OM4 fiber must be used.
For more information, see IBM Z Planning for Fiber Optic Links (FICON/FCP, Coupling Links, and Open System Adapters), GA23-1407. This publication is available in the Library section of Resource Link.
9.2.5 Coupling Express Long Reach
The Coupling Express LR occupies one slot in a PCIe I/O drawer3 or PCIe+ I/O drawer4. It allows z14, z14 ZR1, z13, or z13s connectivity only to another z14, z14 ZR1, z13, or z13s. The Coupling Express LR (FC 0433) is a two-port, long-distance coupling feature that uses a new coupling channel type: CL5.
The Coupling Express LR is a PCIe feature. It is designed to drive distances up to 10 km (unrepeated) or up to 100 km with a qualified DWDM, and supports a link data rate of 10 Gbps and connects using the same 9 µm, Single Mode fiber type as 1X IFB and ISC-3.
Coupling Express LR is used in a Point-to-Point topology (just like 1X IFB and ISC-3) and it cannot be used in a switched environment.
Coupling Express LR is designed to support up to four CHPIDs per port, 32 buffers (that is, 8 or 32 subchannels) per CHPID. The Coupling Express LR feature resides in the PCIe I/O drawer on IBM z14, z13, and z13s and in a PCIe+ I/O drawer in the z14 ZR1. Up to 32 Coupling Express LR features and up to 64 ports are supported on a z14 and z13. Up to 16 Coupling Express LR features and up to 32 ports are supported on a z14 ZR1 and z13s.
The Coupling Express LR cannot be connected to a 1 x IFB coupling feature such as HCA3-O LR InfiniBand coupling feature.
For more information, see IBM Z Planning for Fiber Optic Links (FICON/FCP, Coupling Links, Open Systems Adapters, and zHyperLink Express), GA23-1408. This publication is available in the Library section of Resource Link.
9.2.6 InfiniBand coupling links
|
Note: The IBM z14 (M/T 3906) is be the last IBM Z server to support any kind of InfiniBand coupling connectivity.
|
For more information about long distance coupling, see Hardware Announcement z Systems Long Distance Coupling, 117-031.
IFB coupling links are high-speed links on z14, z13, z13s, zEC12, and zBC12. The IFB coupling links originate from four types of fanouts:
•HCA3-O (FC 0171) for 12x IFB links on z14, z13, z13s, and zEnterprise CPCs
•HCA3-O LR (FC 0170) for 1x IFB links on z14, z13, z13s, and zEnterprise CPCs
•HCA2-O (FC 0163) for 12x IFB links on zEnterprise CPCs
•HCA2-O LR (FC 0168) for 1x IFB links on zEnterprise CPCs
Each fanout that is used for coupling links has an assigned adapter ID number (AID) that must be used for definitions in IOCDS to have a relation between the physical fanout location and the CHPID number.
12x IFB coupling links
The HCA2-O and HCA3-O fanout support IFB coupling links that operate at 6 GBps (12x IFB). IFB coupling links use a fiber optic cable that is connected to a HCA2-O or HCA3-O fanout. The maximum distance for an IFB link is 150 meters. The fiber cables are industry standard OM3 50/125 micrometer multimode optical cables with Multifiber Push-On (MPO) connectors. 12x IFB supports seven or 32 subchannels5 per CHPID.
12x IFB and 12x IFB3 protocols
Two protocols are supported by the HCA3-O for 12x IFB feature:
•12x IFB3 protocol
When HCA3-O fanouts are communicating with HCA3-O fanouts and have been defined with four or fewer CHPIDs per port, the 12x IFB3 protocol is used.
•12x IFB protocol
If more than four CHPIDs are defined per HCA3-O port or HCA3-O features are communicating with HCA2-O features on zEnterprise servers, links run with the 12x IFB protocol.
The HCA3-O feature that supports 12x InfiniBand coupling links improves service times. When no more than four CHPIDs are defined per HCA3-O port, the 12x IFB3 protocol is used. When using the 12x IFB3 protocol, synchronous service times for lock structures are up to 40% faster than when using the 12x IFB protocol.
1x IFB coupling links
The HCA2-O LR and HCA3-O LR fanout support 1x IFB coupling links that operate at up to 5.0 Gbps. 1x IFB coupling links use a fiber optic cable that is connected to a HCA2-O LR or HCA3-O LR fanout. The maximum unrepeated distance for a 1x IFB link is 10 km. When using repeaters, the maximum distance is up to 100 km. The fiber cables that are used for 1x IFB links are standard 9 µm single mode fiber optic cables with an LC duplex connector. 1x IFB supports seven subchannels per CHPID.
Fanout adapter ID
Unlike channels that are installed in an I/O cage, I/O drawer, or PCIe I/O drawer, which are identified by a PCHID number that is related to their physical location, coupling link fanouts and ports are identified by an AID. The adapter ID value depends on its physical location. The AID must be used to assign a CHPID to the fanout in the IOCDS definition. The CHPID assignment is done by associating the CHPID to an AID and port. For more information, see 2.1.6, “Adapter ID” on page 21.
|
Note: The IBM zEnterprise EC12 and the IBM zEnterprise BC12 were the last IBM Z platforms to support the following features as carry forward on an upgrade:
•HCA2-O fanout for 12x IFB coupling links (FC 0163)
•HCA2-O LR fanout for 1x IFB coupling links (FC 0168)
|
9.3 Time functions
There is a long-standing requirement for accurate time and date information in data processing. A single operating system has been replaced by multiple, coupled operating systems on multiple CPCs. This need evolved to a requirement for both accurate and consistent clocks among these systems. Clocks are said to be consistent when the difference or offset between them is sufficiently small. An accurate clock is consistent with a standard time source.
The IBM z/Architecture, STP, facilitates the synchronization of CPC time-of-day (TOD) clocks to ensure consistent time stamp data across multiple CPCs and operating systems. STP provides a way to synchronize TOD clocks in different CPCs with a centralized time reference. This setup, in turn, can be set accurately based on an international time standard (external time source). The architecture defines a time-signal protocol and a distribution network, which allows accurate setting, maintenance, and consistency of TOD clocks.
STP facility
STP is a server-wide facility that is implemented in the Licensed Internal Code of IBM Z platforms and coupling facilities. STP presents a single view of time to PR/SM and provides the capability for multiple CPCs and coupling facilities to maintain time synchronization with each other. Any IBM Z platform can be enabled for STP by installing the STP feature. Each CPC that is planned to be configured in a CTN must be STP-enabled.
|
IBM z14 ZR1 STP Connectivity: If a z14 ZR1 plays a CTN role (PTS/BTS/Arbiter), then the other CTN role playing CPCs (BTS/PTS/Arbiter) must have coupling connectivity to the z14 ZR1 (N, N-1 connectivity only).
|
NTP client support is available to the STP code on the Z platform. With this function, the CPCs can be configured to use an NTP server as an external time source. NTP server with pulse per second (PPS) is supported.
|
Note: See the white paper Important Considerations for STP Server Role Assignments in the IBM Techdocs Library if you configured an STP CTN with three or more servers.
|
STP recovery enhancements
The following are the recovery enhancements to STP:
•New graphical interface for managing STP environment.
•Enhanced Console Assisted Recovery (z14, z14 ZR1, z136, and z13s)
Enhanced Console Assisted Recovery (ECAR) speeds up the process of BTS takeover:
– When the Primary Time Server (PTS/CTS) encounters a checkstop condition, the CEC informs its Support Element (SE) and Hardware Management Console (HMC).
– The PTS SE recognizes the checkstop pending condition, and the PTS SE STP code is called.
– The PTS SE sends an ECAR request by using HMC to the BTS SE.
– The BTS SE communicates with the BTS to start the takeover.
•Starting with IBM z14, an additional STP stratum level is supported (Stratum 4). This additional stratum level has been implemented to alleviate the additional complexity and expense of system reconfiguration using system upgrades. It should only be used as a temporary state during reconfiguration. Environments should not run with systems at Stratum level 4 for extended periods of time because of the lower quality of the time synchronization.
•“Going away” signal
The HCA3 generation and ICA SR of host channel adapters (HCA3-O, HCA3-O LR, or ICA SR) send a reliable, unambiguous “going away” signal to indicate that the server on which the HCA3 is running is about to enter a Failed state (check stopped). When the going-away signal sent by the Current Time Server (CTS) in an STP-only CTN is received by the Backup Time Server (BTS), the BTS can safely take over as the CTS. It can do so without relying on the previous recovery methods of Offline Signal (OLS) in a two-server CTN or the Arbiter in a CTN with three or more servers.
This enhanced recovery method is available on the Z platform. It is available only if you have an ICA SR, HCA3-O, or HCA3-O LR on the CTS, respectively, communicating with an ICA SR, HCA3-O, or HCA3-O LR on the BTS. The available STP recovery design is still available for the cases when a “going away” signal is not received or for failures other than a server failure.
9.3.1 Dynamic Split and Merge for Coordinated Timing Network
With HMC 2.14.1, is possible to dynamically split a CTN or merge two CTNs by using the STP GUI that is available on the HMC. STP understands the images that are present on each CEC in the source STP networks and the sysplex affiliation (more or less) of each of those z/OS images. STP can use this information for integrated checking, which operates with z/OS toleration for CTN change.
CTN split
When splitting a CTN, STP checks to ensure that it does result in a sysplex that “splits” across the two resultant CTNs. New STP roles are automatically assigned in the split-off CTN during the process. STP connectivity and max-stratum checking is performed in both split-off CTNs. Checking results are previewed and confirmed before proceeding
CTN Merge
When merging two CTNs, STP roles are automatically maintained in the merged-into CTN while STP connectivity and max-stratum checking is performed in the merged CTN. Preview and confirm the results before proceeding.
9.3.2 Dynamic I/O reconfiguration for standalone CF CPCs
With z14 Driver Level 36, support for dynamic activation of a new or changed IODF on a standalone CF is supported without requiring a POR/IML of the standalone CF CPC and without requiring the presence of any z/OS or z/VM image running an HCD instance on the same CPC. This solution does not require the use of Dynamic Partition Manager (DPM) mode.
A new LPAR, called the MCS LPAR, is a firmware based appliance version of the HCD instance that is deployed on the standalone CF CPC. This LPAR is driven by an updated HCD/HCM that is running in a z/OS LPAR on a remote z14 system (driver level 36) and managed by a HMC V2.14.1 connected to the standalone CF CPC.
A Power-on Reset with an IOCDS that includes and establishes the MCS LPAR on the standalone CF CPC is needed before this new capability can be used. After this “last” POR is done on the standalone CF CPC, all subsequent dynamic I/O changes can be done without subsequent POR/IMLs.
|
HCD requirements: Dynamic I/O for Standalone CF CPC requires that both systems (driver and standalone CF CPC) are at driver level 36 or later, HCD fixes for the driver system, and HMC 2.14.1.
|
The dynamic I/O for standalone CF CPC is shown in Figure 9-3.
Figure 9-3 Dynamic I/O for Standalone CF CPC
9.3.3 NTP server with PPS support
STP can use an NTP server that has a PPS output signal as its external time source. This type of external time source device is available worldwide from several vendors that provide network timing solutions.
The NTP output of the NTP server must be connected to the SE LAN because the NTP client is running on the SE. Also, the PPS output of the same NTP server must be connected to the PPS input port on the External Clock Facility (ECF) card on zEC12 CPCs, or a port on the oscillator (OSC) card on Z platform. There is no automatic check for the correctness of this connection.
However, a manual approximation of this check can be achieved by having the user check the SE to confirm that the PPS is active, then unplugging the PPS cable from the NTP Server. The user could then check by using the SE to see that the PPS port is no longer receiving a signal. Then, the user could replug the cable to the NTP server and make sure that the SE indicates the PPS signal is now being received at the CPC. This process allows STP to confirm that the NTP server that is used for the source of time is the same one that the CPC is receiving the PPS signal from.
STP tracks to the PPS signal to maintain time accuracy for the CTN. STP maintains an accuracy of 10 microseconds to the PPS input signal. Several variables, such as the cable used to connect the PPS signal and the accuracy of the NTP server to its time source (GPS or radio signals), determine the ultimate accuracy of STP to Coordinated Universal Time.
zEC12 PPS support
Two ECF cards is a standard zEC12 systems feature. The ECF cards provide a dual-path interface for PPS support. A cable connection from the PPS port on the ECF card to the PPS output of the NTP server is required when the server is using STP and is configured in an STP-only CTN by using NTP with PPS as the external time source. Each of the two standard ECF cards has one PPS port for a coaxial cable connection to a PPS port on an NTP server.
z14, z14 ZR1, z13, z13s, and zBC12 PPS support
Each of the two oscillator cards in the IBM z14, z14 ZR1, z13, and z13s has a PPS port to provide connectivity to an NTP server for PPS support.
The two oscillator cards in the first processor drawer of the zBC12 provide a dual-path interface for PPS support. A cable connection from the PPS port on the oscillator card to the PPS output of the NTP server is required when the server is using STP and is configured in an STP-only CTN by using NTP with PPS as the external time source.
Each of the two standard oscillator cards has one PPS port for a coaxial cable connection to a PPS port on an NTP server.
Figure 9-4 on page 166 shows an expanded availability configuration with two ECF features in a zEC12. Each ECF card is connected to a different NTP server with PPS support:
•On z14 and z13, it is connected to the oscillator cards in the back of frame A.
•On z14 ZR1, the NTP server with PPS is connected to the oscillator cards (OSC) located in the CPC drawer (rear side of the CPC drawer).
•On z13s systems, the NTP server with PPS support is connected to the oscillator cards in the oscillator backplane located on the back of the frame.
•On zBC12 systems, it is connected to the oscillator cards in the first processor drawer.
Figure 9-4 NTP server with PPS: Expanded availability configuration
For more information about NTP support, see the Parallel Sysplex advantages web page.
9.3.4 Operating system support
Software requirements vary with the design and use of common time. All current IBM z/OS versions support Server Time Protocol (STP).
For more information about STP operating system support, see the STP tab on the Parallel Sysplex web page.
Consult the appropriate Preventive Service Planning (PSP) buckets (3906DEVICE, 3907DEVICE, 2964DEVICE, 2965DEVICE, 2828DEVICE, and 2827DEVICE) before implementation.
9.4 References
See the Parallel Sysplex web page for help understanding, planning, and implementing a Parallel Sysplex cluster.
The following publications provide detailed information about Parallel Sysplex:
•Planning for Fiber Optic Links, GA23-0367
•Server Time Protocol Planning Guide, SG24-7280
•Server Time Protocol Implementation Guide, SG24-7281
•Server Time Protocol Recovery Guide, SG24-7380
•Getting the Most Out of a Parallel Sysplex, SG24-2073
•IBM zEnterprise EC12 Technical Guide, SG24-8049
•IBM zEnterprise BC12 Technical Guide, SG24-8138
•IBM z13 Technical Guide, SG24-8251
•IBM z13s Technical Guide, SG24-8294
•IBM z14 Technical Guide, SG24-8451
•IBM z14 ZR1 Technical Guide, SG24-8651
1 Check the latest z13 HMC driver level information for feature support.
2 z14 is the last IBM Z server model to support InfiniBand coupling links.
3 Check the latest z13 HMC driver level information for feature support.
4 PCIe+ I/O drawer (FC 4001) has been introduced with z14 ZR1, which is built in a 19” rack. FC 4001 is the 19” form factor PCIe Gen 3 - based I/O drawer. and can host up to 16 PCIe I/O features (adapters). It is not supported on z14 or older servers. Also, the PCIe I/O drawer cannot be carried forward during and MES upgrade to z14 ZR1. z14 ZR1 supports ONLY PCIe+ I/O drawers, up to four.
5 Depending on the version of the HCD, the default setting for the subchannels is set to 7 or 32.
6 Check the latest z13 HMC driver level information for feature support.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.