Extended distance solutions
This chapter describes architectural requirements and implementation for IBM Z platform connectivity over extended distances.
The following topics are covered in this chapter:
10.1 Unrepeated distances
This section lists the maximum unrepeated distance and link budget for each type of IBM Z fiber optic link. Longer distances are possible by using repeaters, switches, channel extenders, and wavelength division multiplexing (WDM).
In Table 10-1, a link is a physical connection over a transmission medium (fiber) that is used between an optical transmitter and an optical receiver. The maximum allowable link loss, or link budget, is the maximum amount of link attenuation (loss of light), expressed in decibels (dB), that can occur without causing a possible failure condition (bit errors). When you use multimode fiber, as the link data rate increases, the unrepeated distance and link budget decreases.
The link budget is derived from combining the channel insertion loss budget with the unallocated link margin budget. The link budget numbers are rounded to the nearest tenth of a dB.
Table 10-1 Fiber optic connections: unrepeated distances
|
Feature type
|
Fiber type
|
Link data rate
|
Fiber
bandwidth
MHz-km
|
Maximum distance1
|
Link
budget
dB
|
|
FICON LX
|
SM 9 µm
|
1 Gbps
|
N/A
|
10 km
|
7.8
|
|
1 Gbps
|
4 km
|
4.8
|
|||
|
2 Gbps
|
10 km
|
7.8
|
|||
|
2 Gbps
|
4 km
|
4.8
|
|||
|
4 Gbps
|
10 km
|
7.8
|
|||
|
4 Gbps
|
4 km
|
4.8
|
|||
|
8 Gbps
|
10 km
|
6.4
|
|||
|
16 Gbps
|
|
10 km
|
6.4
|
||
|
FICON SX
|
MM 62.5 µm
|
1 Gbps
|
200
|
300 m
|
3.00
|
|
MM 50 µm
|
500
|
500 m
|
3.85
|
||
|
2000
|
860 m
|
4.62
|
|||
|
MM 62.5 µm
|
2 Gbps
|
200
|
150 m
|
2.10
|
|
|
MM 50 µm
|
500
|
300 m
|
2.62
|
||
|
2000
|
500 m
|
3.31
|
|||
|
MM 62.5 µm
|
4 Gbps
|
200
|
70 m
|
1.78
|
|
|
MM 50 µm
|
500
|
150 m
|
2.06
|
||
|
2000
|
380 m
|
2.88
|
|||
|
MM 62.5 µm
|
8 Gbps
|
200
|
21 m
|
1.58
|
|
|
MM 50 µm
|
500
|
50 m
|
1.68
|
||
|
2000
|
150 m
|
2.04
|
|||
|
MM 62.5 µm
|
16 Gbps2
|
-
|
-
|
-
|
|
|
MM 50 µm
|
500
|
35 m
|
1.63
|
||
|
2000
|
100 m
|
1.86
|
|||
|
4700
|
125 m
|
1.95
|
|||
|
Gigabit Ethernet LX
|
SM 9 µm
|
1 Gbps
|
N/A
|
5 km
|
4.6
|
|
MM 62.5 µm3
|
500
|
550 m
|
2.4
|
||
|
MM 50 µmc
|
500
|
550 m
|
2.4
|
||
|
Gigabit Ethernet SX
|
MM 62.5 µm
|
1 Gbps
|
200
|
275 m
|
2.6
|
|
MM 50 µm
|
500
|
550 m
|
3.6
|
||
|
10-Gigabit Ethernet LR
|
SM 9 µm
|
10 Gbps
|
N/A
|
10 km
|
6
|
|
10-Gigabit Ethernet SR
|
MM 62.5 µm
|
10 Gbps
|
200
|
33 m
|
2.5
|
|
MM 50 µm
|
500
|
82 m
|
2.3
|
||
|
2000
|
300 m
|
2.6
|
|||
|
25-Gigabit Ethernet SR
|
MM 50 µm
|
25 Gbps
|
2000
|
70 m
|
2.2
|
|
4700
|
100 m
|
2.5
|
1 Some types of features might have extended distance when using RPQs.
2 FICON Express16S features support 16 Gbps on z14, z14 ZR1, z13, and z13s only. A FICON Express16S+ feature has been released for z14 only.
3 Requires fiber optic mode conditioning patch (MCP) cables.
The following notes apply to Table 10-1 on page 170:
•Single-Byte Command Code Sets Connection (SBCON) is the American National Standards Institute (ANSI) standard for the command set that is used by FICON over a Fibre Channel physical interface. It is also known as FC-SB.
•All industry-standard links (FICON, Gigabit Ethernet) follow published industry standards. The minimum fiber bandwidth requirement to achieve the distances that are listed is applicable for multimode (MM) fiber only. There is no minimum bandwidth requirement for single mode (SM) fiber.
•The bit rates that are given might not correspond to the effective channel data rate in a particular application because of protocol overhead and other factors.
•LC duplex and SC duplex connectors are keyed per the ANSI Fibre Channel Standard specifications.
•Mode-conditioning patch (MCP) cable is required to operate certain links over multimode fiber.
•The ISC-3 feature supports an LC duplex connector. A conversion kit (SM SC duplex receptacle to LC duplex connector) can be used to enable existing single mode SC duplex cable infrastructure.
•The FICON Express2 and FICON Express features allow an auto-negotiated link speed of either 1 Gbps or 2 Gbps. The specifications in the table are for 2 Gbps link rate. MCP cables are not supported for 2 Gbps links.
•The FICON Express4 features allow an auto-negotiated link speed of either 1 Gbps, 2 Gbps, or 4 Gbps. MCP cables are not supported for 2 Gbps or 4 Gbps links.
•The FICON Express8 and FICON Express8S features allow an auto-negotiated link speed of either 2 Gbps, 4 Gbps, or 8 Gbps. MCP cables are not supported by FICON Express8 and FICON Express8S features.
•The FICON Express16S features, available on z14 and z14 ZR1 (carry forward only), and also on z13 and z13s, allow an autonegotiated link speed of either 4 Gbps, 8 Gbps, or 16 Gbps. MCP cables are not supported by FICON Express16S features.
•The FICON Express16S+ features, available on z14 only, allow an autonegotiated link speed of either 4 Gbps, 8 Gbps, or 16 Gbps. MCP cables are not supported by FICON Express16S+ features.
•As light signals traverse a fiber optic cable, the signal loses some of its strength. Decibels (dB) is the metric used to measure light power loss. The significant factors that contribute to light power loss are the length of the fiber, the number of splices, and the number of connections. The amount of light power loss (dB) across a link is known as the link budget.
All links are rated for a maximum link budget (the sum of the applicable light power loss factors must be less than the link budget) and a maximum distance (exceeding the maximum distance causes undetectable data integrity exposures). Another factor that limits distance is jitter, but that is typically not a problem at these distances.
•Measure link budget and fiber bandwidth at the appropriate wavelength:
– Long wavelength (1300 nm)
– Short wavelength (850 nm)
For planning purposes, the following worst case values can be used to estimate the link budget. See the references listed and contact the fiber vendor for specific values, which might be different for your configuration:
– Link loss at 1300 nm = 0.50 db/km
– Link loss per splice = 0.15 db/splice (not dependent on wavelength)
– Link loss per connection = 0.50 db/connection (not dependent on wavelength)
•Deviations from these specifications (longer distance or link budget) might be possible. These deviations are evaluated on an individual basis by submitting a request for price quotation (RPQ) to IBM.
|
Note: For more information about extended distances, see 10.4, “Wavelength-division multiplexing” on page 177.
|
10.2 Fibre Connection
This section describes architectural requirements and implementation solutions for Fibre Connection (FICON) channel connectivity over unrepeated and repeated distances. The term FICON represents the architecture as defined by the InterNational Committee of Information Technology Standards (INCITS) and published as ANSI standards. FICON also represents the names of the IBM Z platform feature types:
•FICON Express16S+
•FICON Express16S
•FICON Express8S
•FICON Express8
•FICON Express4
•FICON Express2
•FICON Express
All feature types support a long wavelength (LX) laser version and a short wavelength (SX) laser version. They support native FICON (FC, FCTC) and Fibre Channel Protocol (FCP) channel modes.
For more information, see 4.3, “Connectivity” on page 62.
10.2.1 FICON unrepeated distance
The unrepeated distance that is supported by IBM Z FICON features depends on these factors:
•The feature port transceiver type (LX or SX)
•The fiber type being used:
– 9 µm single mode
– 50 µm or 62.5 µm multimode
Also, for multimode, the fiber bandwidth (MHz-km) of the fiber.
•The speed at which the feature port is operating
•Whether there are MCP cables in the fiber optic link
For more information, see Table 10-1 on page 170, and Planning for Fiber Optic Links, GA23-0367.
10.2.2 FICON repeated distance solutions
This section describes several extended distance connectivity solutions for FICON channel-attached I/O control units and devices.
The repeated distance for a FICON channel is IBM Z qualified to a maximum of 100 km. For all FICON features that use repeaters, the end-to-end distance between the FICON channel and the FICON Director port can be up to 100 km (62 miles).
The same end-to-end distance is also available between the FICON Director port and the control unit port. However, the overall end-to-end distance between the FICON channel and control unit is IBM Z qualified for up to 100 km (62 miles) only.
FICON data rate droop
For FICON channels, the channel-to-control unit end-to-end distance can be increased up to 100 km without data rate performance droop occurring if the FICON Director buffer credits are set appropriately. The number of buffer credits that are required depends on the link data rate and the maximum number of buffer credits that are supported by the FICON Director or control unit, and application and workload characteristics.
Although it is theoretically possible for FICON to maintain high bandwidth at distances greater than 100 km, these distances have not been qualified for use with IBM Z. They are achievable only if enough buffer credits exist to support the link speed.
The distance capability with FICON is becoming increasingly important as movement occurs towards remote I/O vaulting for disaster recovery, business continuity, and IBM Geographically Dispersed Parallel Sysplex (GDPS) for availability.
FICON LX channel path with one FICON Director
One FICON Director increases the maximum supported distance of a FICON LX channel path by using single-mode fiber optic cables (see Figure 10-1). For example, the maximum supported distance is 20 km (12.4 miles) with FICON Express8S LX (long wavelength) features for 2 Gbps1, 4 Gbps, and 8 Gbps LX links. The same is valid for the FICON Express16S features at 16 Gbps, available for z13 and the FICON Express16S+ features at 16 Gbps, and available for z14 only.
Figure 10-1 FICON LX path with one Director
FICON LX channel path with two cascaded FICON Directors
A FICON channel path can include a maximum of two cascaded FICON Directors. The maximum supported distance of the FICON channel path is FICON Director vendor specific. The use of extended distance long wavelength transceivers on the inter-switch links (ISL) between the FICON Directors might be required to achieve vendor-quoted distances. Each ISL requires one fiber trunk (two fibers) between the FICON Directors.
For example, by using the configuration in Figure 10-2 and assuming that the distance between the two FICON Directors is 10 km (6.21 miles), the maximum supported distance is 30 km (18.64 miles) with FICON Express8S 10KM LX features for 2 Gbps, 4 Gbps, and 8 Gbps LX links. The example is also valid for the FICON Express16S features at 16 Gbps, available for z14 (carry forward only) and z13 servers and the FICON Express16S+ features at 16 Gbps, available for z14.
Figure 10-2 FICON LX path with cascaded FICON Directors
FICON channel path: Transceiver intermix
A FICON channel path through one or two FICON Directors consists of multiple optical fiber links. Each link in the channel path can be either LX or SX, allowing the channel path to be made up of a mixture of link types. This configuration is possible because the FICON Director converts optical to electrical and back to optical (known as an OEO conversion, for optical-electrical-optical) of the channel path as it passes through the director.
|
Note: The transceiver type (LX or SX) at each end of a particular link must match.
|
WDM technologies
Other extended distance connectivity technologies are available to extend FICON and other link types, for example, WDM. WDM technology also provides increased flexibility in that multiple links and protocol types can be transported over a single dark fiber trunk. For more information, see 10.4, “Wavelength-division multiplexing” on page 177.
10.3 Coupling links
This section describes architectural requirements and implementation solutions for coupling link connectivity over unrepeated and repeated distances.
Coupling link unrepeated distance
Table 10-2 lists the maximum unrepeated distances and link data rates that are supported for coupling links on IBM Z platforms. For more information, see Table 10-1 on page 170, and the accompanying notes in IBM Z Planning for Fiber Optic Links (FICON/FCP, Coupling Links, Open Systems Adapters, and zHyperLink Express), GA23-1408.
Table 10-2 Coupling link unrepeated distance and link data rate support
|
Coupling link type
|
Maximum unrepeated distance
|
Link data rate
|
|
IC (internal)
|
N/A
|
Memory-to-memory
(the highest bandwidth)
|
|
ISC-3 peer mode at 2 Gbps
|
10 km
12 km1
|
2 Gbps
|
|
ISC-3 peer mode at 1 Gbps2
|
10 km
20 kma
|
1 Gbps
|
|
ICA SR
|
150 meters3
|
8 Gbps
|
|
Coupling Express LR
|
10 km
|
10 Gbps
|
|
12x IFB
|
150 meters
|
6 Gbps
|
|
1x IFB
|
10 km
|
2.5 Gbps or 5 Gbps
|
1 Requires RPQ IBM Z Extended Distance; select RPQ based on the Z generation: zEC12: 8P2581; z13s and zBC12: 8P2781; z14 and z13: 8P2981,.
2 Requires RPQ 8P2197. This RPQ provides an ISC-3 daughter card, which clocks at 1 Gbps in peer mode. This configuration allows the ISC-3 peer mode link to have an unrepeated distance extended up to 20 km.
Under certain conditions, RPQ 8P2263 (or RPQ 8P2340 for the z10 BC) might be required with RPQ 8P2197. Check with your IBM representative.
Under certain conditions, RPQ 8P2263 (or RPQ 8P2340 for the z10 BC) might be required with RPQ 8P2197. Check with your IBM representative.
3 150 meters distance is achieved by using OM4 fiber types only; with OM3 fiber, the distance is 100 meters maximum.
10.4 Wavelength-division multiplexing
WDM is a technique that is used to transmit several independent bit streams over a single fiber optic link (see Figure 10-3). It is an approach to opening up the conventional optical fiber bandwidth by breaking it up into many channels, each at a different optical wavelength (a different color of light). Each wavelength can carry a signal at any bit rate less than an upper limit defined by the electronics, typically up to several gigabits per second.
Figure 10-3 WDM transmission technique
The channels are protocol-independent, so a wide variety of protocols is supported, including FICON, FCP, coupling links, Server Time Protocol (STP), and Gigabit Ethernet.
The actual signal bandwidth that the electronics can handle over one wavelength is such a small fraction of the inter-channel spacing. Because of this feature, the signals do not interfere with one another and can therefore be multiplexed into a single fiber by using a passive grating multiplexer.
There are several extended distance uses of WDM technology:
•FICON/FCP channel connections to remote control units and devices
•LAN and network protocol connections to remote sites
•IBM System Storage® Metro Mirror (synchronous Peer-to-Peer Remote Copy (PPRC))
•IBM System Storage Global Mirror
•IBM System Storage z/OS Global Mirror (asynchronous Extended Remote Copy (XRC))
•Peer-to-Peer Virtual Tape Server (PtP VTS), a form of remote copying tape data
10.4.1 GDPS qualification
GDPS is an enterprise-wide continuous availability and disaster recovery automation solution that can manage recovery from planned and unplanned outages across distributed servers and IBM Z platforms. GDPS can be configured in either a single site or in a multisite configuration. It is designed to manage remote copy configuration between storage subsystems, automate Parallel Sysplex operational tasks, and perform failure recovery from a single point of control, improving application availability.
Historically, this solution was known as a Geographically Dispersed Parallel Sysplex. Today, GDPS continues to be applied as a general term for a suite of business continuity solutions, including those that do not require a dispersed or multisite sysplex environment.
GDPS supports the following forms of remote copy in multisite solutions:
•IBM Metro Mirror, synchronous Peer-to-Peer Remote Copy
•IBM Global Mirror, asynchronous Peer-to-Peer Remote Copy
•IBM z/OS Global Mirror, asynchronous XRC
The GDPS solution is also independent of disk vendors if the vendor meets the specific levels of IBM Metro Mirror, IBM Global Mirror, and IBM z/OS Global Mirror architectures. For more information, see the GDPS web page.
IBM supports only WDM products that are IBM Z qualified for use in GDPS solutions. To obtain this qualification, WDM vendors obtain licensed IBM patents, intellectual property, and know-how that are related to the GDPS architecture. This access allows vendors to use proprietary IBM protocols and applications that are used in a GDPS environment, including coupling and STP links, Metro Mirror, Global Mirror, and z/OS Global Mirror.
Licensing of IBM patents also provides the WDM vendor with technical information about future IBM releases. Qualified vendors typically license this information for an extended period, which allows them to subscribe to the latest GDPS architecture changes and to be among the first to market with offerings that support these features.
|
Note: Check with your WDM vendor for current licensing status.
|
In addition, these vendor products are tested and qualified by IBM technicians with the same test environment and procedures that are used to test the protocols that provide the required connectivity for a GDPS configuration. This testing includes functions, recovery, and, in certain cases, performance measurements. Having access to these test facilities allows IBM to configure a fully functional sysplex and to simulate failure and recovery actions that cannot be tested as part of a working customer environment.
IBM has the facilities to test and qualify these products with both current and previous generation equipment within the IBM Vendor Solutions Connectivity Lab in Poughkeepsie, New York, in the United States. This qualification testing allows IBM specialists to reproduce any concerns that might arise when using this equipment in a client’s application.
Components
The following GDPS components are used during the qualification process:
•IBM Parallel Sysplex
•IBM System Storage
•Optical Wavelength-Division Multiplexer (WDM)
•IBM System Storage Metro Mirror (PPRC), a synchronous form of remote copy
•IBM System Storage Global Mirror
•IBM System Storage z/OS Global Mirror (XRC), an asynchronous form of remote copy
Protocols
The following GDPS connectivity protocols are tested during the qualification process:
•Fibre Connection (FICON)
•Fibre Channel protocol (FCP)
•Fibre Channel InterSwitch Links (ISL)
•InterSystem Channel-3 (ISC-3) peer mode (2 Gbps)
•InterSystem Channel-3 (ISC-3) peer mode (1 Gbps) using RPQ 8P2197
•Server Time Protocol (STP)
•1x InfiniBand (IFB) coupling links
•10 Gigabit Ethernet and RDMA over Converged Enhanced Ethernet (RoCE and RoCE Express2) using Shared Memory Communications - RDMA (SMC-R)
Often, these tested protocols are used in non-GDPS environments as well. The robust testing that is performed during the qualification process provides a high level of confidence when you use these IBM Z qualified optical WDM vendor products in non-GDPS environments.
10.4.2 IBM Z qualified WDM vendor products
The latest list of qualified WDM vendor products can be found through IBM Resource Link.
Select Hardware products for servers in the resource link, and then, select the page titled System z Qualified Wavelength Division Multiplexer (WDM) products for GDPS solutions.
|
Note: It is important to select the particular WDM vendor link in Resource Link and download the qualification letter to verify the details about the WDM product, model, firmware level, and the IBM Z server models for which it is qualified.
|
10.5 References
The following publications contain information that is related to fiber optic link distance:
•Coupling Facility Channel I/O Interface Physical Layer, SA23-0395
•Planning for Fiber Optic Links, GA23-0367
•Fiber Transport Services Direct Attach Planning, GA22-7234
For more information about IBM Z connectivity, see this web page.
For more information about GDPS solutions, see these resources:
•IBM GDPS Family: An introduction to Concepts and Capabilities, SG24-6374
For more information about IBM Z qualified WDM vendor products, use this IBM Redbooks publications search result.
1 FICON Express16S+ features do not support 2 Gbps links.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.