Working with networks
This chapter describes the several miscellaneous tasks that you might need to perform and includes the following topics:
14.1 Setting up a private interconnect
Having networked communications between different hosts that belong to a certain group can be beneficial. For example, certain legal databases must communicate to machines that scan documents for legal issues. Or, a web server and a certain back-end machine might need to communicate with each other without interference from other machines. Before live relocation, it was sufficient to merely set up a VSWITCH without an external interface to accomplish these tasks.
However, when you try to run this interconnect between hosts that run on a cross-central processor complex (CPC) SSI cluster, the private interconnect must be able to connect the network on the guests regardless of which CPC the guest is running on.
14.1.1 Directory Network Authorization
APAR VM65925 for z/VM 6.4, included in the base of z/VM 7.1 and later, provides a new and simplified process for defining and connecting guests to VSWITCHes called Directory Network Authorization (DNA). Instead of separate definitions for a virtual NIC and its authorization, DNA enhances the NICDEF directory statement to include authorization information.
Before DNA, a VSWITCH was defined for user-based (the default) or port-based access control. In user-based mode, a guest is permitted or denied to connect to a VSWITCH, and all ports that are connected to a VSWITCH feature the same access characteristics (such as ACCESS or TRUNK mode, and VLAN memberships). In port-based mode, a guest is granted access to connect to specific port numbers on a VSWITCH, and those ports might include different configured access characteristics.
When DNA is enabled, all VSWITCHes operate like the previous port-based mode. A port definition that corresponds to the access that us implied through the NICDEF directory entry is automatically defined for a guest when that virtual NIC connects to the VSWITCH.
DNA is the default operation mode for VSWITCHes on a system where the APAR is installed. Therefore, setting up an interconnect network can now be done by creating:
•A VSWITCH (the old way)
•A VLAN on an existing VSWITCH (the new “DNA-enabled” way)
14.1.2 Creating a VSWITCH for interconnect
An easy way to connect the network on the guests is to set up a virtual LAN (VLAN) for each of the required private interconnects on the external network. For each of these VLANs, then create a VLAN-aware VSWITCH with PORTTYPE ACCESS.
Complete the following steps:
1. Set up a network switch that connects to the mainframe and configure all necessary VLANs as tagged VLANs to the attached port.
2. Find a free port triplet on the Open Systems Adapter (OSA) device, for example, for the devices 903 - 905.
3. Edit the system configuration and add the following statement to the end of the file:
DEFINE VSWITCH PRV01 RDEV 0903 ETH VLAN 75 PORTT ACCESS
4. Grant access to only the group of virtual machines that are on that network:
MODIFY VSWITCH PRV01 GRANT LINUXADM
MODIFY VSWITCH PRV01 GRANT LINUX5
5. Perform the same steps on all other members of the SSI.
6. Define a private Internet Protocol (IP) range for the group of hosts. It is a preferred practice to track the IP ranges and to not overlap them, even if the hosts do not connect to each other through a network.
14.1.3 Creating an interconnect VLAN on a VSWITCH
To use an existing VSWITCH for an interconnect VLAN, the VSWITCH must be VALN-aware. Consider the following points:
•If the VSWITCH is not yet VLAN-aware, it must be converted to being VLAN-aware before others VLANs can be added. Consult your network team about how this process is to be done. Some of the changes include the following examples:
– The network ports your VSWITCH attaches to change from Access ports to Trunk ports
– On your VSWITCH definition, you must specify the VLAN option and provide the Default VLAN ID. This is the VLAN ID to which the network team assigned all your VSWITCH traffic. When your VSWITCH becomes VLAN-aware, it becomes responsible for the default tagging instead of the network.
•If the VSWITCH is VLAN-aware, ask the network team to define a new VLAN to the network interfaces that are used by the VSWITCH. No definition is needed on the VSWITCH.
To attach the guests to the interconnect VLAN, they must have more virtual NICs defined. The extra NIC must have the VLAN keyword added, which specifies the VLAN ID of the interconnect VLAN. For example, the following example shows two NICs on a guest, one for normal network traffic and one for interconnect:
NICDEF 600 TYPE QDIO LAN SYSTEM MAINVSW
NICDEF 620 TYPE QDIO LAN SYSTEM MAINVSW VLAN 120
In this example, the VSWITCH was defined with VLAN DEFAULT 100, which assigns all network interfaces that are not specified with a VLAN ID to VLAN 100. The second interface specifies VLAN 120, which is the interconnect VLAN.
14.2 Creating a HiperSockets device between logical partitions
IBM HiperSockets devices can be used within a CEC to enable fast and secure connectivity between a Linux server and z/OS. The following actions are described:
14.2.1 Verifying HiperSockets hardware definitions
Connectivity requires a HiperSockets IQD CHPID and devices that can be accessed by both the z/OS LPAR and the Linux z/VM LPAR. In Figure 2-3 on page 62, we defined a HiperSockets connection CHPID F0 between z/OS LPAR A12 and z/VM LPAR A02 by using device 7000.
This diagram is defined in the following input/output configuration program (IOCP) statements:
CHPID PATH=(CSS(0,1,2,3),F0),SHARED, *
NOTPART=((CSS(0),(A04,A0C,A0D,A0E,A0F),(=)),(CSS(1),(A1B*
,A1D,A1E,A1F),(=)),(CSS(2),(A2E,A2F),(=)),(CSS(3),(A32,A*
3D,A3E,A3F),(=))),TYPE=IQD
CNTLUNIT CUNUMBR=7000, *
PATH=((CSS(0),F0),(CSS(1),F0),(CSS(2),F0),(CSS(3),F0)), *
UNIT=IQD
IODEVICE ADDRESS=(7000,16),UNITADD=00,CUNUMBR=(7000),UNIT=IQD
VM LPAR A02 and z/OS LPAR A12 can access the HiperSockets CHPID F0, and it is an IQD type.
14.2.2 Creating a TCP/IP stack on z/OS
To create a TCP/IP stack within z/OS to use the HiperSockets device, it is recommended to get assistance from your network team. For more information about HiperSockets connectivity, see IBM HiperSockets Implementation Guide, SG24-6816.
Complete the following steps to create a TCP/IP stack on z/OS:
1. Create a TCP/IP stack (which is called TCPIPF in this example) with a TCP/IP profile that uses the F0 CHPID:
VIEW TCPIPF.SC42.TCPPARMS(TCPPROF) - 01.05
Command ===>
000085
000086 DEVICE IUTIQDF0 MPCIPA
000087 LINK HIPERLF0 IPAQIDIO IUTIQDF0
000088
...
000090 HOME
000093 10.1.1.42 HIPERLF0
..
000097 BEGINROUTES
..
000102 ROUTE 10.1.1.0 255.255.255.0 = HIPERLF0 MTU 8192
000103 ENDROUTES
000104
000107 START IUTIQDF0
2. Put the CHPID identifier within the IUTIQDxx device statement. If it meets your sites’ requirements, place the CHPID identifier in the LINK statements. Give the link a HOME address and ROUTE address according to your site networking requirements. Start your TCPIPF address space that uses this profile.
3. Issue the command D TCPIP,TCPIPF,NETSTAT,DEVL to verify the link information.
14.2.3 Configuring the HiperSockets interface on Linux
Complete the following steps to create a TCP/IP stack on Linux:
1. Request a free HiperSockets triplet from your system administrator.
2. Log on as MAINT, and verify the availability of the triplet:
===> q 7000-7002
OSA 7000 FREE , OSA 7001 FREE , OSA 7002 FREE
Ready; T=0.01/0.01 16:11:43
3. Attach the HiperSockets devices to the Linux image by using virtual device numbers. The command is issued from Linux1 in this example:
===> attach 7004 LINUX2 E000
OSA 7000 ATTACHED TO LINUX1 E000
===> attach 7005 LINUX2 E001
OSA 7001 ATTACHED TO LINUX1 E001
===> attach 7003 LINUX E002
OSA 7002 ATTACHED TO LINUX1 E002
4. Verify the HiperSockets device type:
===> q 7003-7005
OSA 7003 ATTACHED TO LINUX2 E002 DEVTYPE HIPER CHPID F0 IQD
OSA 7004 ATTACHED TO LINUX2 E000 DEVTYPE HIPER CHPID F0 IQD
OSA 7005 ATTACHED TO LINUX2 E001 DEVTYPE HIPER CHPID F0 IQD
5. Make the changes permanent with the following DIRM commands:
===> DIRM FOR LINUX1 DEDICATE E000 7004
===> DIRM FOR LINUX1 DEDICATE E001 7005
===> DIRM FOR LINUX1 DEDICATE E002 7003
Using Red Hat Enterprise Linux 7.1
Complete the following steps to create the cio_ignore -r 0.0.e000,0.0.e001,0.0.e002 device:
1. From the Linux image, create a device group for the E000 devices:
# echo 0.0.e000,0.0.e001,0.0.e002 > /sys/bus/ccwgroup/drivers/qeth/group
2. Bring the device online:
# echo 1 > /sys/devices/qeth/0.0.e000/online
3. Get the name of the devices from this command:
# cat /sys/devices/qeth/0.0.e000/if_name
enccw0.0.e000
4. Create a network configuration file by using the nmcli command:
# nmcli con add type ethernet con-name hipersocket ifname enccw0.0.e000 ip4 10.0.0.1/21
# nmcli con mod hipersocket 802-3-ethernet.s390-nettype "qeth"
# nmcli con mod hipersocket 802-3-ethernet.s390-subchannels "0.0.e000,0.0.e001,0.0.0e002"
# znetconf -A
5. Verify the enccw0.0.e000 status with the ip and lsqeth command:
# ip addr show
...
3: enccw0.0.e000: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 8192 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 06:00:f0:09:00:03 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.1/21 brd 10.0.7.255 scope global enccw0.0.e000
valid_lft forever preferred_lft forever
inet6 fe80::400:f0ff:fe09:3/64 scope link
valid_lft forever preferred_lft forever
# lsqeth
Device name : enccw0.0.e000
-------------------------------------------------------------------------
card_type : HiperSockets
cdev0 : 0.0.e000
cdev1 : 0.0.e001
cdev2 : 0.0.e002
chpid : F0
online : 1
portname : no portname required
portno : 0
route4 : no
route6 : no
state : UP (LAN ONLINE)
priority_queueing : always queue 2
fake_broadcast : 0
buffer_count : 128
layer2 : 0
isolation : none
sniffer : 0
SUSE Linux Enterprise Server
If you use the SUSE Linux Enterprise Server distribution of Linux, perform the following steps:
1. Configure the second network interface card (NIC) with the qeth_configure command:
# qeth_configure -t hsi 0.0.7000 0.0.7001 0.0.7002 1
2. Check whether the device was created:
# cat /proc/net/dev
3. If hsi0 was created, you will see a file that is called /etc/sysconfig/network/ifcfg-hsi0. You will need to edit this file by using the following command:
# vi /etc/sysconfig/network/ifcfg-hsi0
BOOTPROTO='static'
IPADDR='10.1.1.46/24'
STARTMODE='onboot'
NAME='HIPERSOCKETS (0.0.7400)'
4. Start the hsi0 device with the ifup command:
# ifup hsi0
5. Check the status of the interface:
# ip a s hsi0
The HiperSockets device is now up.
14.2.4 Verifying connectivity
To verify that the HiperSockets device is functioning, perform the following steps:
1. Ping from z/OS UNIX Systems Services:
USER1 @ SC42:/u/user1>ping 10.1.1.43
CS V1R13: Pinging host 10.1.1.43
Ping #1 response took 0.000 seconds.
2. Ping from the Red Hat Enterprise Linux that runs on ITSOZVM1:
[root@virtcook1 etc]# ping 10.1.1.42
PING 10.1.1.42 (10.1.1.42) 56(84) bytes of data.
64 bytes from 10.1.1.42: icmp_seq=1 ttl=64 time=0.025 ms
3. Ping from the SUSE Linux Enterprise Server that runs on ITSOZVM2:
linuxadmin:/etc/sysconfig/network # ping 10.1.1.46
PING 10.1.1.46 (10.1.1.46) 56(84) bytes of data.
This process shows that the HiperSockets device is working.
14.3 Configuring a port group by using Link Aggregation Control Protocol
To aggregate multiple OSA-Express ports, port groups can be defined on z/VM and attached to a virtual switch. Traffic is distributed over the multiple ports automatically.
The initial support for VSwitch Link Aggregation required that the OSA ports were used only by a single z/VM VSWITCH and were not shared with another system (even a Linux guest in the same LPAR attaching to the OSA directly).
Later, the Multi-VSwitch Link Aggregation capability was introduced. This support, also known as Global VSwitch, allows a port group to be defined as shared between z/VM VSwitches, even across multiple LPARs.
14.3.1 Exclusive-mode port group
Connectivity by using a port group in Exclusive Mode requires OSA devices that are used by only one z/VM LPAR. This example uses four-port OSA express cards, which use two ports for each CHPID (see Figure 14-1 on page 422).
Figure 14-1 Port group Priv01 connectivity
|
Note: Port number 1, not port number 0, was used for this connection.
|
ITSOZVM1 port group priv1 has the following details:
CHPID 00 portnumber 1 OSA device 2106
CHPID 01 portnumber 1 OSA device 2126
ITSOZVM2 port group priv1 has the following details:
CHPID 00 portnumber 1 OSA device 2046
CHPID 01 portnumber 1 OSA device 2066
Use the following steps to accomplish this task:
1. Create the port group on the first SSI member (ITSOZVM1 in this example):
===> set port group priv01 join 2106.p01 2126.p01
2. Create the port group on the second SSI member (ITSOZVM2 in this example):
===> set port group priv01 join 2046.p01 2066.p01
|
Note: Consider the following points:
•Link Aggregation Control Protocol (LACP) is set as active, by default. To use LACP, the network switch needs LACP to be active on the ports to which the CHPIDs connect. If LACP is not active in the switch, the port group does not activate when it is defined to a VSWITCH.
•If you receive message HCPSWU2832E, the LPAR does not have exclusive use of the device. Another LPAR has the device online. However, exclusive use does not require that only one LPAR in the I/O configuration data set (IOCDS) has the CHPID defined and the devices dedicated.
|
3. Define the virtual switch by using the priv01 port group on all members of the SSI (ITSOZVM1 and ITSOZVM2 in this example):
===> define vswitch vswprv1 rdev none ethernet vlan aware group priv01 gvrp
14.3.2 Multiple VSWITCH Link Aggregation
As you can see from the previous example, exclusive-mode link aggregation does not scale over many z/VM systems. The Global VSwitch capability relieved this limitation by allowing link aggregation OSA ports to be shared between z/VM LPARs.
To allow exclusive-mode link aggregation to scale over many z/VM systems, an IVL Domain is a group of up to 16 z/VM systems that can share link aggregation ports. Connectivity for IVL Domains is implemented over an Ethernet-based IVL Network, which is defined by using IVL VSwitches and OSA ports.
Up to 16 z/VM systems can be part of an IVL Domain, and an IVL Network can support many IVL Domains. Up to eight Domains are supported without the use of VLANs on the IVL Network; up to eight Domains per IVL VLAN are supported with VLANs on the IVL Network).
The ports used for the IVL Network must be separate from the link aggregation ports. However, because the IVL Network does not carry much traffic, OSA ports that are used for non-link-aggregated connections can be used for the IVL Network. However, it is highly recommended that the IVL Network be a separate VLAN from other network traffic.
The process that is used to create a Global VSwitch includes the overall tasks:
•Allocate ports for use by the IVL Network and the Global VSwitch.
•Define the IVL VSwitch on each of the z/VM systems (making sure to use the suitable Domain and VLAN definitions as required).
•Verify connectivity across the IVL Network.
•Define the Port Group by using the LACP ACTive SHAred option.
•Define the VSwitch by using the GLOBAL option.
Each of these steps is described next.
Allocating ports for the IVL Network and Global VSwitch
Decide which available ports are to be used for the IVL Network and the Global VSwitch. We recommend that the Global VSwitch be defined across the fastest cards that are available in your environment. Also, although it is not technically a requirement, we recommend that all the cards that are used in the Global VSwitch be the same type (or at least the same link speed).
Because the traffic requirement of the IVL Network is modest, OSA-Express 1000BaseT or Gigabit Ethernet cards can be used (even if the Global VSwitch uses 10 Gigabit cards).
Defining the IVL VSwitch on each system
The IVL VSwitch is defined by using the DEFINE VSWITCH command, as with a normal VSwitch, but the type of IVL must be used. Example 14-1 shows the result of creating the IVL VSwitch on the first of our systems.
Example 14-1 Creating the IVL VSwitch on the first system
define vswitch ivl type ivl rdev 1913.p01 1933.p01
VSWITCH SYSTEM IVL is created
HCPIVA3164I System RDBKZVMF is connected to IVL Domain A through VSWITCH IVL wit
h MAC address 02-00-00-00-00-00.
HCPSWU3230I Priority Queuing is not enabled for device 1913.P01 for VSWITCH SYST
EM IVL.
HCPSWU3230I Priority Queuing is not enabled for device 1933.P01 for VSWITCH SYST
EM IVL.
Ready; T=0.01/0.01 03:38:21
HCPSWU2830I VSWITCH SYSTEM IVL status is ready.
HCPSWU2830I DTCVSW3 is VSWITCH controller for device 1913.P01.
HCPSWU3181I The system is initiating IVL Member Discovery on IVL Domain A.
HCPSWU2830I DTCVSW4 is VSWITCH controller for backup device 1933.P01.
In our example, we used the default domain “A” and no VLAN support in the IVL VSwitch (our network switch provided VLAN isolation for us).
On the second system, we repeated the operation with the values for RDEV reversed. This configuration improves resilience by having each system use a different OSA port for its primary IVL Network connection. Example 14-2 shows the creation of the IVL VSwitch on the second system.
Example 14-2 Creating the IVL VSwitch on the second system
define vswitch ivl type ivl rdev 1933.p01 1913.p01
VSWITCH SYSTEM IVL is created
HCPIVA3164I System RDBKZVMC is connected to IVL Domain A through VSWITCH IVL wit
h MAC address 02-03-11-00-00-00.
Ready; T=0.01/0.01 05:28:07
HCPSWU2830I VSWITCH SYSTEM IVL status is ready.
HCPSWU2830I DTCVSW3 is VSWITCH controller for device 1933.P01.
HCPSWU3181I The system is initiating IVL Member Discovery on IVL Domain A.
HCPSWU3175I System RDBKZVMF has been added to IVL Domain A.
HCPSWU2830I DTCVSW4 is VSWITCH controller for backup device 1913.P01.
We also saw a message on the first system console to indicate that the second system was discovered by the first:
HCPSWU3175I System RDBKZVMC has been added to IVL Domain A.
Verifying IVL connectivity
We used the QUERY VSWITCH command to check the status of our IVL Network. Example 14-3 shows the result of this query.
Example 14-3 QUERY VSWITCH on the IVL VSwitch
q vswitch ivl
VSWITCH SYSTEM IVL Type: IVL Domain: A Maxconn: INFINITE
PERSISTENT RESTRICTED ETHERNET Accounting: OFF
PORTBASED LOCAL
VLAN Unaware
MAC address: 02-00-00-00-00-0E MAC Protection: Unspecified
IPTimeout: 5 QueueStorage: 8
Isolation Status: OFF VEPA Status: OFF
Uplink Port:
State: Ready PriQueuing: FORCED OFF
PMTUD setting: EXTERNAL PMTUD value: 9000 Trace Pages: 8
RDEV: 1913.P01 VDEV: 0609 Controller: DTCVSW3 ACTIVE
Adapter ID: 3907000BB4B7.0174
RDEV: 1933.P01 VDEV: 0603 Controller: DTCVSW4 BACKUP
Adapter ID: 3907000BB4B7.010C
IVL Port:
Adapter Owner: SYSTEM NIC: FFFD.P00 Name: UNASSIGNED Type: QDIO
RX Packets: 1132 Discarded: 1062 Errors: 0
TX Packets: 1388 Discarded: 0 Errors: 0
RX Bytes: 52312 TX Bytes: 152704
Device: FFFD Unit: 000 Role: DATA Port: 2100
Options: Ethernet Broadcast
Unicast MAC Addresses:
02-00-00-00-00-00
Multicast MAC Addresses:
03-FF-FF-FF-FF-01
This display shows the “IVL Port” section, which confirms it is an IVL VSwitch. The Multicast MAC Address that is listed ends in 01, which is the address that is assigned to IVL Domain A.
More information about the status of the IVL is shown by running the QUERY VMLAN command. Example 14-4 shows the VMLAN details from our first system.
Example 14-4 QUERY VMLAN command to show IVL details
q vmlan
VMLAN maintenance level:
Latest Service: Base
VMLAN MAC address assignment:
System MAC Protection: OFF
MACADDR Prefix: 020000 USER Prefix: 020000
MACIDRANGE SYSTEM: 000001-FFFFFF
USER: 000000-000000
VMLAN default accounting status:
SYSTEM Accounting: OFF USER Accounting: OFF
VMLAN general activity:
PERSISTENT Limit: INFINITE Current: 3
TRANSIENT Limit: INFINITE Current: 0
Trace Pages: 8
VMLAN Directory Network Authorization: ENABLED
IVL Domain: A MAC address: 03-FF-FF-FF-FF-01 VLAN: <none>
IVL Domain Heartbeat Timeout: 30
IVL Domain Capability: C000000000000000
Member: RDBKZVMF MAC address: 02-00-00-00-00-00
State: Active
Heartbeat Count: <local>
Member Capability: C000000000000000 Maintenance Level: V720
Member: RDBKZVMC MAC address: 02-03-11-00-00-00
State: Active
Heartbeat Count: 6
Member Capability: C000000000000000 Maintenance Level: V710
Connectivity to the members of the IVL Domain can be verified by using the SET VSWITCH IVL IVLPORT command. Example 14-5 shows the result when we tested connectivity in our test setup.
Example 14-5 SET VSWITCH IVL IVLPORT command
set vswitch ivl ivlport ping all
Command complete
HCPIVA3187I Pinging MAC 03-FF-FF-FF-FF-01 : bytes=8096 of data.
Ready; T=0.01/0.01 06:47:32
HCPIVA3187I Reply from MAC 02-03-11-00-00-00 : bytes=8096 time=0630 us
Defining the Shared Port Group
After the IVL Network was verified, we then defined the port group. Through the IVL Domain, the port group is created on the second system when it is defined on the first system.
Example 14-6 shows the commands that we issued on the first system to create the shared port group. First, we defined the group as a shared port group; then, we added the two OSA ports to the group.
Example 14-6 Using SET PORT GROUP commands to define a shared port group
set port group grpglbl lacp active shared
Port group GRPGLBL is created
Ready; T=0.01/0.01 07:05:26
set port group grpglbl join 1910 1930
Port group GRPGLBL is updated
Ready;
To verify that the IVL Domain was functioning correctly, we ran a QUERY PORT GROUP command on the second system. When running the QUERY PORT GROUP command without parameters, we saw the response No active groups found. This message is expected because the port group does not become active until it is connected to a VSwitch.
Displaying inactive port groups can be done by using the ALL INActive parameter on the QUERY PORT GROUP command, or by displaying the group by name.
We used the display-by-name method to show the group. As shown in Example 14-7, we saw that the group was defined on the second system automatically through the IVL.
Example 14-7 QUERY PORT GROUP command
q port group grpglbl
Group: GRPGLBL.0 Inactive LACP Mode: Active Shared
VSWITCH <none> ifIndex: 2112
Load Balancing: Collaborative Interval: 300
RDEV: 1930.P00 Adapter ID: 3907000BB4B7.010C
RDEV: 1910.P00 Adapter ID: 3907000BB4B7.0174
Ready;
We saw the same output when we issued the command on the first system.
Defining the Global VSwitch
A global port group can be defined in LACP mode only. We had to coordinate with our network management team to have the correct configuration made in our network switch (see 14.3.4, “Link Aggregation Control Protocol” on page 434 for more information about what can occur if the switch configuration is incorrect). After the switch setup was complete, we created the Global VSwitch on one of the systems (RCBKZVMC), as shown in Example 14-8.
Example 14-8 Creating a Global VSwitch on one system
define vswitch vswglbl ethernet group grpglbl global
VSWITCH SYSTEM VSWGLBL is created
Ready;
HCPSWU3165I Device 1930.P00 for VSWITCH VSWGLBL is now the Active LAG Port Contr
oller for shared port group GRPGLBL.
HCPSWU2830I VSWITCH SYSTEM VSWGLBL status is ready.
HCPSWU2830I DTCVSW1 is VSWITCH controller for device 1930.P00.
HCPSWU2855I Device 1930.P00 for VSWITCH VSWGLBL is enabled for port group GRPGLB
L by LACP.
HCPSWU3165I Device 1910.P00 for VSWITCH VSWGLBL is now the Active LAG Port Contr
oller for shared port group GRPGLBL.
HCPSWU2830I DTCVSW3 is VSWITCH controller for device 1910.P00.
HCPSWU2855I Device 1910.P00 for VSWITCH VSWGLBL is enabled for port group GRPGLB
L by LACP.
Over the course of a few seconds, the LACP negotiation starts with the network switch over the first port in the port group. This process is indicated by the first pair of HCPSWU2830I and HCPSWU2855I messages, followed by HCPSWU3165I when the LACP negotiation completes. Later (in our case it was about 20 - 30 seconds) the second port in the group is added and the second pair of HCPSWU2830I and HCPSWU2855I messages appears.
We performed a display of the new VSwitch by using QUERY VSWITCH DETAILS. Example 14-9 shows the output of this command.
Example 14-9 QUERY VSWITCH DETAILS on the Global VSwitch
q vswitch vswglbl details
VSWITCH RDBKZVMC.VSWGLBL Type: QDIO Connected: 0 Maxconn: INFINITE
PERSISTENT RESTRICTED ETHERNET Accounting: OFF
USERBASED GLOBAL
VLAN Unaware
MAC address: 02-03-11-00-00-07 MAC Protection: Unspecified
IPTimeout: 5 QueueStorage: 8
Isolation Status: OFF VEPA Status: OFF
Uplink Port:
State: Ready PriQueuing: OFF
PMTUD setting: EXTERNAL PMTUD value: 9000 Trace Pages: 8
Group: GRPGLBL.0 Active LACP Mode: Active Shared
RDEV: 1930.P00 VDEV: 0603 Controller: DTCVSW1 ACTIVE
Adapter ID: 3907000BB4B7.010C
Uplink Port Connection:
MAC address: 02-03-11-00-00-08
RX Packets: 82 Discarded: 0 Errors: 0
TX Packets: 114 Discarded: 0 Errors: 0
RX Bytes: 10168 TX Bytes: 14136
Device: 0603 Unit: 000 Role: DATA Port: 2049
Partner Switch Capabilities: No_Reflective_Relay
LAG Port Controller: Active
RDEV: 1910.P00 VDEV: 0603 Controller: DTCVSW3 ACTIVE
Adapter ID: 3907000BB4B7.0174
Uplink Port Connection:
MAC address: 02-03-11-00-00-09
RX Packets: 82 Discarded: 0 Errors: 0
TX Packets: 106 Discarded: 0 Errors: 0
RX Bytes: 10168 TX Bytes: 13144
Device: 0603 Unit: 000 Role: DATA Port: 2050
Partner Switch Capabilities: No_Reflective_Relay
LAG Port Controller: Active
Member: RDBKZVMC.VSWGLBL
State: Synchronized
Ready;
The VSwitch details display shows some information about the port group. The members of the Global VSwitch at the bottom of the display. Because only RDBKZVMC (the first system) has the Global VSwitch defined, it is the only system listed.
We ran a display of the port group to see if any other information was displayed there. The output of QUERY PORT GROUP VSWGLBL was much the same as we saw in the VSwitch display, but adding DETAILS to the query displayed much LACP information (see Example 14-10).
Example 14-10 QUERY PORT GROUP DETAILS output
q port group grpglbl details
Group: GRPGLBL.0 Active LACP Mode: Active Shared
VSWITCH RDBKZVMC.VSWGLBL ifIndex: 2112
Load Balancing: Collaborative Interval: 300
GROUP Information:
PORT Information - Total Frames per Interval:
Local Local Total
Device Status Previous Current Previous
------ ------- ------------ ------------ -----------------------
1930 Active 10 10 10
1910 Active 16 6 16
------------------------------------------------------------------
Total Port Group Frames: 26
Last Load Balance: RDBKZVMC Date: 10/18/20 Time: 02:59:37
ROUTING Information - Frame Distribution per Interval:
MAC Device Previous Current
----- --------- ------------ ------------
0 1930 0 0
1 1910 0 0
2 1930 0 0
3 1910 0 0
4 1930 0 0
5 1910 0 0
6 1930 0 0
7 1910 0 0
RDEV: 1930.P00 VDEV: 0603 Controller: DTCVSW1 ACTIVE
Adapter ID: 3907000BB4B7.010C
Uplink Port Connection:
MAC address: 02-03-11-00-00-08
RX Packets: 1153 Discarded: 0 Errors: 0
TX Packets: 1562 Discarded: 0 Errors: 0
RX Bytes: 142972 TX Bytes: 193688
Device: 0603 Unit: 000 Role: DATA Port: 2049
Partner Switch Capabilities: No_Reflective_Relay
LAG Port Controller: Active
PROTOCOL Counters:
LACP RX: 1149 Marker RX: 0
LACP TX: 1558 Marker TX: 4 Timeouts: 4
ACTOR Information:
System ID: 32768,02-03-11-00-00-07 Oper Key: 2
Port Priority: 32768 Port: 2049 Group Key: 2
State: 3D - LACP_Active Slow AGG SYNC DIST COLL
PARTNER Information:
System ID: 32768,A4-8C-DB-94-58-00 Oper Key: 101
Port Priority: 32768 Port: 9 Group Key: 101
State: 3D - LACP_Active Slow AGG SYNC DIST COLL
RDEV: 1910.P00 VDEV: 0603 Controller: DTCVSW3 ACTIVE
Adapter ID: 3907000BB4B7.0174
Uplink Port Connection:
MAC address: 02-03-11-00-00-09
RX Packets: 1147 Discarded: 0 Errors: 0
TX Packets: 1554 Discarded: 0 Errors: 0
RX Bytes: 142228 TX Bytes: 192696
Device: 0603 Unit: 000 Role: DATA Port: 2050
Partner Switch Capabilities: No_Reflective_Relay
LAG Port Controller: Active
PROTOCOL Counters:
LACP RX: 1147 Marker RX: 0
LACP TX: 1554 Marker TX: 0 Timeouts: 0
ACTOR Information:
System ID: 32768,02-03-11-00-00-07 Oper Key: 2
Port Priority: 32768 Port: 2050 Group Key: 2
State: 3D - LACP_Active Slow AGG SYNC DIST COLL
PARTNER Information:
System ID: 32768,A4-8C-DB-94-58-00 Oper Key: 101
Port Priority: 32768 Port: 7 Group Key: 101
State: 3D - LACP_Active Slow AGG SYNC DIST COLL
Member: RDBKZVMC
Scope: Synchronized
LAG Synchronization token: 0000000000000000
Mode: Connected
Member: RDBKZVMF
Scope: Synchronized
LAG Synchronization token: 0000000000000000
Mode: Inactive
Ready;
At the end of this details display, we can see that the status RDBKZVMC (the system we created the VSwitch on) is Connected, while the status of RDBKZVMF (the second system) is Inactive. This result occurs because the Global VSwitch is not yet created on the second system.
On the second system, we checked its view of the port group to make sure it was active and ready for use. We then issued the command to define the Global VSwitch on RDBKZVMF. These commands and their output are shown in Example 14-11.
Example 14-11 Setting up the Global VSwitch on the second system
q port group grpglbl
Group: GRPGLBL.0 Inactive LACP Mode: Active Shared
VSWITCH <none> ifIndex: 2112
Load Balancing: Collaborative Interval: 300
RDEV: 1930.P00 Adapter ID: 3907000BB4B7.010C
RDEV: 1910.P00 Adapter ID: 3907000BB4B7.0174
Ready;
define vswitch vswglbl ethernet group grpglbl global
VSWITCH SYSTEM VSWGLBL is created
Ready;
HCPSWU2830I VSWITCH SYSTEM VSWGLBL status is ready.
HCPSWU2830I DTCVSW3 is VSWITCH controller for device 1930.P00.
HCPSWU2855I Device 1930.P00 for VSWITCH VSWGLBL is enabled for port group GRPGLB
L by LACP.
HCPSWU2830I DTCVSW4 is VSWITCH controller for device 1910.P00.
HCPSWU2855I Device 1910.P00 for VSWITCH VSWGLBL is enabled for port group GRPGLB
L by LACP.
As we saw with the first system, the VSwitch is defined and becomes active after a short period. We issued Q VSWITCH DETAILS on RDBKZVMF. The results are shown in Example 14-12.
Example 14-12 QUERY VSWITCH DETAILS for the Global VSwitch on the second system
q vswitch vswglbl details
VSWITCH RDBKZVMF.VSWGLBL Type: QDIO Connected: 0 Maxconn: INFINITE
PERSISTENT RESTRICTED ETHERNET Accounting: OFF
USERBASED GLOBAL
VLAN Unaware
MAC address: 02-06-50-00-00-0B MAC Protection: Unspecified
IPTimeout: 5 QueueStorage: 8
Isolation Status: OFF VEPA Status: OFF
Uplink Port:
State: Ready PriQueuing: OFF
PMTUD setting: EXTERNAL PMTUD value: 9000 Trace Pages: 8
Group: GRPGLBL.0 Active LACP Mode: Active Shared
RDEV: 1930.P00 VDEV: 0603 Controller: DTCVSW3 ACTIVE
Adapter ID: 3907000BB4B7.010C
Uplink Port Connection:
MAC address: 02-06-50-00-00-0C
RX Packets: 50 Discarded: 0 Errors: 0
TX Packets: 4 Discarded: 0 Errors: 0
RX Bytes: 6200 TX Bytes: 496
Device: 0603 Unit: 000 Role: DATA Port: 2049
Partner Switch Capabilities: No_Reflective_Relay
LAG Port Controller: Standby
RDEV: 1910.P00 VDEV: 0603 Controller: DTCVSW4 ACTIVE
Adapter ID: 3907000BB4B7.0174
Uplink Port Connection:
MAC address: 02-06-50-00-00-0D
RX Packets: 48 Discarded: 0 Errors: 0
TX Packets: 0 Discarded: 0 Errors: 0
RX Bytes: 5952 TX Bytes: 0
Device: 0603 Unit: 000 Role: DATA Port: 2050
Partner Switch Capabilities: No_Reflective_Relay
LAG Port Controller: Standby
Member: RDBKZVMF.VSWGLBL
State: Synchronized
Member: RDBKZVMC.VSWGLBL
State: Synchronized
Ready;
We see the following important details in this display:
•The MAC addresses against each of the ports are different on each system. This result shows that although the Global VSwitch functions like a logical single VSwitch that is present in both systems, it is two separate VSwitches that share the global port group. For this reason, the VSwitch must be defined on each system, while the port group did not need to be defined.
•Now that the Global VSwtich is defined on our second system, RDBKZVMF now appears as Synchronized at the end of the QUERY VSWITCH DETAILS output.
For completeness, we repeated the Q PORT GROUP DETAILS display on RDBKZVMF to compare the output (see Example 14-13).
Example 14-13 QUERY PORT GROUP DETAILS display on the second system
q port group grpglbl details
Group: GRPGLBL.0 Active LACP Mode: Active Shared
VSWITCH RDBKZVMF.VSWGLBL ifIndex: 2112
Load Balancing: Collaborative Interval: 300
GROUP Information:
PORT Information - Total Frames per Interval:
Local Local Total
Device Status Previous Current Previous
------ ------- ------------ ------------ -----------------------
1930 Active 32 11 42
1910 Active 27 17 37
------------------------------------------------------------------
Total Port Group Frames: 79
Last Load Balance: RDBKZVMF Date: 10/18/20 Time: 08:12:57
ROUTING Information - Frame Distribution per Interval:
MAC Device Previous Current
----- --------- ------------ ------------
0 1930 0 0
1 1910 0 0
2 1930 0 0
3 1910 0 0
4 1930 0 0
5 1910 0 0
6 1930 0 0
7 1910 0 0
RDEV: 1930.P00 VDEV: 0603 Controller: DTCVSW3 ACTIVE
Adapter ID: 3907000BB4B7.010C
Uplink Port Connection:
MAC address: 02-06-50-00-00-0C
RX Packets: 1765 Discarded: 0 Errors: 0
TX Packets: 4 Discarded: 0 Errors: 0
RX Bytes: 218860 TX Bytes: 496
Device: 0603 Unit: 000 Role: DATA Port: 2049
Partner Switch Capabilities: No_Reflective_Relay
LAG Port Controller: Standby
PROTOCOL Counters:
LACP RX: 1765 Marker RX: 0
LACP TX: 0 Marker TX: 4 Timeouts: 4
ACTOR Information:
System ID: 32768,02-03-11-00-00-07 Oper Key: 2
Port Priority: 32768 Port: 2049 Group Key: 2
State: 3D - LACP_Active Slow AGG SYNC DIST COLL
PARTNER Information:
System ID: 32768,A4-8C-DB-94-58-00 Oper Key: 101
Port Priority: 32768 Port: 9 Group Key: 101
State: 3D - LACP_Active Slow AGG SYNC DIST COLL
RDEV: 1910.P00 VDEV: 0603 Controller: DTCVSW4 ACTIVE
Adapter ID: 3907000BB4B7.0174
Uplink Port Connection:
MAC address: 02-06-50-00-00-0D
RX Packets: 1767 Discarded: 0 Errors: 0
TX Packets: 0 Discarded: 0 Errors: 0
RX Bytes: 219108 TX Bytes: 0
Device: 0603 Unit: 000 Role: DATA Port: 2050
Partner Switch Capabilities: No_Reflective_Relay
LAG Port Controller: Standby
PROTOCOL Counters:
LACP RX: 1767 Marker RX: 0
LACP TX: 0 Marker TX: 0 Timeouts: 0
ACTOR Information:
System ID: 32768,02-03-11-00-00-07 Oper Key: 2
Port Priority: 32768 Port: 2050 Group Key: 2
State: 3D - LACP_Active Slow AGG SYNC DIST COLL
PARTNER Information:
System ID: 32768,A4-8C-DB-94-58-00 Oper Key: 101
Port Priority: 32768 Port: 7 Group Key: 101
State: 3D - LACP_Active Slow AGG SYNC DIST COLL
Member: RDBKZVMF
Scope: Synchronized
LAG Synchronization token: 0000000000000000
Mode: Connected
Member: RDBKZVMC
Scope: Synchronized
LAG Synchronization token: 0000000000000000
Mode: Connected
Ready;
At the bottom of the display we can see that both systems that are defined to the port group are now showing a status of Connected, which is what we expect now that the Global VSwitch is defined on both systems.
Looking closer, we can see that although the MAC address detail for each port corresponds to RDBKZVMF, the MAC addresses that are shown in the LACP ACTOR details areas are the MAC addresses of RDBKZVMC. This result shows that the first system to define the Global VSwitch becomes the system that manages the LACP protocol communication with the network switch.
14.3.3 Global VSwitch recovery
We tested what occurs if RDBKZVMC was removed from the group. We issued the SET VSWITCH VSWGLBL DISCONN command on RDBKZVMC to disconnect it from the port group. Example 14-14 shows the messages that appeared on RDBKZVMF when we issued the disconnect command, and the output of a port group detail display (with some lines removed).
Example 14-14 Global VSwitch recovery, and status display
HCPSWU3165I Device 1930.P00 for VSWITCH VSWGLBL is now the Active LAG Port Contr
oller for shared port group GRPGLBL.
HCPSWU3165I Device 1910.P00 for VSWITCH VSWGLBL is now the Active LAG Port Contr
oller for shared port group GRPGLBL.
q port group grpglbl details
Group: GRPGLBL.0 Active LACP Mode: Active Shared
VSWITCH RDBKZVMF.VSWGLBL ifIndex: 2112
Load Balancing: Collaborative Interval: 300
. . .
Last Load Balance: RDBKZVMF Date: 10/18/20 Time: 08:32:57
. . .
RDEV: 1930.P00 VDEV: 0603 Controller: DTCVSW3 ACTIVE
Adapter ID: 3907000BB4B7.010C
Uplink Port Connection:
MAC address: 02-06-50-00-00-0C
. . .
ACTOR Information:
System ID: 32768,02-03-11-00-00-07 Oper Key: 2
Port Priority: 32768 Port: 2049 Group Key: 2
State: 3D - LACP_Active Slow AGG SYNC DIST COLL
. . .
RDEV: 1910.P00 VDEV: 0603 Controller: DTCVSW4 ACTIVE
Adapter ID: 3907000BB4B7.0174
Uplink Port Connection:
MAC address: 02-06-50-00-00-0D
. . .
ACTOR Information:
System ID: 32768,02-03-11-00-00-07 Oper Key: 2
Port Priority: 32768 Port: 2050 Group Key: 2
State: 3D - LACP_Active Slow AGG SYNC DIST COLL
. . .
Member: RDBKZVMF
Scope: Synchronized
LAG Synchronization token: 0000000000000000
Mode: Connected
Member: RDBKZVMC
Scope: Synchronized
LAG Synchronization token: 0000000000000000
Mode: Inactive
Ready;
Again we can see at the bottom of the display that the status of RDBKZVMC is Inactive as a result of being disconnected from the group. We can also see that at the time the disconnection of RDBKZVMC occurred, RDBKZVMF performed a load balance of the LACP protocol.
Also, the MAC addresses in the ACTOR fields did not change, even though RDBKZVMC is no longer attached to the VSwitch. Instead of changing the MAC addresses, which is disruptive to LACP, the new controlling system (in this case RDBKZVMF) takes over the MAC addresses of RDBKZVMC’s VSwitch to allow LACP to continue. This process happens internally in the port group because the MAC addresses do not appear in a QUERY VSWITCH DETAILS listing.
14.3.4 Link Aggregation Control Protocol
When setting up VSwitch Link Aggregation, it is highly recommended that the network switch definition is set up for Link Aggregation Control Protocol (LACP) operation. As defined in IEEE 802.1AX (formerly 802.3ad), LACP is the default operating mode of a z/VM port group. LACP is required when a port group is shared in Multi-VSwitch LAG configuration.
Various techniques are available for link aggregation in the networking world. Terms, such as Etherchannel and port trunk are used to describe different techniques that are used by various switch and operating system vendors to describe their link aggregation methods. These techniques are often proprietary and incompatible with each other.
Example 14-15 shows an example where a port group (in the example, it is an exclusive port group, but the same situation exists for a shared port group) is not functioning because LACP is unavailable. This issue occurs is because the network switch does not support LACP, or the switch ports the z/VM port group is attached to are not configured for LACP.
Example 14-15 Unsuccessful port group activation example
define vswitch vswglbl ether group glblgrp
VSWITCH SYSTEM VSWGLBL is created
Ready; T=0.01/0.01 02:00:17
HCPSWU2830I VSWITCH SYSTEM VSWGLBL status is ready.
HCPSWU2830I DTCVSW1 is VSWITCH controller for device 1910.P00.
HCPSWU2855I Device 1910.P00 for VSWITCH VSWGLBL is disabled for port group GLBLG
RP by LACP.
HCPSWU2830I DTCVSW2 is VSWITCH controller for device 1930.P00.
HCPSWU2855I Device 1930.P00 for VSWITCH VSWGLBL is disabled for port group GLBLG
RP by LACP.
q port group glblgrp
Group: GLBLGRP Active LACP Mode: Active Exclusive
VSWITCH SYSTEM VSWGLBL ifIndex: 2112
Load Balancing: Independent Interval: 300
RDEV: 1910.P00 VDEV: 0606 Controller: DTCVSW1 ACTIVE
Adapter ID: 3907000BB4B7.0174
Status: Error Reason: LACP not enabled on partner
Uplink Port Connection:
MAC address: 02-00-00-00-00-0B
RX Packets: 0 Discarded: 2 Errors: 2
TX Packets: 70 Discarded: 0 Errors: 0
RX Bytes: 0 TX Bytes: 8680
Device: 0606 Unit: 000 Role: DATA Port: 2049
Partner Switch Capabilities: No_Reflective_Relay
RDEV: 1930.P00 VDEV: 0603 Controller: DTCVSW2 ACTIVE
Adapter ID: 3907000BB4B7.010C
Status: Error Reason: LACP not enabled on partner
Uplink Port Connection:
MAC address: 02-00-00-00-00-0C
RX Packets: 0 Discarded: 3 Errors: 1
TX Packets: 66 Discarded: 0 Errors: 0
RX Bytes: 0 TX Bytes: 8184
Device: 0603 Unit: 000 Role: DATA Port: 2050
Partner Switch Capabilities: No_Reflective_Relay
Ready; T=0.01/0.01 02:03:37
The port group GLBLGRP was defined with LACP, and the VSwitch VSWGLBL was then defined by using the group. The error reason “LACP not enabled on partner” is shown against both of the OSA ports in the group.
Only an exclusive-mode port group can operate without LACP, and in that configuration, traffic for the group is distributed between the ports in the group without being managed by a control protocol. Depending on the operating characteristics of the network switch, this configuration might result in traffic not being handled optimally. Some modern network equipment can operate poorly and even experience traffic loss when traffic appears across different ports unexpectedly.
Red Hat Enterprise Linux
If you are on a Red Hat Enterprise Linux system, perform the following steps to create the network device ETH1:
1. From the Linux image, create a device group for the 0604 devices:
# echo 0.0.0604,0.0.0605,0.0.0606 > /sys/bus/ccwgroup/drivers/qeth/group
2. Bring the device online with the following command:
# echo 1 > /sys/devices/qeth/0.0.0604/online
3. Get the name of the device:
# cat /sys/devices/qeth/0.0.0604/if_name
eth1
4. Create a network configuration file by using the name eth1 in the file: /etc/sysconfig/network-scripts/ifcfg-eth1:
===> vi /etc/sysconfig/network-scripts/ifcfg-eth1
#IBM QETH
DEVICE=eth1
BOOTPROTO=static
IPADDR=10.1.1.47
NETMASK=255.255.255.0
NETTYPE=qeth
ONBOOT=yes
SUBCHANNELS=0.0.0604,0.0.0605,0.0.0606
TYPE=ethernet
ARP=no
5. Start the eth1 network device with the ifup command:
===> ifup eth1
6. Verify the status of eth1 with the ifconfig command:
===> ifconfig eth1
eth1 Link encap:Ethernet HWaddr 02:00:0B:00:00:0B
inet addr:10.1.1.47 Bcast:10.1.1.255 Mask:255.255.255.0
inet6 addr: fe80::bff:fe00:b/64 Scope:Link
UP BROADCAST RUNNING NOARP MULTICAST MTU:1492 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2464 (2.4 KiB) TX bytes:350 (350.0 b)
SUSE Linux Enterprise Server 12
If you are on a SUSE Linux Enterprise Server system, perform the following steps to create the network device ETH1:
1. Run the following command to create the device on LNXADMIN:
# qeth_configure -l -t qeth 0.0.0604 0.0.0605 0.0.0606 1
2. Create the interface eth1 by using the file /etc/sysconfig/network/ifcfg-eth1:
# vi /etc/sysconfig/network/ifcfg-eth1
BOOTPROTO='static'
IPADDR='10.1.1.48/24'
STARTMODE='onboot'
NAME='OSA Express(0.0.0604)'
3. Open the eth1 device with the ifup command:
# ifup eth1
4. Test the connectivity between each Linux image.
You now have a functioning network interface that uses port groups.
14.4 Linux network commands
Regardless of which Linux distribution you are using, some commands are available for interacting with network functions.
Users who worked with Linux (or UNIX) for several years might be used to older commands that are bundled with the net-tools package, most of which are deprecated.
The iproute2 suite replaced the net-tools package. If you are still relying on the deprecated commands, begin moving to the new iproute2 version as soon as you can. Some commonly used commands are listed in Table 14-1. Many of the modern commands can be abbreviated; the full command is shown with parenthesis around the optional full-length version.
Table 14-1 Linux networking commands
|
Obsolete commands
|
Modern commands
|
Purpose
|
|
arp
|
ip n(eighbor)
|
Neighboring IP information
|
|
ifconfig
|
ip a(ddress)
ip l(ink)
ip -stats
|
IP addressing values and stats
|
|
ipmaddr
|
ip m(addr)
|
Multicast values and status
|
|
iptunnel
|
ip t(unnel)
|
Set and review IP tunnel values
|
|
netstat
route
|
ip r(oute)
ss
|
IP routing and state information
|
|
nameif
|
ip link
ifrename
|
Rename an interface
|
|
mii-tool
|
ethtool
|
Work with Ethernet values
|
|
iwconfig
|
iw
|
Work with WiFi NICs
|
|
route
|
ip r(oute)
|
IP routing and state information
|
For more information about iproute2, see the following web pages:
For more information about the deprecated net-tools package, see this web page.
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.