In the first two chapters, we set up UCSPE and looked at how to connect it physically, both to its own components and to the rest of the network. The physical cabling we looked at in the first chapter was, however, purely for management. This would allow us to control our UCS, but the blades and rack servers would have no connectivity to the rest of the network. In this chapter, we will be focusing on how to add the networking components that will let our UCS talk to the rest of the world.
UCS networking
At the moment, our UCS servers will be disconnected from the rest of the network, so we need to add means for them to pass packets to the rest of the network. We have a couple of ways to achieve this. We can use “Uplink” ports, or port-channels. We will start by looking at uplink ports.
Uplink ports
We start to configure uplink ports by going to the Equipment tab in UCS manager, scrolling down to the fabric interconnect, and then into the ports. Uplink ports can either be configured on the fixed module (fixed ports that are part of the FI) or on the expansion module (a module purchased separately and installed in the FI). We select the port we want to configure and then, from the “Reconfigure” menu, set it as an Uplink port (Figure 3-1).
We will then need to confirm that we do wish to reconfigure the port, and then acknowledge the action once completed.
The next step in creating our uplinks is to head into the network tab in UCS Manager, and select the LAN option from the left-hand side. We will be able to see our uplink interface listed under the relevant fabric (“Fabric A” for FI-A, “Fabric B” for FI-B) (Figure 3-2).
On the other side of the page, we have options we can set for the interface (Figure 3-3). Firstly, we can give it a label, which can be helpful to quickly identify which network device and port we are connected to.
Next, we can set a flow control policy, which controls how the port acts (in the sending and receiving of pause frames) when the receive buffer is full. We create a flow control policy by going to LAN ➤ Policies ➤ root ➤ Flow Control Policies and clicking “Add.” We give it a name, set the priority and Receive and Send to “on” (otherwise the flow-control packets won’t be sent or received) (Figure 3-4).
We then assign this to the interface (Figure 3-5).
The Link Profile controls how the interface works with UDLD (UniDirectional Link Detection). We can create a new Link Profile by going to LAN ➤ Policies ➤ LAN Cloud ➤ Link Profile ➤ default and clicking “Create UDLD Link Policy.” Here we name the policy and set it to enabled and set the mode, either normal or aggressive (Figure 3-6). Normal mode will detect mis-cabling issues, whereas aggressive mode will detect when a link has become unidirectional. Normal mode is not used very much, as Aggressive mode will give us the good stuff that we need, such as “bad” ports being disabled so that failover can happen.
Now that we have a UDLD policy, we can create a link policy by going up one level (LAN ➤ Policies ➤ LAN Cloud ➤ Link Profile) and clicking “Add.” We can name the policy and assign the UDLD-Aggressive link policy to it (Figure 3-7).
The next step is to assign this profile to our interface (Figure 3-8), making sure that we click “Save Changes” at the bottom of the screen.
The following settings control the port speed and the Forwarding Equivalence Class (FEC), which is a form of quality of service.
Clearly, one uplink interface alone will not be enough; we should, at a very minimum, add an uplink on the second FI. Ideally, we would have second interfaces on each FI, going to the other upstream switch to provide a level of redundancy (Figure 3-9).
While two interfaces are good, we are not making the best of our capabilities. With uplink ports, traffic is pinned to one of these links. One isn’t much fun when we could use all four cables at the same time (turning single 40GBps links into a combined 80GBps link).
To do this, we need to create another uplink on FI-A (1/2). We can do this without going to the Equipment tab, we just need to go to LAN ➤ LAN Cloud ➤ Uplink Eth Interfaces and click on “Add,” and select Port 2, by double clicking on it, under the fixed module (Figure 3-10):
We also need to add two uplinks (eth1/1 and eth 1/2) to FI-B, using the same method.
Now we have an even number of links, we can create our port channels.
We create the port channels by going top LAN ➤ LAN Cloud ➤ Fabric A ➤ Port Channels. Click the “Add.” We set the port channel number, and give it a name (Figure 3-11). Click “Next”
In the next window, select the interfaces to add to the port channel (Figure 3-12).
Click the double arrow button to move them into the port channel (Figure 3-13).
Once you have added the interfaces, click “Finish.” You will receive a message to say that the port channel has been created (Figure 3-14).
Repeat the process, adding port channel 2 to FI-B.
Our UCS port channel setup will look like this (Figure 3-15):
From the point of view of the Nexus switches above our FI’s, the configuration would look like this (Figure 3-16):
The Nexus interfaces and port channels would be configured as follows:
NX-9K01# sh run int eth1/1
interface Ethernet1/1
description ### FI-A PORT 1 UPLINK ###
switchport
switchport mode trunk
switchport trunk allowed vlan all
spanning-tree port type edge
spanning-tree bpduguard enable
speed 40000
no negotiate auto
channel-group 1 mode active
no shutdown
NX-9K01# sh run int eth1/2
interface Ethernet1/2
description ### FI-B PORT 1 UPLINK ###
switchport
switchport mode trunk
switchport trunk allowed vlan all
spanning-tree port type edge
spanning-tree bpduguard enable
speed 40000
no negotiate auto
channel-group 2 mode active
no shutdown
NX-9K01# sh run int po 1
interface port-channel1
description ### VPC to FI-A ###
switchport
switchport mode trunk
switchport trunk allowed vlan all
speed 40000
no negotiate auto
no lacp suspend-individual
vpc 1
NX-9K01# sh run int po 2
interface port-channel2
description ### VPC to FI-B ###
switchport
switchport mode trunk
switchport trunk allowed vlan all
speed 40000
no negotiate auto
no lacp suspend-individual
vpc 2
And the other switch configuration:
NX-9K02# sh run int eth 1/1
interface Ethernet1/1
description ### FI-A PORT 2 UPLINK ###
switchport
switchport mode trunk
switchport trunk allowed vlan all
spanning-tree port type edge
spanning-tree bpduguard enable
speed 40000
no negotiate auto
channel-group 1 mode active
no shutdown
NX-9K02# sh run int eth 1/2
interface Ethernet1/2
description ### FI-B PORT 2 UPLINK ###
switchport
switchport mode trunk
switchport trunk allowed vlan all
spanning-tree port type edge
spanning-tree bpduguard enable
speed 40000
no negotiate auto
channel-group 2 mode active
no shutdown
NX-9K02# sh run int po 1
interface port-channel1
description ### VPC to FI-A ###
switchport
switchport mode trunk
switchport trunk allowed vlan all
speed 40000
no negotiate auto
no lacp suspend-individual
vpc 1
NX-9K02# sh run int po 2
interface port-channel2
description ### VPC to FI-B ###
switchport
switchport mode trunk
switchport trunk allowed vlan all
speed 40000
no negotiate auto
no lacp suspend-individual
vpc 2
In this configuration, we can set the interfaces to run together (instead of singularly) and also benefit from a considerable speed increase. While this is not something we can achieve within the sandboxed environment that is UCSPE (as our port channel status will show as “Indeterminate,” we can see this in a real-life example (Figure 3-17):
Summary
In this chapter, we configured uplink ports to connect our UCS to the rest of the network. In the next chapter, we will start configuring the policies we need for our servers.