Chapter Objectives
Describe two classes of cables present in a cluster
Examine a system for labeling and tracking cables and connections
The number of cables present in a cluster, and the amount of work they present to a cluster builder, is frequently underestimated. Along with so many cables comes the burden of tracking (and later debugging) the connections made by the cables. This chapter presents a prototype cable labeling and tracking approach that may be adapted to your cluster construction efforts.
Any cluster you build will have cables; it is an inevitable fact. Cables interconnecting the individual components in a cluster represent a large potential source of interesting failure modes and system management headaches. When a compute slice fails, it needs to be uncabled, repaired or replaced, and then recabled. If a cable fails, which does happen, it is necessary to locate and replace it with a minimum amount of impact to the entire cluster, which may continue running while the replacement is taking place.
How well you can manage your cables and troubleshoot the connections when something goes wrong will affect the total time-to-solution and inevitably your cluster's reliability. (This is especially important if you have contractual uptime requirements for your cluster.) This chapter discusses a potential approach to labeling and tracking the many cable connections in your cluster.
How many and what type of cables are involved depends on the size and nature of your cluster. Managing cables during the construction and ongoing operation of a cluster can become a major chore if you do not have a flexible and understandable approach. It can be a total disaster if you have no approach at all. Some typical cable types found in clusters are
Let's take a single compute slice from a hypothetical cluster as an example. The customer requirements for a cluster dictate that each compute slice has one serial port connection for console management, an HSI using Myrinet, one GbE LAN for data access, one 10/100 Mb LAN for monitoring and management data, 300 GB of local storage (two Ultra-320 SCSI enclosures with four, 73-GB disks each, two SCSI buses), and dual power supplies. These requirements yield 12 cables (or fibers) per compute slice (two power and four SCSI for the disk enclosures, two power for the compute slice with redundant power supply, serial console, management LAN, data LAN, and HSI).
Assuming that each compute slice and its disk enclosures take four EIA standard units (each EIA unit is 1.74 inches), this allows space for nine compute slices and associated storage in a standard 41U rack. This leaves 5U unused in the example compute rack. Why I decided to use only nine systems, instead of the maximum possible ten in the compute rack, will become apparent as we proceed. A picture of the example compute and management rack arrangement is shown in Figure 19-1.
Using our example rack, how many of the cables must pass outside the single compute rack to the switches that are located in the management rack? With the current design, there are 108 total cables in the compute rack if you ignore external power connections to the compute rack's PDUs. There is a total of 81 cables connected to devices inside the compute rack and 27 passing outside to equipment in the management rack. Figure 19-1 shows this cabling arrangement.
As mentioned, the cabling example has 108 cables in the compute rack and at least 26 in the management rack. But all cabling is not created equal—some cables are easier to trace, install, and manage than others. Cluster cables may be broken down into two distinct groups according to whether the interconnected devices are located in the same rack. The next sections discuss the two “classes” of cables that we encounter in building a cluster.
Let's call cables that are contained inside a single rack intrarack cables. This class of cable connects to one device in a rack, passes through the rack's cable management system, to a second device in the same rack. Tracking the connections for intrarack cables may not be easy, but at least we do not have to lift floor tiles or sort through the contents of a cable trough to trace this class of cable.
The simplest type of cluster cable, ignoring loop-back cables, is the intrarack cable, but there is another class of cable with which all cluster designers must deal. These cables will pass through the cable management system in one rack, out through an access area in the rack, through an access hole in a floor tile, through a cable trough under the raised floor, up through another floor tile, up into the second rack, through the cable management system in the second rack, and finally attach to the second device. Cables that pass between two or more racks are called interrack cables. (There are specialized SCSI cables, called Y-cables that may connect to three devices. The cable has a single SCSI connector on one end and splits into two separate cable ends and connectors on the other.)
Table 19-1 details the type, class, and number of each cable connection in our example cluster. We would like any cable labeling or identification scheme to enable us to find the correct connection in both racks, without physically following the cable for its entire length. A method that we select should handle both inter- and intrarack cables equally and should minimize the chance of getting the wrong cable.
Table 19-1. Example Compute Rack Cabling
Each Compute Rack Item | Type of Cables | Class of Connection | Cluster Total |
|---|---|---|---|
Redundant power for compute slice | Two power | Intrarack | 18 |
Disk storage unit | Two SCSI and one power each | Intrarack | 54 |
Management LAN | One category 5e cable | Interrack | 9 |
Data LAN | One category 5e cable | Interrack | 9 |
Myrinet HSI | Interrack | 9 | |
Console | One serial | Interrack | 9 |
Per-compute slice connections | Compute slice connections | Intra-/Interrack | 108 |
The first thing we need is a unique way of identifying and locating the end connections for a given cable. To make the information unique within the cluster, the individual racks in the cluster must be individually identified, followed by the piece of equipment, and finally the type and number of the connection. Remember that some equipment will have multiple connections of the same type, like redundant power supply power cords on network switches and compute slices.
Uniquely numbering the racks in your cluster is not a difficult thing. Once the unique rack numbers are chosen (starting from 00 or 01, depending on whether you are a programmer or not), we need a way of uniquely identifying individual devices in the rack. One way to do this is to select the starting EIA Unit for the device in question. Let's start our numbering at the bottom of the rack from 00 and go upward. Most racks have between 41U and 49U as a maximum height, The tallest rack you can use will probably be set by the height of your computer room ceiling and local fire or earthquake codes.
TIP
When numbering items with your cable-labeling system, remember that you may want to be able to use spreadsheets and other text-processing tools to sort the items. Because of the differing column widths for rack 1 and rack 999, you may want to use leading zeros in your numbers to keep the columns of equal width. This can make your documentation neater looking.
So with a rack number between 00 and 99, and locations within the rack between 00 and 41, you can uniquely select a rack and the starting location of a device. For the next portion of the connection identifier, start at the top left and number identical connection types from top to bottom, left to right. You may also need to account for connections either to the front or to the back of a given device (some systems, instead of having keyboard and mouse connections on the back, have USB connections on the front of the system) or special connections like PDUs that run from front to back in the rack.
Let's take all of this information into account, and use the example configuration of devices shown in Figure 19-2. If the management rack is 00 and the compute rack is 01, then an example connection point identifier for the second power connection on the third system in the first compute rack is: rack01-sys02-pwr01. You may wish to designate two- or three-letter codes for the various types of connections on the devices to save extra typing and space in the documentation. Remember that the cost of shortening the type field in this way is a mental translation or a lookup in a table of values, so try to make them mnemonic.
Now, if you think about the other end of the example power cable's connection, you will discover a potential flaw in the system: Some devices, like PDUs are not mounted in the “regular” portion of the rack. (There are other exceptions with which you may have to deal. Most racks have kits that allow you to bolt them together into a single unit, called baying kits. If you are going to do this, you should think about the ramifications to whatever documentation process you have chosen.) This may be dealt with as an exception to the use-the-EIA-unit-as-a-device-label approach. You may have to try different schemes to find one that works for your equipment and preferences. The important characteristic is clarity.
Whatever numbering scheme you choose, whether zero based or one based, be consistent throughout the cluster documentation. Be sure to note any exceptions, like PDUs that do not fit the rest of the scheme prominently. Make sure you keep the exceptions to a minimum. Someone else will have to read and use the documents.
Being able to identify the connection points uniquely for a cable is very useful, but it is only part of what we need to manage the cables in a cluster successfully. We also need to choose a way to label the cables themselves, and to ensure that we have the correct cable, even when it spans multiple racks. So let's build on our connection ID approach in the next sections.
Now that we have determined a scheme for identifying the end points of a cable, we need to choose a method of labeling the cables themselves. Just looking at a bundle of cables, trying to tell which one is which doesn't work very well, even if the cables are neatly done. (A well-constructed and maintained cluster should not have cabling that looks as if it were done by a rat on LSD.) A visible label affixed to the cable is a big help in verifying that you have the correct connection and cable before you unplug the server's SAN connection instead of the 1000base-FX GbE line on which you are working.
A logical approach to putting labels on cables is to place the target connection point identifier on each end of the cable. For example, using the previous information, you might label the power cable rack01-sys02-pwr01 on the end connected to the system, and rack01-pdu02-pwr05 on the end connected to the PDU itself. This way you know where the cable is supposed to be connected on each end. As long as you don't have to move the cable, or change the connection, you will not have to relabel the cable.
This approach may work okay for intrarack cables, but what happens when the cable is an interrack cable? You might have an Ethernet cable labeled rack01-sys02-gbe01 in one rack, and at the other end, in another rack, is the label rack09-swtch16-gbe24. How do you match the first label to the second to make sure that you have the correct cable, without tracing the entire length of cable? With the connection-ID-on-the-ends scheme, there really is no way to identify the specific cable on which you are working, or to ensure that you have the correct one in both racks by examining the connection point IDs. This is a recipe for mistakes and cluster downtime.
A better approach is to “serialize” the individual cables. That is, assign a unique serial number to the cable, based on its type. By locating the connection point ID in one rack, matching it to the cable's serial number and to the connection point ID in the other rack, you are absolutely sure you have the correct cable in both racks. This approach also does not require you to relabel the cable when the connections change—just update the documentation.
Using the connection identifiers to describe the cable end points and a cable serial number for the connection, based on type, makes it possible to track cables between racks quickly. This approach is shown in Figure 19-3. An example table for documentation is shown in Table 19-2.
Table 19-2. Example Cable Documentation
Left Connection Point ID | Cable Serial Number | Right Connection Point ID |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
The examples in Table 19-2 show a number of different connection types and connection point IDs: GbE, power, high-speed Myrinet interconnect, and an entry for a SCSI Y-cable with one “left” end and two “right” ends. (Left and right will depend on how you are standing in relationship to the cluster. Take this into account.) How creative you get with the serial numbers and connection point IDs will depend on how much label space you have, among other things. The important thing is picking a system. sticking to it consistently, and always keeping the documentation up-to-date.
A spreadsheet with the cable information may be made more readable with the addition of colors for the various cable types. The color may substitute for characters in the serial number or connection point IDs. (You may need to be sensitive to system managers or other personnel who are color blind when you consider using color in the documentation and cable labels. Don't rely on color to encode too much information.)
Given the cable-labeling system in this chapter, just how much work should we expect to invest in both inter- and intrarack cabling? A rough estimate of the total number of cables in our example cluster, with one compute rack and one master rack, is 134, not counting any rack power connections.
Of those cables, a total of 36 are interrack, leaving 98 intrarack connections in the compute rack and the master rack. See Interrack Configuration on page 537 for information on specific assembly activities involved in interrack cabling and configuration.
Interrack cabling is more than just another class of cables, it is an extra source of work in the cluster assembly and checkout phases. Because this type of cabling must be run under the computer room floor, every interrack cable adds substantial time to the process of getting the cluster up and running. There are two rack cable management systems to negotiate for each cable, along with the under-the-floor situation.
The time values for the intrarack cabling assume negotiation of the rack's cable management system along with making the actual connection. The information in Table 19-3 details some estimated times based on our example cluster in Figure 19-1. The interrack time values include time for negotiating the cable management systems in two racks as well as installing the cable under the raised flooring. Your actual times will vary depending on the distance between racks, the state of your cable troughs, and other physical constraints.
Table 19-3. Example Cabling Tasks and Time Estimates: First Design
Task Description | Estimated Time, min | Number of Tasks | Total Time, min |
|---|---|---|---|
Serialize cables | 5 [a] | 134 | 670 |
Make intrarack connection | 5 | 98 | 490 |
Verify/document intrarack connection | 2 | 98 | 196 |
Make interrack connection | 15 | 36 | 540 |
Verify/document interrack connection | 2 | 36 | 72 |
Verify entire compute rack | 2 | 134 | 268 |
Total time | 2236 | ||
[a] Assuming that the labels are already made and there are two labels per cable. | |||
In the example compute rack pictured in Figure 19-1, part of the rack was left unfilled, even though we could have racked another compute slice and associated storage devices. Let's revisit that decision. It was made in the hope that we could minimize the interrack cabling.
One way of minimizing interrack cabling is to convert as many interrack cables into intrarack cables as is possible. You can do this by adding extra switch hardware to each rack. The systems in the rack connect to the local switch, and that switch uses network “trunking” or link aggregation to attach to the core switch.
Care must be taken to ensure that the performance requirements of the various networks are still met with this approach. For management LANs, the bandwidth requirements are not critical, it is possible to replace 18 interrack connections with two by the addition of a serial port switch and a 10/100baseT switch in the compute rack. This design is shown in Figure 19-4.
Keeping the data network and HSI at full bandwidth is more difficult. In most cases, the HSI switches are too expensive to replicate, and the additional latency added by a layer of switches may become a performance problem. Likewise, if the data LAN is required to run “full tilt” without bottlenecks, the cluster designer must pay close attention to the backplane (gigabits per second) rating of the switches and their trunking capability. It may be necessary to replicate the switch to keep it from becoming a bottleneck to the bandwidth.
The second design example, in Figure 19-4, shows the changes resulting from the addition of the extra “edge” switches in the compute rack. Table 19-4 shows adjustments to the task time estimates based on the changes in the design. The overall cabling time for the two racks increased by approximately 30 minutes. The hardware changes also increase the amount of switch configuration time necessary during the next phase of the installation.
Table 19-4. Example Cabling Tasks and Time Estimates: Second Design
Task Description | Estimated Time, min | Number of Tasks | Total Time, min |
|---|---|---|---|
Serialize cables | 5 [a] | 150 | 750 |
Make intrarack connection | 5 | 127 | 635 |
Verify/document intrarack connection | 2 | 127 | 254 |
Make interrack connection | 15 | 19 | 285 |
Verify/document interrack connection | 2 | 19 | 38 |
Verify entire compute rack | 2 | 150 | 300 |
Total time | 2262 | ||
[a] Assuming that the labels are already made and there are two labels per cable. | |||
The reduction of interrack cables may not seem worth the effort in our example cluster. Consider, however, a cluster comprising four compute racks and a single management rack. Reducing the number of interrack cables from 144 to 76 not only saves labor installing interrack cables, it saves ports on the core switches for the management and data LANs, and reduces the under-the-floor time installing the cluster's cables.
Because of the increased hardware cost for the extra switches, whether the economy is really there will depend on the extra cost versus your labor rates and expected reduction in complexity, reduction in the interrack cabling tasks, and potential increases in overall cluster reliability. The increased troubleshooting efficiency and decreased time-to-solution also need consideration.
TIP
In addition to labeling the cables, it is useful to have a labeling scheme for the hardware units in the cluster, including things like configuration, MAC addresses, purchase date, purchase order number, and other useful facts. Bar-coded labeling is another innovation that could be worth its weight in gold. Imagine being assigned to take physical inventory of the cluster hardware for your asset management group.
This chapter presented a prototype cable labeling and documentation scheme for an example (very small) cluster. Along with identifying two classes of cables present in a cluster, variations of the connection point identifier and cable serialization recommendations are proved in real- world clusters. This approach may be adapted to your own design and implementation efforts. Any choice of cable documentation system should be carefully evaluated to ensure that it “scales” to whatever size cluster you are designing or building.