Let's take a quick look at a schematic of the “guts” of a dual-processor Intel Pentium Xeon server. This is a fictitious design, based on architectural information on the Broadcom ServerWorks GC-LE dual-processor server chip set and the Hewlett-Packard specifications for the Proliant Model DL-360. This is only an approximation as to what is really inside the system on the motherboard, so don't hold me to the details too closely. For the sake of this analysis, let's just assume that we started by looking at this system because we were familiar with the particular vendor.

Figure 4-2 details the major components and buses that tie them together. You should notice the similarity between this figure and the generalized SMP system architecture pictured in Figure 1-2. Looking at the internal design of a system in this way may be instructive. It can help gauge expected performance of the compute slice or explain observations about performance characteristics.

Figure 4-2. Dual Intel Pentium Xeon server architecture

Whether you want raw floating-point performance for scientific computing, or high levels of integer performance and I/O for transaction processing, there are common, desirable design characteristics. I typically look at the design of the system components and interconnects, examining the following.

When examining this system, you can see that there is a lot of capability built into a 1U package. There are no obvious problems with the configuration (such as bottlenecks between the network and RAM). The next step is to begin characterizing the system's performance and compare it with alternatives. This is unfortunately easier to do for technical applications than for commercial processing.

One measure of system floating-point ability is the “theoretical peak” floating-point performance. This is simply calculated by multiplying the CPU instruction clock rate by the number of floating-point instructions that may be completed per cycle. It sets an upper bound on what is possible for the CPUs in the system and allows us to determine the efficiency of the system implementation. For our example, a dual 3.2-GHz Pentium Xeon compute slice, the calculation for theoretical peak floating-point operations per second (FLOPS) is

In reality, however, the system performance on benchmarks such as Linpack,^[2] which is a measure of CPU-bound floating-point performance, is likely to yield somewhere around 8.53 GFLOPs. Why? The location of the potential bottleneck is left as an exercise for you, the reader, but a good guess is probably the memory subsystem.^[3] In other types of benchmarks, such as Web serving or transaction processing, the likely bottlenecks are elsewhere in the system.

A similar calculation for the Hewlett-Packard dual 1.5-GHz Itanium 2 processor system shown in Figure 4-3 is

Figure 4-3. Dual Itanium 2 6M system architecture

The system in question, a Hewlett-Packard rx2600 (similar to the workstation version, the zx6000) examined in the next section, yields 11.42 GFLOPs peak in benchmarks. This is because the memory is multiple banks of interleaved dual in-line memory modules (DIMMs), with a bandwidth of 6.4 GB/s to each bank.

The theoretical peak calculation for the dual-processor 2.0-GHz Opteron system is

This system is interesting because each CPU has its own integrated memory controller and local RAM. A system with RAM distributed in this manner has a penalty to be paid for accessing non-local RAM on another processor. Taking full advantage of this architecture would seem to require an operating system that is ccNUMA capable. The fact is, the details of memory location are hidden by the hardware of the memory controller. The question to consider is: Just what is the system memory bandwidth for a hardware configuration like the system shown in Figure 4-4?

Figure 4-4. Theoretical dual AMD Opteron system architecture

The “it depends” answer applies to this situation. Note that bandwidth-to-local processor memory is 5.3 GB/s across two channels and to nonlocal processor memory is 6.4 GB/s. Looking closer, you can see that the 6.4 GB/s is actually divided between separate read and write channels between the processors. So, for a problem that fits into the local RAM, you can expect local bandwidth and latency, and for problems that span local and remote RAM, you can expect a mixture of local and remote performance characteristics. Indications are that the architecture scales well to at least four processors. (Because this configuration is so new, I could not find any four-processor systems using the Opteron processor. They are sure to appear two days after this book is published.)

As we have seen, looking at the system hardware diagrams can sometimes make a difference when it comes to understanding benchmark numbers. Note that the theoretical peak performances for the Itanium and Xeon systems are roughly equivalent, even though the Itanium system operates at less than half of the clock rate of the Xeon system. Clock rates alone are not a good performance comparison point. The other surprise is the memory configuration of the Opteron system.

One way of comparing systems is to examine the performance characteristics that are important to your application. The major categories of performance are integer and floating point when looking at a particular CPU and system configuration. Some representative numbers for the systems we are examining are shown in Table 4-1. We need to make sure that the numbers we are examining are “apples-to-apples” comparisons, and that they have something to do with the real world.

A very important parameter in the choice of compute slices is total system cost. After all, you are building a cluster from multiple (tens to hundreds) of individual systems. Three individual systems were configured and priced: a 2U Hewlett-Packard dual 1.5-GHz Intel Itanium 2 6M, a 1U Hewlett-Packard dual 3.2-GHz Intel Pentium 4 Xeon, and a 1U IBM dual-2.0 GHz AMD Opteron 246 system. Each system was configured with dual 36-GB disks, 4 GB of RAM, two processors, and a RAID controller (if available). The relative performance in terms of floating-point and integer performance is shown in Figure 4-5.

Figure 4-5. Performance for three dual-processor compute slices

The system list prices^[4] are shown in Table 4-1, along with single-CPU performance numbers from the Standard Performance Evaluation Corporation (SPEC) CPU2000 benchmark.^[5] The SPECint_2000 and SPECfp_2000 benchmarks are subcomponents of the SPEC CPU2000 benchmark.

The SPEC CPU2000 benchmark includes the CFP2000 and CINT2000 benchmarks, both of which contain single-CPU benchmarks (SPECint_2000 and SPECfp_2000) and multi-CPU benchmarks (SPECint_rate_2000 and SPECfp_rate_2000). The CINT2000 benchmark is a composite of 12 individual application benchmarks and the CFP2000 benchmark comprises 14. (Just to make things more interesting, there are also “base” benchmarks, such as SPECfp_base2000 and SPECfp_rate_base2000, that have stringent rules on compilation options and other environmental factors. Numbers are reviewed by the SPEC organization before they are published.) Multi-CPU versions of the benchmarks are presented in Table 4-2. Multi-CPU benchmarks run multiple copies of the individual applications and measure the relative system scaling.

We have the price and we may calculate price performance ratios for the three compute slices to compare the relative costs. This represents how much you pay for a given unit of performance, either integer or floating point. The calculation also works for transaction processing and Web operations, and some industry benchmarks specify their results in dollars per transaction per second.

So how did our three example systems compare? As you can see in Table 4-2, the Itanium 2-based system from Hewlett-Packard wins in terms of highest integer and floating-point performance overall, based on the SPECint_rate_2000 benchmark. The IBM eServer 325 Opteron 246-based system^[6] outperforms the Hewlett-Packard Xeon-based system in terms of floating-point performance, but is almost identical in terms of integer performance. The integer performance of all three systems is pretty close, but there is a substantial difference in floating-point performance when the Opteron and Xeon are compared with the Itanium 2 system.

Figure 4-6 shows the price-to-performance ratios for the systems, based on list price and the SPECint_rate and SPECfp_rate benchmarks. Notice that the high cost of the Itanium system drives the cost per operation up for both floating-point and integer operations. There are some reasons for the extra cost in the Itanium system.

Figure 4-6. List price-to-performance ratio for three example compute slices

The EPIC processor is new and fundamentally different from the other reduced instruction set computing (RISC) systems, the system is 64-bit and 32-bit capable (as is the Opteron), and the maximum RAM capacity is 24 GB. The memory latency and bandwidth, along with the I/O capabilities make the system the fastest compute slice available when coupled with the Quadrics HSI. There are specific applications and situations when the Itanium systems are simply unparalleled (please excuse the pun). The price-to-performance ratio of the Intel Xeon system is lower than the AMD Opteron for integer performance, but is higher in terms of floating-point performance. The choice, based solely on performance or price-to-performance is not clear-cut.

Up to this point, we have taken a look at single systems and their architecture and performance characteristics. But before we jump into a choice of compute slice for our cluster, we need to compare some other factors using all the compute slices. But just how many is that?

Let's say we have a requirement to provide one teraflop of floating-point performance in our cluster. How many compute slices would that be? Now we run into a little “issue” with benchmarks.

We have been comparing systems based on their SPEC performance characteristics, but the SPEC benchmarks don't map cleanly to teraflops. The Linpack benchmark, however, does map to teraflops, provided someone has run the benchmark for the particular processor and system you are evaluating. In our case, two of the systems did not have current Linpack numbers as of this writing.

Another factor to keep in mind is that benchmark and application scaling across the cluster is not perfect. The numbers in Table 4-3 are for single systems. To get one teraflop across the cluster, we would need to measure the actual performance and adjust the number of systems in the cluster. If 80% scaling across the cluster is good (80% scaling is darned good), then we would actually need 102 Itanium 2 systems, 144 Pentium Xeon systems, and 185 Opteron systems to meet our goal. This increases the size of the cluster to seven Itanium compute racks, five Pentium Xeon compute racks, and six Opteron compute racks.

To make the best choice for a given cluster, we would have to examine other factors like the status of 64-bit applications, total memory capacity in the system, the cooling “budget” for the cluster's location, vendor price discounts, and integer versus floating-point performance requirements for real applications.

If you go back to Figure 4-2, you will notice that the two, 32-bit Intel Pentium Xeon processors in the system are capable of addressing a total of eight gigabytes of physical RAM. For this system to be an SMP system, the processors need to share uniformly all eight gigabytes of physical system RAM (in other words, have a uniform addressing scheme). This requires at least 33 hardware address bits on the memory bus for both processors, because the maximum value of a 32-bit binary number is 4,294,967,295 (or 2³² minus 1) or four gigabytes.

Intel 32-bit processors incorporate physical address extension (PAE) features to allow the system hardware to address up to 64 GB of physical RAM. This means that there may be up to 36 bits in the physical RAM address used by the processors, if the extra lines are implemented by the hardware designers on the system's motherboard. This hardware feature allows a system's hardware to accommodate more that the four gigabytes of physical RAM that would be possible with 32 address bits.

So, to provide more than the four gigabyte physical RAM that is addressable by a 32-bit processor, the underlying processors and chip set must support a larger physical address with more than 32 address lines, and the hardware designers must take advantage of this by implementing the extra address lines. The operating system also must be able to manage the larger amount of physical RAM and properly parcel it out to 32-bit processes.

Even the two example compute slice systems that have 64-bit-capable processors (the Itanium 2 and the Opteron) do not fully implement the complete 64 possible hardware memory address lines, mainly for practical reasons of cost. The Opteron systems, for example, allow up to 40 bits of physical memory address (taking up space on the motherboard as circuit connections), which allows addressing up to one terabyte of physical RAM. This amount of RAM, even at today's prices would be very expensive, and would fit into any system enclosure only with great difficulty. (This would be 512 DIMMs at two gigabytes each. That's a lot of memory slots on the motherboard!)

The 64-bit processors can address up to 16,535 (2¹⁴) petabytes (2⁵⁰ bytes) if the full 64 bits were used for physical RAM addresses, but because of physical constraints, hardware designers usually implement only a subset of the possible address bits. Once the hardware can address all the physical RAM, it is up to the operating system's virtual memory system to provide a “virtual address space” to execute a process.

It is the job of the operating system and its virtual memory subsystem to manage physical memory assigned to a given process. This is done by dividing physical memory into fixed-size “pages” that are allocated to processes when needed from a page pool in main memory. The operating system keeps track of which physical page is associated with a particular “virtual” page being used by a process and manages the system's page pool. Once this relationship is determined, the hardware keeps frequently used physical-to-virtual page translations in a set of registers called the translation lookaside buffer or TLB.

How large a program can get, in terms of code and data size, is determined by the maximum virtual address space available (32 or 64 bits) from the hardware and the operating system. The total number of processes that may run on the system simultaneously, without “bumping” another process' physical pages, is determined by the total size of physical RAM. Two processes running on a dual-processor system, each requiring four gigabytes of memory, will not interfere as long as the system has eight gigabytes of physical RAM or more (ignoring operating system memory requirements).

The IA-32 architecture uses four kilobyte (2¹² bytes) physical pages. With four gigabytes of physical RAM, this yields a total of 1,048,576 pages (or 2²⁰ pages). Notice that

An address consisting of 32 bits may be divided into a page number (20 bits) and an offset into that page (12 bits). The IA-64 architecture supports multiple page sizes, but the Linux kernel may be compiled to use one of 4, 8, 16, or 64 kilobyte pages.

Not all pages in a given process' “virtual address space” need to be allocated or “mapped” to physical pages, but the total number of pages that a process can access in its virtual address space, fully mapped or not, differs between a 32-bit and a 64-bit process. A 32-bit process can address at most four gigabytes of virtual memory divided between instruction code and data. A 64-bit process may address multiple petabytes of virtual memory divided between code and data.

Figure 4-7 shows two situations: The first is a system with two four-gigabyte process address spaces sharing four gigabytes of physical RAM, and the second is the same two processes sharing 12 GB of physical RAM. In the first case, there are not enough physical pages available to satisfy the two processes completely, so virtual memory “paging” will occur. In the second case, the 12 GB of physical RAM completely fills the needs of both processes' address space, so some physical pages are left over. In both cases, the operating system pages mapped into the processes address space are shared through the magic of virtual memory.

Figure 4-7. Processes sharing physical memory

If you are going to run multiple processes on your compute slices, then it is important to have enough physical RAM to allow the processes' to coexist without “fighting” for physical pages. If the operating system and the virtual memory system have to share physical pages between processes, virtual memory “paging” occurs. Performance suffers greatly when a system begins paging.

Before your eyes glaze over completely, let's discuss why all this is important in choosing a compute slice for your cluster. If you are processing chunks or pieces of data that will never grow beyond the four-gigabyte limit imposed by 32-bit process virtual addresses,^[7] then you don't need 64-bit hardware, 64-bit operating systems, or 64-bit applications. If, however, you are going to be working with extremely large data sets (beyond four gigabytes) on a per-process basis, then you have no option but to use a 64-bit application, operating system, and hardware or to subdivide the problem into smaller pieces.

The hardware compute slice and effects on the software applications you will run are not independent of each other. For example, both the Opteron and the Itanium processors in our example compute slices are designed to allow both 32- and 64-bit applications to coexist. Just because the hardware is capable of this behavior does not mean that it will be available in your chosen environment. The operating system must also provide facilities to support simultaneous 32- and 64-bit operation.

But why would you want to be able to run both “flavors” of applications on the same hardware? There are a number of answers, including legacy code, that cannot easily be ported to 64-bit operation. Making an application “64-bit clean” is not an easy task, particularly if the original programmers did not adhere to portability standards.

Several things happen in 64-bit code that do not occur “naturally” in 32-bit code.

If you have complete control of your application and the source code, then it becomes your job to port to any new environment. If you use third-party applications, you must check to ensure that the application is available for the type of environment (and compute slice) you are choosing.

In addition to the issues we have discussed so far, there is the additional consideration of the programming model chosen by the operating system. In the integer-long-pointer 32-bit (ILP-32) model, the data types integer, long, and pointer are represented by 32-bit values. There are two competing models in the 64-bit world: integer-long-pointer 64-bit (ILP-64) and long-pointer 64-bit (LP-64). In the first, integer, long, and pointer data types are all 64-bits in length. This can tend to break code that was originally written to the ILP-32 model, because the integer type changes size from 32 to 64 bits.

The LP-64 model represents long and pointer data types as 64 bits, and leaves integers at 32 bits. Although this model does not guarantee trouble-free 64-bit porting efforts, it minimizes the number of changes made to the original 32-bit programming assumptions. Which hardware and operating system you choose for your compute slice will determine whether you have this issue.

As of this writing, the IA-64 Linux maintainers have decided that it is not necessary to use the hardware's ability to run 32-bit Itanium applications at the same time as 64-bit applications. You will need to compile your code for 64 bits for it to perform on this hardware. The Itanium hardware allows execution of 32-bit IA-32 (Pentium III) code, and Linux provides this functionality only to a limited extent. A good deal of performance is given up in the process, and code compiled for later Intel processors may not run under this facility. This attitude may change in the future; we will see. We do not discuss the software issues further here.

We can think of the PCI bus as being implemented by an I/O convertor or bridge with a 64-bit input bus running at 66.6 MHz. At this speed, the maximum throughput to the bridge is 532 MB per second. Underneath the bridge (no, we aren't looking for trolls) there may be individual buses running 32 bits at 66.6 MHz or 64 bits at 33.3 MHz. The throughput on these buses is 266 MB per second.

The peripheral component interconnect express (PCI-X) allows for a 64-bit main bus running at 133.33 MHz, yielding a throughput of 1.06 GB per second. Subbuses are specified at 66.66 megahertz. A PCI-X interface card is backward compatible with PCI, but the data rate will drop to the 33.33 MHz value associated with the PCI 64-bit bus. PCI bus data rates are shown in Table 4-7.

In addition to the PCI interface bus type, there are two sizes of PCI interface cards defined: full-size and short (sometimes called half size). You should make sure which types of slots are available in your compute slice hardware before making assumptions about interface cards. Sometimes one or more of the available PCI slots are half size, and this can really cramp your I/O style.

For examples of PCI bus implementations and speeds, see the example systems in Figure 4-2 through Figure 4-4. All three of these systems have PCI-X interface slots and use PCI for internal core I/O devices.

If your cluster is doing graphics rendering, then you may wish to investigate compute slices with an accelerated graphics port (AGP) slot. An AGP bus provides 32-bit data at various clock rates. AGP was originally intended to provide higher throughput than that of the first implementation of the PCI bus, to support high-performance two- and three-dimensional graphics interface cards.

The AGP 1.0 standard was introduced in 1996, and it provided for both AGP 1x (264 MB per second) and AGP 2x (528 MB per second). The AGP 2.0 standard provides AGP 4x (one gigabyte per second), and the more recent AGP 3.0 specification allows AGP 8x at two gigabytes per second. The “speeds and feeds” for AGP are listed in Table 4-8.