In the never-ending quest for increased performance, vendors over the years have developed several different hardware and software architectures to use as a basis for parallel processing. In this chapter, we present the different hardware architectures used for parallel processing in databases. This chapter discusses the basic characteristics of these hardware architectures and contrasts their relative strengths and weakness. In addition, this chapter also discusses the three types of software architectures commonly used by DBMS vendors to implement parallel processing in their database software; this discussion provides you with a framework for understanding the software architecture choices made by Oracle when implementing parallel processing under the different hardware architectures. Finally, this chapter describes the availability of Oracle’s parallel processing features on each type of hardware architecture.
Parallel processing refers to the use of multiple processors to reduce the time needed to complete a given task. Instead of one processor’s executing an entire task, several processors each work on a separate piece of the task. Obviously, parallel processing requires computer hardware that can support more than a single processor. Several architectural approaches to multiple processor systems have been developed over the years. The following four are in commercial use today:
Symmetric Multiprocessing (SMP)
Clustered
Massively Parallel Processing (MPP)
Non Uniform Memory Access (NUMA)
The next four sections describe each of these architectures.
In a Symmetric Multiprocessing (SMP) architecture, multiple processors (CPUs) within a single computer share the same system memory and I/O resources. (See Figure 2.1 for an illustration.) The sharing is accomplished through the use of a high-speed system bus. One copy of the operating system runs on the computer and controls all of the processors. This operating system must be designed to support multiple processors, and it must have a scheduling algorithm that utilizes all the processors in the system evenly. The SMP architecture also is referred to as a tightly coupled or shared memory architecture.
Tip
Please assume that all of the figures presented in this chapter are conceptual. The figures show notable features of the architectures being discussed and illustrate differences between them. Although the concepts remain the same, the detailed hardware implementations will vary significantly among various vendors.
Each processor in an SMP system can execute programs independently, with each processor accessing a separate area of memory. A single-processor system running in a time-sharing mode really can work on only one job at a time. It accomplishes the illusion of doing several things at once by rapidly cycling through all the different jobs that are running. An SMP system, on the other hand, really can work on several jobs simultaneously, because separate processors are dedicated to each job. (This is equivalent to the inter-query parallelism that you read about in Chapter 1.) Because processor job assignment is entirely transparent to applications, database applications can reap the benefits of increased throughput from SMP systems without requiring that any changes be made to the RDBMS software. In addition, the many processors in an SMP system can be made to work cooperatively together on a single large program in order to complete it faster than any single processor could. (This is the equivalent of intra-query parallelism, which you also read about in Chapter 1.) Intra-query parallelism, however, does require specific support from the RDBMS software and from database applications.
SMP architectures have a long history, and most major hardware vendors provide SMP-based systems. SMP architectures bring several advantages to the table:
SMP provides a smooth way to increase the performance of a system; you simply add more processors. You also may add additional memory and disk capacity if required.
Software designed for uniprocessor systems will work on an SMP system without any modification.
SMP systems represent a cost-effective way to achieve scalability in cases in which a uniprocessor system cannot meet performance requirements. SMP is an ideal platform so long as your scalability requirements are within the current limits of SMP hardware and software technologies.
SMP architecture is mature and widely used.
Because only one copy of the operating system runs on an SMP system, the administrative overhead of managing an SMP system is similar to that of managing a uniprocessor system.
Along with all their good points, SMP architectures carry with them two significant disadvantages:
The SMP architecture does not scale well to a large number of processors.
The SMP architecture does not provide high availability.
The number of processors that can be added to an SMP system is limited, because all processors share the same memory. Add too many processors, and you end up with memory and system-bus bottlenecks. The maximum number of processors that can be configured on an SMP system varies from one vendor and operating system to the next. Windows NT supports from two to a maximum of only four processors. Unix-based SMP systems, on the other hand, often support anywhere from 2 to a maximum of 64 processors.
Because only one copy of the operating system runs on an SMP system, the operating system represents a single point of failure. In addition, certain hardware failures also can bring down the system. Consequently, SMP systems by themselves aren’t a good choice when high availability is important.
In clustered architectures, a small set of nodes (between 2 and 64 based on specific vendor implementation) are interconnected and share storage devices. Each node is an independent system with its own processor and memory, running its own copy of the operating system. Figure 2.2 illustrates a two-node cluster. Each node in the cluster can be a single-processor system or an SMP system. In Figure 2.2, each node is an SMP system having two processors. Cluster architecture also is referred to as a loosely coupled architecture , because it allows different processor and memory configurations in each node.
Clustered systems have advantages over both SMP systems and other parallel architectures:
Clusters provide greater performance and scalability than can be possible with any single SMP system. The scalability of a clustered system can be increased in two ways: by increasing the number of processors within each node and/or by increasing the number of nodes in the cluster.
Clustering provides a relatively inexpensive way to achieve scalability by leveraging commonly used processor architectures and standard networking technologies.
Clusters can provide high availability (HA). Often the networking between nodes is done with dual connections to provide redundancy. If one node of the cluster fails, other nodes in the cluster will continue to operate, and users connected to the node that failed can be redistributed to the surviving nodes.
Clustered systems also have some disadvantages:
Additional system software is required to manage and monitor clustered systems. For example, to manage IBM RS/6000 clusters running the AIX operating system, you also need IBM’s HACMP (High Availability Clusters Multiprocessing) software.
Managing a clustered system is a much more complex task than managing a single-node SMP system. The complexity increases as the number of nodes in the cluster increases.
Compared to an SMP system, a clustered system requires more programming effort to coordinate processing on the different nodes, as well as to balance the processing workload across those nodes.
In a Massively Parallel Processing (MPP) architecture, several nodes are connected via a proprietary high-speed network. Each node has its own CPU and memory. Figure 2.3 shows an MPP system with four nodes. Because each node has its own memory, MPP systems are referred to as distributed memory systems. Each node runs its own copy of the operating system, and inter-node communication is done through message passing. The number of nodes that can be configured in an MPP architecture varies from as few as two to as many as several hundred. As with clustered systems, the individual nodes in an MPP system can be uniprocessor nodes, or they can be SMP nodes. MPP systems were developed originally for highly parallelizable scientific computations but are increasingly being used for data warehousing applications.
MPP systems and clusters have many similarities from an architectural point of view. Both use a distributed memory architecture, and in both cases each node runs its own copy of the operating system. Both architectures use message passing for communication between nodes and require additional application programming effort to exploit parallelism. The main difference between MPP and cluster architectures is that the number of nodes that can be configured with MPP is quite large compared to the number of nodes that can be configured in a cluster. In addition, the characteristics of the interconnect differ between MPP and cluster architectures.
The main advantage of MPP systems is that they do not experience the memory bottlenecks that limit the scalability of SMP systems because each node in an MPP system has its own memory. While the individual nodes of an MPP system may themselves be SMP systems, they typically have only a very few processors, keeping them well below the limits of SMP scalability.
MPP systems are suitable for large data warehousing applications, and a large MPP configuration easily can support several thousand users and 10,000 or more transactions per minute.
MPP systems have two major disadvantages:
Because the memory for each node in an MPP system is separate from the others, special features must be built into the operating system to enable communication between the nodes. In MPP systems, when one node needs data held by another, messages are exchanged between the nodes in order to transfer that data.
MPP systems are more difficult to manage than SMP systems, because each node in an MPP system runs its own copy of the operating system and its own copy of the database software. Thus, the MPP architecture is best used when you have reached the current limits of SMP or cluster scalability.
In a Non Uniform Memory Access (NUMA) architecture, multiple SMP systems are interconnected to form a single large system. This is illustrated in Figure 2.4, which shows a NUMA system comprised of two such SMP systems. The SMP systems that you see in the figure really are not complete systems, because each does not run its own copy of the operating system. For this reason, the SMP systems in a NUMA architecture are not referred to as nodes, but as groups . Each SMP group in this figure consists of two CPUs, and each group has its own dedicated memory and I/O controller. The two groups are connected using a high-speed interconnect. Note that this figure looks very similar to Figure 2.2, which depicts a cluster architecture. Unlike a cluster, however, NUMA functions as a large, single SMP system. In a NUMA system, all the memory from all the SMP groups is treated as a large, single, contiguous memory space. In addition, only one copy of the operating system runs across all the nodes. One way to think of a NUMA system is as a cluster within one computer.
In Figure 2.4, the network interface box connects each SMP group to the high-speed interconnect. It also has the added functionality of creating a single, large, continuous view of the memory contained in those SMP groups. To the processors, all memory appears the same. The only difference is in access time. When a processor accesses local memory (memory in its own SMP group), the access is fast. When a process accesses nonlocal memory (memory from another SMP group), that access has to go across the interconnect and as a result is much slower. This is where the NUMA architecture gets its name, because the memory access time is not uniform.
Recall that SMP systems are based on a shared memory architecture. MPP and clustered systems represent distributed memory architectures, because memory is distributed across nodes. NUMA systems represent a hybrid architecture referred to as a distributed shared memory architecture. NUMA attempts to combine the best of both distributed and shared architectures. NUMA systems have the simplicity of SMP systems, because they operate as a single, large system. You don’t have the complexity of message passing that you get with an MPP system. In addition, as your processing needs increase, you can grow a NUMA system by adding more SMP groups, without the scalability limitations inherent in a stand-alone SMP architecture.
Several vendors, such as IBM (NUMA-Q), Data General (AV-2000), SGI (Origin 2000), and NCR have adopted NUMA technology and have implemented variations on NUMA architecture, including NUMA clusters. NUMA is a recently developed architecture, and it remains to be seen if the architecture will live up to its promise and become as commonly used as SMP systems are today.
The NUMA architecture was developed to address the scalability limits of SMP systems and has the following advantages:
It provides the scalability of a loosely coupled architecture such as MPP.
Software applications do not have to change to accommodate the architecture.
NUMA systems do have some disadvantages:
While local-memory access is fast, nonlocal memory access is not. A high percentage of nonlocal memory accesses can slow down the system.
Performance depends not only on the high-speed interconnect, but also on the proper tuning of the operating system. To fully exploit the NUMA architecture, databases and applications need to be NUMA-aware.
As with SMP systems, a single copy of the operating system makes for a single point of failure.