Since its introduction in September 2008, Exadata has fast become both a familiar term and a familiar presence in the IT/database realm. The system has undergone several changes in its short history, from storage solution to complete database appliance. Although it is not yet a household name, the number of Exadata installations has increased to the point where it will soon become commonplace in data centers across the country. So, what is Exadata? It might be better to begin by stating what Exadata isn’t. Exadata is not

• the greatest thing since sliced bread;
• the only database machine that can single-handedly eliminate every occurrence of contention in your application;
• the long-awaited silver bullet to solve all of your other database performance problems;
• a black box, constructed by the wizards of Middle Earth, understandable only to the anointed few.

What Is Exadata?

Now that you know what Exadata isn’t, let’s discuss what it is. Exadata is a system, composed of matched and tuned components providing enhancements available with no other configuration, that can improve the performance of the database tier. This system includes database servers, storage servers, an internal InfiniBand network with switches, and storage devices (disks), all configured by Oracle Advanced Customer Support personnel to meet the customer’s requirements. (Changes to that configuration, including changing the default storage allocations between the data and recovery disk groups, can be made with assistance from Oracle Advanced Customer Support.) Figure 1-1 illustrates the general layout of an Exadata system.

Can Exadata improve every situation? No, but it wasn’t designed to. Originally designed as a data warehouse/business intelligence appliance, the releases from V2 on have added Online Transaction Processing (OLTP) applications. Yet not every feature of Exadata is applicable to every query, application, or situation that may arise. Chances are good that if your application is not suffering from contention issues, Exadata may provide reduced response time and better throughput than systems using off-the-shelf components.

Exadata did not start out being what it is today; originally, it was conceived as an open-source storage solution for RAC (Real Application Clusters) installations, intended to address the problem of transferring large volumes of data across the grid infrastructure, known internally within Oracle as SAGE (Storage Appliance for Grid Environments). In September 2008, Oracle unveiled the HP Oracle Database Machine, spotlighting the Exadata Storage Servers. At that time, Exadata referred only to the storage server components, although the HP Oracle Database Machine included all of the components found in later releases of the Exadata product.

The system was marketed as a data warehouse solution. Then, a year later, Exadata V2 was released, this time marketed as a complete and integrated database appliance, including database servers, storage servers, an internal InfiniBand network, and software designed so the components worked in concert as a unified whole. Oracle marketed this appliance as the first database machine for OLTP, so now Exadata was the machine of choice for both data warehousing and OLTP systems. The following year (2010) saw the release of Exadata X2, which continued the improvements by adding a second configuration (X2-8) to the mix. This provided customers with two options for a full-rack implementation. And in September 2012, Exadata X3 hit the market, improving performance yet again and offering a fifth configuration option, the Eighth Rack.

Available Configurations

The X3 series of Exadata machines, which replaced the X2 series, is available in the following five configurations:

• X3-2 Eighth Rack: Two database servers, each with two eight-core Xeon processors with eight cores enabled and 256GB of RAM, three storage servers, and thirty-six disk drives
• X3-2 Quarter Rack: Two database servers, each with two eight-core Xeon processors and 256GB of RAM, three storage servers, and thirty-six disk drives
• X3-2 Half Rack: Four database servers, each with two eight-core Xeon processors and 256GB of RAM, seven storage servers, eighty-four disk drives, and a spine switch for expansion
• X3-2 Full Rack: Eight database servers, each with two eight-core Xeon processors and 256GB of RAM, fourteen storage servers, one hundred sixty-eight disk drives, and a spine switch for expansion
• X3-8 Full Rack: Two database servers, each with eight ten-core Xeon processors and 2TB of RAM, fourteen storage servers, one hundred sixty-eight disk drives, a spine switch for expansion, and no keyboard/video/mouse module

In general, the X3 series of Exadata machines is twice as powerful as the discontinued X2 series on a “like for like” comparison. As mentioned earlier, in the X3-2 series, there is a new Eighth Rack configuration that provides slightly less computing power (a total of sixteen processor cores, eight of which are enabled) than the X2-2 Quarter Rack (which offered a total of twelve processor cores, all of which were enabled). This reduces the licensing costs as compared to the X3-2 Quarter Rack, making the Eighth Rack a very suitable and cost-effective X3-2 entry point into the Exadata arena.

Storage

How much raw storage you have depends on whether you choose High Capacity or High Performance drives—High Capacity Serial Attached SCSI (SAS) drives have 3TB each of raw storage running at 7,200RPM and the High Performance drives have 600GB each running at 15,000RPM. For a Quarter Rack configuration with High Capacity disks, 108TB of total raw storage is provided, with roughly 40TB available for data after normal Automatic Storage Management (ASM) redundancy is configured. Using High Performance disks, the total raw storage for a Quarter Rack machine is 21.1TB, with approximately 8.4TB of usable data storage with normal ASM redundancy. High redundancy reduces the storage by roughly another third on both configurations; the tradeoff is the additional ASM mirror in case of disk failure, as high redundancy provides two copies of the data. Normal redundancy provides one copy.

The disks are accessed through the storage servers (or cells), running their own version of Linux with a subset of the Oracle kernel built in. It is interesting to note that there is no direct access to the storage from the database servers; the only way they can “see” the disks is through ASM. In the X3-2 Quarter Rack and Eighth Rack configurations, there are three storage cells, with each storage cell controlling twelve disks. Each storage server provides two six-core Xeon processors and 24GB of RAM. Between the various configurations of Exadata, the differences become the number of database servers (often referred to as compute nodes) and the number of storage servers or cells—the greater the number of storage cells, the more storage the Exadata machine can control internally. As previously noted, the storage servers also run an integrated Oracle kernel. This allows the database servers to “pass off” (or offload) parts of qualifying queries, so that the database servers only have to handle the reduced data volume of the result sets, rather than scanning every data or index block for the objects of interest. This is known as a Smart Scan. How Smart Scans work and what triggers them are covered in Chapter 2.

Smart Flash Cache

Another part of the Exadata performance package is the Smart Flash Cache, 384GB of solid-state flash storage for each storage cell, configured across four Sun Flash Accelerator F20 PCIe cards. With a Quarter Rack configuration (three storage servers/cells), 1.1TB of flash storage is available; a Full Rack provides 5.3TB of flash storage. The flash cache is usable as a smart cache to service large volumes of random reads, or it can be configured as flash disk devices and mounted as an ASM disk group. That topic will be covered in greater depth in Chapter 4.

Even More Storage

Expansion racks consist of both storage servers and disk drives. There are three different configurations available: Quarter Rack, Half Rack, and Full Rack, any of which can connect to any Oracle Exadata machine. For the Quarter Rack configuration, an additional spine switch will be necessary; the IP address for the spine switch is left unassigned during configuration, so that if one is installed, the address will be available, with no need to reconfigure the machine.

Besides adding storage, these racks also add computing power for Smart Scan operations, with the smallest expansion rack containing four storage servers and forty-eight disk drives, adding eight six-core CPUs to the mix. The largest expansion rack provides 18 storage servers with 216 disk drives. Actual storage will depend on whether the system is using High Capacity or High Performance disk drives; the drive types cannot be mixed in a single Exadata machine/expansion rack configuration, so if the Exadata system is using High Capacity drives, the expansion rack must also contain High Capacity disks, if those disks are being added to existing disk groups.

One reason for this requirement is that ASM stripes the data across the total number of drives in a disk group, thus the size and geometry of the disk units must be uniform across the storage tier. It is not necessary for the disks of the expansion rack to be added to an existing disk group; a completely separate disk group can be created from that storage. You cannot mix storage types within the expansion rack, but the disk type does not need to match that of the host system, if a separate disk group is to be created from that storage. The beauty of these expansion racks is that they integrate seamlessly with the existing storage on the host Exadata system. If these disks are added to existing disk groups, ASM automatically triggers a rebalance to evenly distribute the extents across the total number of available disks.

Things to Know

An Exadata system, available in four configurations, is a complex arrangement of database servers, storage servers, disk drives, and an internal InfiniBand network with modifications designed to address many performance issues in a unique way. It’s the first system with a “divide-and-conquer” approach to query processing that can dramatically improve performance and reduce query response time. It also includes Smart Flash Cache, a write-back cache that can handle large volumes of reads and is designed for Online Transaction Processing (OLTP) systems. This cache can also be configured as flash disk devices. Additional storage is available in the form of Exadata Expansion Racks, which can be added to any Exadata configuration to extend the storage and add storage-cell computing power. The storage in the expansion rack can be the same type (High Capacity, High Performance) as in the Exadata Machine; however, in some cases, Oracle recommends High Capacity drives for expansion racks, regardless of the storage found in the host Exadata system.

Moving on, we’ll discuss the various performance enhancements Exadata provides and how, at least in a limited way, these enhancements are implemented. This is not meant to be an exhaustive text but a “getting started” guide to help lead you through the maze. There are other, more technical, texts you can read to gain a deeper knowledge of the machine, but with this book’s background in hand, it will be easier to understand what Exadata does that other systems can’t.