What Is High Availability?

As shown in the previous example of the online store application, business urges IT departments to provide solutions to meet the availability requirements of business applications. As the centerpiece of most business applications, database availability is the key to keeping all the applications available.

In most IT organizations, Service Level Agreements (SLAs) are used to define the application availability agreement between business and IT organization. They can be defined as the percentage availability, or the maximum downtime allowed per month or per year. For example, an SLA that specifies 99.999%  availability means less than 5.26 minutes downtime allowed annually. Sometimes an SLA also specifies the particular time window allowed for downtime; for example, a back-end office application database can be down between midnight and 4 a.m. the first Saturday of each quarter for scheduled maintenance such as hardware and software upgrades.

Since most high availability solutions require additional hardware and/or software, the cost of these solutions can be high. Companies should determine their HA requirements based on the nature of the applications and the cost structure. For example some back-end office applications such as a human resource application may not need to be online 24x7. For those mission–critical business applications that need to be highly available, an evaluation of the cost of downtime may be calculated too; for example, how much money can be lost due to 1 hour of downtime. Then we can compare the downtime costs with the capital costs and operational expenses associated with the design and implementation of various levels of availability solution. This kind of comparison will help business managers and IT departments come up with realistic SLAs that meet their real business and affordability needs and that their IT team can deliver.

Many business applications consist of multi-tier applications that run on multiple computers in a distributed network. The availability of the business applications depends not only on the infrastructure that supports these multi-tier applications, including the server hardware, storage, network, and OS, but also on each tier of the applications, such as web servers, application servers, and database servers. In this chapter, I will focus mainly on the availability of the database server, which is the database administrator’s responsibility.

Database availability also plays a critical role in application availability. We use downtime to refer to the periods when a database is unavailable. The downtime can be either unplanned downtime or planned downtime. Unplanned downtime can occur without being prepared by system admin or DBAs—it may be caused by an unexpected event such as hardware or software failure, human error, or even a natural disaster (losing a data center). Most unplanned downtime can be anticipated; for example, when designing a cluster it is best to make the assumption that everything will fail, considering that most of these clusters are commodity clusters and hence have parts which break. The key when designing the availability of the system is to ensure that it has sufficient redundancy built into it, assuming that every component (including the entire site) may fail. Planned downtime is usually associated with scheduled maintenance activities such as system upgrade or migration.

Unplanned downtime of the Oracle database service can be due to data loss or server failure. The data loss may be caused by storage medium failure, data corruption, deletion of data by human error, or even data center failure. Data loss can be a very serious failure as it may turn out to be permanent, or could take a long time to recover from. The solutions to data loss consist of prevention methods and recovery methods. Prevention methods include disk mirroring by RAID(Redundant Array of Independent Disks) configurations such as RAID 1 (mirroring only) and RAID 10 (mirroring and striping) in the storage array or with ASM (Automatic Storage Management) diskgroup redundancy setting. Chapter 5 will discuss the details of the RAID configurations and ASM configurations for Oracle Databases. Recovery methods focus on getting the data back through database recovery from the previous database backup or flashback recovery or switching to the standby database through Data Guard failover.

Server failure is usually caused by hardware or software failure. Hardware failure can be physical machine component failure, network or storage connection failure; and software failure can be caused by an OS crash, or Oracle database instance or ASM instance failure. Usually during server failure, data in the database remains intact. After the software or hardware issue is fixed, the database service on the failed server can be resumed after completing database instance recovery and startup. Database service downtime due to server failure can be prevented by providing redundant database servers so that the database service can fail over in case of primary server failure. Network and storage connection failure can be prevented by providing redundant network and storage connections.

Planned downtime for an Oracle database may be scheduled for a system upgrade or migration. The database system upgrade can be a hardware upgrade to servers, network, or storage; or a software upgrade to the OS, or Oracle Database patching and upgrade. The downtime for the upgrade will vary depending on the nature of the upgrade. One way to avoid database downtime for system upgrades is to have a redundant system which can take over the database workloads during the system upgrade without causing a database outage. Migration maintenance is sometimes necessary to relocate the database to a new server, a new storage, or a new OS. Although this kind of migration is less frequent, the potential downtime can be much longer and has a much bigger impact on the business application. Some tools and methods are designed to reduce database migration downtime: for example, Oracle transportable tablespace, Data Guard, Oracle GoldenGate, Quest SharePlex, etc.

In this chapter, I focus on a specific area of Oracle Database HA: server availability. I will discuss how to reduce database service downtime due to server failure and system upgrade with Oracle RAC. For all other solutions to reduce or minimize both unplanned and planned downtime of Oracle Database, we can use the Oracle Maximum Availability Architecture (MAA) as the guideline. Refer to the Oracle MAA architecture page, www.oracle.com/technetwork/database/features/availability/maa-090890.html, for the latest developments.

Database Scalability

In the database world, it is said that one should always start with application database design, SQL query tuning, and database instance tuning, instead of just adding new hardware. This is always true, as with a bad application database design and bad SQL queries, adding additional hardware will not solve the performance problem. On the other hand, however, even some well-tuned databases can run out of system capacity as workloads increase.

In this case, the database performance issue is no longer just a tuning issue. It also becomes a scalability issue. Database scalability is about how to increase the database throughput and reduce database response time, under increasing workloads, by adding more computing, networking, and storage resources.

The three critical system resources for database systems are CPU, memory, and storage. Different types of database workloads may use these resources differently: some may be CPU bound or memory bound, while others may be I/O bound. To scale the database, DBAs first need to identify the major performance bottlenecks or resource contentions with a performance monitoring tool such as Oracle Enterprise Manager or AWR (Automatic Workload Repository) report. If the database is found to be I/O bound, storage needs to be scaled up. In Chapter 5, we discuss how to scale up storage by increasing storage I/O capacity such as IOPs (I/O operations per second) and decrease storage response time with ASM striping and I/O load balancing on disk drives.

If the database is found to be CPU bound or memory bound, server capacity needs to be scaled up. Server scalability can be achieved by one of the following two methods:

• Scale-up or vertical scaling: adding additional CPUs and memory to the existing server.
• Scale-out or horizontal scaling: adding additional server(s) to the database system.

The scale-up method is relatively simple. We just need to add more CPUs and memory to the server. Additional CPUs can be recognized by the OS and the database instance. To use the additional memory, some memory settings may need to be modified in OS kernel, as well as the database instance initialization parameters. This option is more useful with x86 servers as these servers are getting more CPUs cores and memory (up to 80 cores and 4TB memory per server of the newer servers at the time of writing). The HP DL580 and DL980 and Dell R820 and R910 are examples of these powerful X86 servers. For some servers, such as those which are based on Intel’s Sandybridge and Northbridge architectures, adding more memory with the older CPUs might not always achieve the same memory performance. One of the biggest issues with this scale-up method is that it can hit its limit when the server has already reached the maximal CPU and memory capacity. In this case, you may have to either replace it with a more powerful server or try the scale-out option.

The scale-out option is to add more server(s) to the database by clustering these servers so that workloads can be distributed between them. In this way, the database can double or triple its CPU and memory resources. Compared to the scale-up method, scale-out is more scalable as you can continue adding more servers for continuously increasing workloads.

One of the factors that will help to determine whether the scale-up or the scale-out option is more appropriate for your environment is the transaction performance requirements. If a lower transaction latency is the goal, the scale-up method may be the option to choose, as reading data from local memory is much faster than reading data from a remote server across the network due to the fact that memory speed is much faster than networking speed, even for the high-speed InfiniBand network. If increasing database transaction throughput is the goal, scale-out is the option to be considered, as it can distribute transaction loads to multiple servers to achieve much higher transaction throughput.

Other factors to be considered include the costs of hardware and software licenses. While the scale-up method may need a high-cost, high-end server to allow vertical scalability, the scale-out method will allow you to use low-cost commodity servers clustered together. Another advantage of the scale-out method is that this solution also confers high availability, which allows database transactions to be failed over to other low-cost servers in the cluster, while the scale-up solution will need another high-cost server to provide a redundant configuration. However, the scale-out method usually needs special licensed software such as Oracle RAC to cluster the applications on multiple nodes. While you may be able to save some hardware costs with the scale-out model, you need to pay for the licencing cost of the cluster software.

The scale-out method takes much more complex technologies to implement. Some of the challenges are how to keep multiple servers working together on a single database while maintaining data consistency among these nodes, and how to synchronize operations on multiple nodes to achieve the best performance. Oracle RAC is designed to tackle these technical challenges and make database servers work together as one single server to achieve maximum scalability of the combined resources of the multiple servers. Oracle RAC’s cache fusion technology manages cache coherency across all nodes and provides a single consistent database system image for applications, no matter which nodes of the RAC database the applications are connected to.

Database Clustering Architecture

To achieve horizontal scalability or scale-out of a database, multiple database servers are grouped together to form a cluster infrastructure. These servers are linked by a private interconnect network and work together as a single virtual server that is capable of handling large application workloads. This cluster can be easily expanded or shrunk by adding or removing servers from the cluster to adapt to the dynamics of the workload. This architecture is not limited by the maximum capacity of a single server, as the vertical scalability (scale-up) method is. There are two types of clustering architecture:

• Shared Nothing Architecture
• Shared Everything Architecture

The shared nothing architecture is built on a group of independent servers with storage attached to each server. Each server carries a portion of the database. The workloads are also divided by this group of servers so that each server carries a predefined workload. Although this architecture can distribute the workloads among multiple servers, the distribution of the workloads and data among the servers is predefined. Adding or removing a single server would require a complete redesign and redeployment of the cluster.

For those applications where each node only needs to access a part of the database, with very careful partitioning of the database and workloads, this shared nothing architecture may work. If the data partition is not completely in sync with the application workload distribution on the server nodes, some nodes may need to access data stored in other nodes. In this case, database performance will suffer. Shared nothing architecture also doesn’t work well with a large set of database applications such as OLTP (Online transaction processing), which need to access the entire database; this architecture will require frequent data redistribution across the nodes and will not work well. Shared nothing also doesn’t provide high availability. Since each partition is dedicated to a piece of the data and workload which is not duplicated by any other server, each server can be a single point of failure. In case of the failure of any server, the data and workload cannot be failed over to other servers in the cluster.

In the shared everything architecture, each server in the cluster is connected to a shared storage where the database files are stored. It can be either active-passive or active-active. In the active-passive cluster architecture, at any given time, only one server is actively accessing the database files and handling workloads; the second one is passive and in standby. In the case of active server failure, the second server picks up the access to the database files and becomes the active server, and user connections to the database also get failed over to the second server. This active-passive cluster provides only availability, not scalability, as at any given time only one server is handling the workloads.

Examples of this type of cluster database include Microsoft SQL Server Cluster, Oracle Fail Safe, and Oracle RAC One Node. Oracle RAC One Node, introduced in Oracle Database 11.2, allows the single-instance database to be able to fail over to the other node in case of node failure. Since Oracle RAC One Node is based on the same Grid Infrastructure as Oracle RAC Database, it can be converted from one node to the active-active Oracle RAC Database with a couple of srvctl commands. Chapter 14 will discuss the details of Oracle RAC One Node.

In the active-active cluster architecture, all the servers in the cluster can actively access the database files and handle workloads simultaneously. All database workloads are evenly distributed to all the servers. In case of one or more server failures, the database connections and workloads on the failed servers get failed over to the rest of the surviving servers. This active-active architecture implements database server virtualization by providing users with a virtual database service. How many actual physical database servers are behind the virtual database service, and how the workloads get distributed to these physical servers, is transparent to users. To make this architecture scalable, adding or removing physical servers from the cluster is also transparent to users. Oracle RAC is the classic example of the active-active shared everything database architecture.

RAC Architecture

Oracle Real Application Cluster (RAC) is an Oracle Database option, based on a share everything architecture. Oracle RAC clusters multiple servers that then operate as a single system. In this cluster, each server actively accesses the shared database and forms an active-active cluster configuration. Oracle first introduced this active-active cluster database solution, called Oracle Parallel Server (OPS), in Oracle 6.2 on VAX/VMS. This name was used until 2001, when Oracle released Oracle Real Application Clusters (RAC) in Oracle Database 9i. Oracle RAC supersedes OPS with many significant enhancements including Oracle Clusterware and cache fusion technology.

In the Oracle RAC configuration, the database files are stored in shared storage, which every server in the cluster shares access to. As shown in Figure 1-1, the database runs across these servers by having one RAC database instance on a server. A database instance consists of a collection of Oracle-related memory plus a set of database background processes that run on the server. Unlike a single-node database, which is limited to one database instance per database, a RAC database has one or more database instances per database and is also built to add additional database instances easily. You can start with a single node or a small number of nodes as an initial configuration and scale out to more nodes with no interruption to the application. All instances of a database share concurrent access to the database files.

Oracle RAC is designed to provide scalability by allowing all the RAC instances to share database workloads. In this way, Oracle RAC Database presents users with a logical database server that groups computing resources such as CPUs and memory from multiple RAC nodes. Most times, with proper configuration using RAC features such as services, Single Client Access Name (SCAN), and database client failover features, changes on the cluster configuration such as adding or removing nodes can be done as transparently to the users as possible. Figure 1-1 illustrates an Oracle RAC configuration where users are connected to the database and can perform database operations through three database instances.

This architecture also provides HA during a failure in the cluster. It can tolerate N-1 node failures, where N is the total number of the nodes. In case of one or more nodes failing, the users connecting to the failed nodes are failed over automatically to the surviving RAC nodes. For example, as shown in Figure 1-1, if node 2 fails, the user connections on instance 2 fail over to instance 1 and instance 3. When user connections fail over to the surviving nodes, RAC ensures load balancing among the nodes of the cluster.

Oracle RAC 12cR1 introduced a new architecture option called Flex Clusters. In this new option, there are two types of cluster nodes: Hub nodes and Leaf nodes. The Hub Nodes are same as the traditional cluster nodes in Oracle RAC 11gR2. All of the Hub Nodes are interconnected with the high-speed interconnect network and have direct access to shared storage. The Leaf Nodes are a new type of node with a lighter-weight stack. They are connected only with the corresponding attached Hub Nodes and they are not connected with each other. These Leaf Nodes are not required to have direct access to shared storage. Instead, they will perform storage I/O through the Hub Nodes that they attach to. The Flex Cluster architecture was introduced to improve RAC scalability. Chapter 4 will discuss the detailed configuration of this new feature in 12c.

Hardware Requirements for RAC

A typical Oracle RAC database requires two or more servers, networking across the servers, and the storage shared by the servers. Although the servers can be SMP Unix servers as well as low-cost commodity x86 servers, it has been an industry trend to move the database server from large SMP Unix machines to low-cost x86-64 servers running on Linux OS, such as Red Hat Enterprise Linux and Oracle Linux.

It is recommended that all the servers in any Oracle RAC cluster should have similar hardware architecture. It is mandatory to have the same OS, with possibly different patches among the servers on the same Oracle RAC. In order to ensure load balancing among the RAC cluster nodes, in 11gR2, server pool management is based on the importance of the server pool and the number of servers associated with the server pool, and there is no way to differentiate between the capacities of the servers. All the servers on the RAC cluster are assumed to have similar (homogeneous) capacity configuration such as CPU counts and total memory, as well as physical networks. If the servers are different in capacity, this will affect resource distribution and session load balancing on the RAC. In Oracle RAC 12c, the policy-based cluster management can manage clusters that consist of heterogeneous servers with different capabilities such as CPU power and memory sizes. With the introduction of server categorization, server pool management has been enhanced to understand the differences between servers in the cluster.

Each server should also have proper local storage for the OS, Oracle Grid Infrastructure software home, and possibly for Oracle RAC software home if you decide to use the local disk to store the RAC Oracle Database binary. Potentially, in the event of a RAC node failure, the workloads on the failed node will be failed over to the working nodes; so each RAC node should reserve some headroom for the computing resources to handle additional database workloads that are failed over from other nodes.

The storage where the RAC database files reside needs to be accessible from all the RAC nodes. Oracle Clusterware also stores two important pieces of Clusterware components—Oracle Cluster Registry (OCR) and voting disk files—in the shared storage. The accessibility of the shared storage by each of the RAC nodes is critical to Clusterware as well as to RAC Database. To ensure the fault tolerance of the storage connections, it is highly recommended to establish redundant network connections between the servers and shared storage. For example, to connect to a Fibre Channel (FC) storage, we need to ensure that each sever on the cluster has dual HBA(Host Bus Adapter) cards with redundant fiber links connecting to two fiber channel switches, each of which connects to two FC storage controllers. On the software side, we need to configure multipathing software to group the multiple I/O paths together so that one I/O path can fail over I/O traffic to another surviving path in case one of the paths should fail. This ensures that at least one I/O path is always available to the disks in the storage.

Figure 1-2 shows a example of configurating two redundant storage network connections to a SAN storage. Depending on the storage network protocols, the storage can be linked with servers using either the FC or iSCSI network. To achieve high I/O bandwidth of the storage connections, some high-speed storage network solutions, such as 16GbE FC and 10gBe iSCSI, have been adapted for the storage network. The detailed configuration of shared storage is discussed in Chapter 5.

The network configuration for an Oracle RAC configuration includes the public network for users or applications to connect to the database, and the private interconnect network for connecting the RAC nodes in the cluster. Figure 1-2 illustrates these two networks in a two-node RAC database. The private interconnect network is accessible only to the nodes in the cluster. This private interconnect network carries the most important heartbeat communication among the RAC nodes in the cluster. The network is also used by the data block transfer between the RAC instances.

A redundant private interconnect configuration is highly recommended for a production cluster database environment: it should comprise at least two network interface cards (NICs) that are connected to two dedicated physical switches for the private interconnect network. These two switches should not be connected to any other network, including the public network. The physical network cards can be bound into a single logical network using OS network bonding utilities such as Linux Bonding or Microsoft NIC Teaming for HA and load balancing.

Oracle 11.2.0.2 introduced a new option for bonding multiple interconnect networks with an Oracle Redundant Interconnect feature called Cluster High Availability IP (HAIP), which provides HA and bonds the interfaces for aggregation at no extra cost to the Oracle environment. Oracle HAIP can take up to four NICs for the private network. Chapter 9 details some best practices for private network configuration. To increase the scalability of the Oracle RAC database, some advanced network solutions have been introduced. For example, as alternatives, 10g GbE network and InfiniBand network are widely used for the private interconnect, to alleviate the potential performance bottleneck.

In Oracle Clusterware 12cR1, Flex Clusters are introduced as a new option. If you use this option, Leaf Nodes are not required to have direct access to the shared storage, while Hub Nodes are required to have direct access to the shared storage, like the cluster nodes in an 11gR2 cluster. Figure 1-2 also illustrates the Flex Cluster structure where servers 1 to 3 are Hub Nodes that have direct access to storage, while server 4 is a Leaf Node that does not connect to shared storage and relies on a Hub Node to perform I/O operations. In release 12.1, all Leaf Nodes are in the same public and private network as the Hub Nodes.

It is recommended to verify that the hardware and software configuration and settings comply with Oracle RAC and Clusterware requirements, with one of these three verification and audit tools depending on the system:

• For a regular RAC system, use RACCheck RAC Configuration Audit Tool (My Oracle Support [MOS] note ID 1268927.1)
• For an Oracle Exadata system, run Exachk Oracle Exadata Database Machine exachk or HealthCheck (MOS note ID 1070954.1)
• For an Oracle Database Appliance, use ODAchk Oracle Database Appliance (ODA) configuration Audit Tool (MOS note ID: 1485630).

RAC Components

In order to establish an Oracle RAC infrastructure, you need to install the following two Oracle licensed products:

• Oracle Grid Infrastructure: This combines Oracle Clusterware and Oracle ASM. Oracle Clusterware clusters multiple interconnected servers (nodes). Oracle ASM provides the volume manager and database file system that is shared by all cluster nodes.
• Oracle RAC: This coordinates and synchronizes multiple database instances to access the same set of database files and process transactions on the same database.

Figure 1-3 shows the architecture and main components of a two-node Oracle RAC database. The RAC nodes are connected by the private interconnect network that carries the Clusterware heartbeat as well as the data transfer among the RAC nodes. All the RAC nodes are connected to shared storage to allow them to access it. Each RAC node runs Grid Infrastructure, which includes Oracle Clusterware and Oracle ASM. Oracle Clusterware performs cluster management, and Oracle ASM handles shared storage management. Oracle RAC runs above the Grid Infrastructure on each RAC node to enable the coordination of communication and storage I/O among the RAC database instances. In the next two sections, we will discuss the functionality and components of these two Oracle products.

[grid@k2r720n1 ∼]$ crsctl stat res –t –
 
--------------------------------------------------------------------------------
NAME               TARGET    STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ACFSHOME.dg
                   ONLINE    ONLINE       k2r720n1
ora.DATA.dg
                   ONLINE    ONLINE       k2r720n1
ora.LISTENER.lsnr
                   ONLINE    ONLINE       k2r720n1
ora.PCIE_DATA.dg
                   ONLINE    OFFLINE      k2r720n1
ora.VOCR.dg
                   ONLINE    ONLINE       k2r720n1
ora.asm
                   ONLINE    ONLINE       k2r720n1                 Started
ora.gsd
                   OFFLINE   OFFLINE      k2r720n1
ora.net1.network
                   ONLINE    ONLINE       k2r720n1
ora.ons
                   ONLINE    ONLINE       k2r720n1
 
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1            ONLINE    ONLINE       k2r720n1
ora.cvu
      1            ONLINE    ONLINE       k2r720n1
ora.k2r720n1.vip
      1            ONLINE    ONLINE       k2r720n1
ora.k2r720n2.vip
      1            ONLINE    INTERMEDIATE k2r720n2
ora.khdb.db
      1            ONLINE    ONLINE       k2r720n1                 Open
      2            ONLINE    ONLINE       k2r720n2                 Open
ora.oc4j
      1            ONLINE    ONLINE       k2r720n1
ora.scan1.vip
      1            ONLINE    ONLINE       k2r720n1

grid@k2r720n1 ∼]$ srvctl status database -d khdb
Instance khdb1 is running on node k2r720n1
Instance khdb2 is running on node k2r720n2

Oracle RAC: Cache Fusion

Oracle RAC is an option that you can select during Oracle database software installation. Oracle RAC and Oracle Grid Infrastructure together make it possible to run a multiple-node RAC database. Like the single-node database, each RAC database instance has memory structure such as buffer cache, shared pool, and so on. It uses the buffer cache in a way that is a little different from a single instance. For a single instance, the server process first tries to read the data block from the buffer cache. If the data block is not in the buffer cache, the server process will do the physical I/O to get the database block from the disks.

For a multi-node RAC database instance, the server process reads the data block from an instance’s buffer cache, which has the latest copy of the block. This buffer cache can be on the local instance or a remote instance. If it is on a remote instance, the data block needs to be transferred from the remote instance through the high-speed interconnect. If the data block is not in any instance’s buffer cache, the server process needs to do the physical read from the disks to the local cache. The instance updates the data block in the buffer cache and then the DBwriter writes the updated dirty blocks to the disk in a batch during the checkpoint or when the instance needs to get free buffer cache slots.

However, in Oracle RAC, multiple database instances are actively performing read and write operations on the same database, and these instances can access the same piece of data at the same time. To provide cache coherency among all the cluster nodes, the writer operation is serialized across all the cluster nodes so that at any moment, for any piece of data, there is only one writer. This is because if each instance acted on its own for the read and update operations on its own buffer cache, and the dirty block wrote to the disk without coordination and proper management among the RAC instances, these instances might access and modify the same data blocks independently and end up by overwriting each others’ updates, which would cause data corruption.

In Oracle RAC, this coordination relies on communication among RAC instances using the high-speed interconnect. This interconnect is based on a redundant private network which is dedicated to communication between the RAC nodes. Oracle Clusterware manages and monitors this private interconnect using the cluster heartbeat between the RAC nodes to detect possible communication problems.

If any RAC node fails to get the heartbeat response from another RAC node within a predefined time threshold (by default 30 seconds), Oracle Clusterware determines that there is a problem on the interconnect between these two RAC nodes, and therefore the coordination between the RAC instances on these two RAC nodes may fail and a possible split-brain condition may occur in the cluster. As a result, Clusterware will trigger a node eviction event to reboot one of the RAC nodes, thus preventing the RAC instance from doing any independent disk I/O without coordinating with another RAC instance on another RAC node. This methodology is called I/O fencing.

Oracle uses an algorithm called STONITH (Shoot The Other Node In The Head), which allows the healthy nodes to kill the sick node by letting the sick node reboot itself. Since 11.2.0.2 with the introduction of reboot-less node eviction, in some cases the node reboot may be avoided by just shutting down and restarting the Clusterware. While Oracle Clusterware guarantees interconnect communication among the RAC nodes, Oracle RAC provides coordination and synchronization and data exchanging between the RAC instances using the interconnect.

In the Oracle RAC environment, all the instances of a RAC database appear to access a common global buffer cache where the query on each instance can get the up-to-date copy of a data block, also called the “master copy,” even though the block has been recently updated by another RAC instance. This is called global cache coherency. In this global cache, since resources such as data blocks are shared by the database process within a RAC instance and across all RAC instances, coordination of access to the resources is needed across all instances. Coordination of access to these resources within a RAC instance is done with latches and locks, which are the same as those in a single-instance database. Oracle cache fusion technology is responsible for coordination and synchronization of access to these shared resources between RAC instances to achieve global cache coherency:

1. Access to shared resources between instances is coordinated and protected by the global locks between the instances.
2. Although the actual buffer cache of each instance still remains separate, each RAC instance can get the master copy of the data block from another instance’s cache by transferring the data block from the other cache through the private interconnect.

Oracle Cache Fusion has gone through several major enhancements in various versions of Oracle Database. Before the Cache Fusion technology was introduced in Oracle 8.1.5, the shared disk was used to synchronize the updates—one instance needs to write the updated data block to the storage immediately after the block is updated in the buffer cache so that the other instance can read the latest version of the data block from the shared disk.

In Oracle 8.1.5, Cache Fusion I was introduced to allow the Consistent Read version of the data block to be transferred across the interconnect. Oracle 9i introduced Cache Fusion II to dramatically reduce latency for the write-write operations. With Cache Fusion II, if instance A needs to update a data block which happens to be owned by instance B, instance A requests the block through the Global Cache Service (GCS), instance B gets notification from the GCS and releases the ownership of the block and sends the block to instance A through the interconnect. This process avoids the disk write operation of instance B and disk read operation of instance A, which were required prior to Oracle 9i. This was called a disk ping and was highly inefficient for this multiple instance’s write operation.

Since the introduction of Cache Fusion II , in Oracle RAC Database, coordination and synchronization between the RAC database instances have been achieved by two RAC services: the Global Cache Service (GCS) and Global Enqueue Service (GES) along with a central repository called the Global Resource Directory (GRD). These two services are the integrated part of Oracle RAC, and they also rely on the clusterware and private interconnects for communications between RAC instances. Both GES and GCS coordinate access to shared resources by RAC instances. GES manages enqueue resources such as the global locks between the RAC instances, and the GCS controls global access to data block resources to implement global cache coherency.

Let’s look at how these three components work together to implement global cache coherency and coordination of access to resources in the RAC across all the RAC instances.

In Oracle RAC, multiple database instances share access to resources such as data blocks in the buffer cache and the enqueue. Access to these shared resources between RAC instances needs to be coordinated to avoid conflict. In order to coordinate and manage shared access to these resources, information such as data block ID, which RAC instance holds the current version of this data block, and the lock mode in which this data block is held by each instance is recorded in a special place called the Global Resource Directory (GRD). This information is used and maintained by GCS and GES for global cache coherency and coordination of access to resources such as data blocks and locks.

The GRD tracks the mastership of the resources, and the contents of the GRD are distributed across all the RAC instances, with the amount being equally divided across the RAC instances using a mod function when all the nodes of the cluster are homogeneous. The RAC instance that holds the GRD entry for a resource is the master instance of the resource. Initially, each resource is assigned to its master instance using a hashing algorithm. The master instance can be changed when the cluster is reconfigured when adding or removing an instance from the cluster. This process is referred as the “reconfiguration.”

In addition to reconfiguration, the resource can be remastered through Dynamic Resource Mastering (DRM). DRM can be triggered by resource affinity or an instance crash. Resource affinity links the instance and resources based on the usage pattern of the resource on the instance. If a resource is accessed more frequently from another instance, the resource can be remastered on another instance. The master instance is a critical component of global cache coherency. In the event of failure of one or more instances, the remaining instances will reconstruct the GRD. This ensures that the global GRD is kept as long as one instance of the RAC database is still available.

The GCS is one of the services of RAC that implement Oracle RAC cache fusion. In the Oracle RAC environment, a data block in an instance may be requested and shared by another instance. The GCS is responsible for the management of this data block sharing between RAC instances. It coordinates access to the database block by RAC instances, using the status information of the data blocks recorded in the entry of the GRD. The GCS is responsible for data block transfers between RAC instances.

The GES manages the global enqueue resources much as the GCS manages the data block resource. The enqueue resources managed by GES include library cache locks, dictionary cache locks, transaction locks, table locks, etc.

Figure 1-4 shows a case in which an instance requests a data block transfer from another instance.

Instance 1 needs access to a data block. It first identifies the resource master instance of the block, which is instance 2, and sends a request to instance 2 through GCS.

From the entry of the GRD for the block in resource master instance 2, the GCS gets the lock status of the data block and identifies that the holding instance is instance 3, which holds the latest copy of the data block; then GCS requests instance 3, the shared resource of the data block, and the block transfer to instance 1.

Instance 3 transfers the copy of the block to instance 1 via the private interconnects.

After receiving the copy of the block, instance 1 sends a message to the GCS about receiving the block, and the GCS records the block transfer information in GRD.

RAC Background Processes

Each RAC database instance is a superset of a single-node instance. It has the same set of background processes and the same memory structure, such as the System Global Area (SGA) and the Program Global Area (PGA). As well as these, RAC instances also have the additional processes and memory structure that are dedicated to the GCS processes, GES, and the global cache directory. These processes are as follows:

• LMS: Lock Manager Server process
• LMON: Lock Monitor processes
• LMD: Lock Monitor daemon process
• LCK: Lock process
• DIAG: Diagnostic daemon

1. LMS process: The Lock Manager Server is the Global Cache Service (GCS) process. This process is responsible for transferring the data blocks between the RAC instances for cache fusion requests. For a Consistent Read request, the LMS process will roll back the block and create the Consistent Read image of the block and then transfer the block to the requesting instance through the high-speed interconnect. It is recommended that the number of the LMS processes is less than or equal to the number of physical CPUs. Here the physical CPUs are the “CPU cores”; for example, for a server with two sockets that has four cores, the number of the physical CPU is 8. By default, the number of LMS processes is based on the number of the CPUs on the server. This number may be too high as one LMS process may be sufficient for up to four CPUs. There are a few ways to control the number of the LMS processes. You can modify the values for the init.ora parameter CPU_COUNT, which will also indirectly control the number of LMS processes that will be started during the Oracle RAC Database instance startup. The number of LMS processes is directly controlled by the init.ora parameter GCS_SERVER_PROCESSES. For a single CPU server, only one LMS is started. If you are consolidating multiple small databases on a cluster environment, you may want to reduce the number of LMS processes per instance, as there may be multiple instances of RAC databases on a single RAC node. Refer to the Oracle support note [ID 1439551.1] “Oracle (RAC) Database Consolidation Guidelines for Environments Using Mixed Database Versions” for detailed guidelines for setting LMS processes for multiple databases of RAC.
2. LMON process: The Lock Monitor process is responsible for managing the Global Enqueue Service (GES). It is also responsible for reconfiguration of lock resources when an instance joins or leaves the cluster and responsible for dynamic lock remastering.
3. LMD process: The Lock Monitor daemon process is the Global Enqueue Service (GES). The LMD process manages the incoming remote lock requests from other instances in the cluster.
4. LCK process: The Lock process manages non–cache fusion resource requests, such as row cache and library cache requests. Only one LCK process (lck0) per instance.
5. DIAG process: The Diagnostic daemon process is responsible for all the diagnostic work in a RAC instance.

In addition, Oracle 11gR2 introduced a few new processes for RAC. These processes are as follows:

• ACMS: Atomic Controlfile to Memory Server
• GTXn: Global Transaction process
• LMHB: LM heartbeat monitor (monitors LMON, LMD, LMSn processes)
• PING: Interconnect latency measurement process
• RMS0: RAC management server
• RSMN: Remote Slave Monitor

The following command shows these five background processes on a RAC node. This example shows that both the khdb1 instance and the ASM1 instance have a set of background processes. The Grid user owns the background processes for the ASM instance and the Oracle user owns the background processes for the RAC database instance ‘khdb1’. If you have run multiple RAC databases on the RAC node, you will see multiple sets of the background processes. The process-naming convention is ‘ora_<process>_<instance_name>’, for example ‘ora_lms2_khdb1’ and ‘asm_lms0_+ASM1’.

$ ps -ef | grep –v grep | grep 'lmon|lms|lck|lmd|diag|acms|gtx|lmhb|ping|rms|rsm'
grid      6448     1  0 Nov08 ?        00:13:48 asm_diag_+ASM1
grid      6450     1  0 Nov08 ?        00:01:43 asm_ping_+ASM1
grid      6455     1  0 Nov08 ?        00:34:52 asm_lmon_+ASM1
grid      6457     1  0 Nov08 ?        00:26:20 asm_lmd0_+ASM1
grid      6459     1  0 Nov08 ?        00:58:00 asm_lms0_+ASM1
grid      6463     1  0 Nov08 ?        00:01:17 asm_lmhb_+ASM1
grid      6483     1  0 Nov08 ?        00:02:03 asm_lck0_+ASM1
oracle   21797     1  0 Nov19 ?        00:07:53 ora_diag_khdb1
oracle   21801     1  0 Nov19 ?        00:00:57 ora_ping_khdb1
oracle   21803     1  0 Nov19 ?        00:00:42 ora_acms_khdb1
oracle   21807     1  0 Nov19 ?        00:48:39 ora_lmon_khdb1
oracle   21809     1  0 Nov19 ?        00:16:50 ora_lmd0_khdb1
oracle   21811     1  0 Nov19 ?        01:22:25 ora_lms0_khdb1
oracle   21815     1  0 Nov19 ?        01:22:11 ora_lms1_khdb1
oracle   21819     1  0 Nov19 ?        01:22:49 ora_lms2_khdb1
oracle   21823     1  0 Nov19 ?        00:00:41 ora_rms0_khdb1
oracle   21825     1  0 Nov19 ?        00:00:55 ora_lmhb_khdb1
oracle   21865     1  0 Nov19 ?        00:04:36 ora_lck0_khdb1
oracle   21867     1  0 Nov19 ?        00:00:48 ora_rsmn_khdb1
oracle   21903     1  0 Nov19 ?        00:00:45 ora_gtx0_khdb1

Besides these processes dedicated to Oracle RAC, a RAC instance also has other background processes which it has in common with a single node database instance. On Linux or Unix, you can see all the background processes of a RAC instance by using a simple OS command, for example: $ ps -ef | grep 'khdb1'

Or run the following query in SQL*Plus:

sqlplus> select NAME, DESCRIPTION  from  v$bgprocess where PADDR != '00'

High AvailabilityAgainst Unplanned Downtime

The Oracle RAC solution prevents unplanned downtime of the database service due to server hardware failure or software failure. In the Oracle RAC environment, Oracle Clusterware and Oracle RAC work together to allow the Oracle Database to run across multiple clustered servers. In the event of a database instance failure, no matter whether the failure is caused by server hardware failure or an OS or Oracle Database software crash, this clusterware provides the high availabilityand redundancy to protect the database service by failing over the user connections on the failed instance to other database instances.

Both Oracle Clusterware and Oracle RAC contribute to this high availability database configuration. Oracle Clusterware includes the High Availability (HA) service stack which provides the infrastructure to manage the Oracle Database as a resource in the cluster environment. With this service, Oracle Clusterware is responsible for restarting the database resource every time a database instance fails or after a RAC node restarts. In the Oracle RAC Database environment, the Oracle Database along with other resources such as the virtual IP (VIP) are managed and protected by Oracle Clusterware. In case of a node failure, Oracle Clusterware fails over these resources such as VIP to the surviving nodes so that applications can detect the node failure quickly without waiting for a TCP/IP timeout. Then, the application sessions can be failed over to the surviving nodes with connection pool and Transparent Application Failover (TAF).

If a database instance fails while a session on the instance is in the middle of a DML operation such as inserting, updating, or deleting, the DML transaction will be rolled back and the session will be reconnected to a surviving node. The DML of the transaction would then need to be started over. Another great feature of the clusterware is the Oracle Notification Services (ONS). ONS is responsible for publishing the Up and Down events on which the Oracle Fast Application Notification (FAN) and Fast Connect Failover (FCF) rely to provide users with fast connection failover to the surviving instance during a database instance failure.

Oracle RAC database software is cluster-aware. It allows Oracle RAC instances to detect an instance failure. Once an instance failure is detected, the RAC instances communicate with each other and reconfigure the cluster accordingly. The instance failure event triggers the reconfiguration of instance resources. During the instances’ startup, these instance resources were distributed across all the instances using a hashing algorithm. When an instance is lost, the reconfiguration reassigns the new master instance for those resources that used the failed instance as the master instance. This reconfiguration ensures that the RAC cache fusion survives the instance failure. The reconfiguration is also needed when an instance rejoins the cluster once the failed server is back online, as this allows further redistribution of the mastership with the newly joined instance. But this reconfiguration process that occurs when adding a new instance takes less work than the one that occurs with a leaving instance, as when an instance is leaving the cluster, those suspected resources need to be replayed and the masterships need to be re-established.

DRM is different from reconfiguration. DRM is a feature of Global Cluster Service that changes the master instance of a resource based on resource affinity. When the instance is running on an affinity-based configuration, DRM remasters the resource to another instance if the resource is accessed more often from another node. Therefore, DRM occurs when the instance has a higher affinity to some resources than to others, whereas reconfiguration occurs when an instance leaves or joins the cluster.

In the Oracle 12c Flex Cluster configuration, a Leaf node connects to the cluster through a Hub node. The failure of the Hub Node or the failure of network between the Hub node and the Leaf nodes results in the node eviction of the associated Leaf nodes. In Oracle RAC 12cR1, since there is no user database session connecting to any Leaf Nodes, the failure of a Leaf Node will not directly cause user connection failure. The failure of the Hub Node is handled in essentially the same way as the failover mechanism of a cluster node in 11gR2.

RAC resource mastering is performed only on the Hub node instances, not on a Leaf node. This ensures that the failure of a Leaf node does not require remastering and also ensures that masters have affinity to the Hub node instances.

Oracle RAC and Oracle Clusterware also work together to allow application connections to perform seamless failover from the failed instance to the surviving instance. The applications can use these technologies to implement smooth failover for their database operations such as query or transactions. These technologies include:

1. Transparent Application Failover (TAF)
2. Fast Connect Failover (FCF)
3. Better Business continuity & HA using Oracle 12c Application Continuity (AC)

KHDB_Sales =
 (DESCRIPTION =
  (ADDRESS = (PROTOCOL = TCP)(HOST = kr720n-scan)(PORT = 1521))
  (CONNECT_DATA =
   (SERVER = DEDICATED)
   (SERVICE_NAME =khdb_sales.dblab.com)
    (FAILOVER_MODE =
    (TYPE=session)
    (METHOD=basic)
    (RETRIES=10)
    (DELAY=10)
  ) ))

KHDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = kr720n1-vip)(PORT = 1521))
    (ADDRESS = (PROTOCOL = TCP)(HOST = kr720n2-vip)(PORT = 1521))
    (CONNECT_DATA =
      (SERVICE_NAME =khdb.dblab.com)
       )
    )
   )

High Availability Against Planned Downtime

Oracle RAC helps achieve database HA by reducing database service downtime due to the scheduled maintenance of the database infrastructure. Scheduled maintenance work may include server hardware upgrades or maintenance, server OS upgrade, and Oracle software upgrades. Depending on the task, these maintenance jobs may require bringing down the database instance or OS, or the server hardware itself. With Oracle RAC, maintenance work can be done in rolling fashion without the need to bring down the entire database service. Let’s see how to perform rolling-fashion maintenance for different types of maintenance tasks.

Hardware maintenance of the database server may be needed during the lifetime of the server, for example upgrading or replacing hardware components such as CPUs, memory, and network cards. Although server downtime is required for this kind of maintenance, with Oracle RAC the database connections on the database instance of the impacted server can be relocated to other instances on the cluster. The maintenance can be done in rolling fashion by the following steps:

• 1.  Relocate the database connections to the other instance;
• 2.  Shut down the entire software stack in this order:

• a)  Database instance
• b)  ASM and Clusterware
• c)  Operating system
• d)  Server hardware

• 3.  Perform the server hardware upgrade;
• 4.  Restart the software stack in the reverse order
• 5.  Repeat steps 1 to 4 for all other servers in the cluster.

Since the database connections are relocated to the other instance during hardware maintenance, the database service outage is eliminated. This rolling-fashion system maintenance enables system upgrades without database service downtime.

A similar rolling-upgrade method applies to OS upgrades, as well as other utility upgrades such as firmware, BIOS, network driver, and storage utility upgrades. Follow steps 1 to 5 except for the server hardware shutdown at step 2, and perform the OS or utility upgrade instead of the hardware upgrade at step 3.

You can also perform a rolling upgrade of Oracle Clusterware and ASM to avoid Clusterware and ASM downtime. In Oracle RAC 12.1, you can use Oracle Universal Installer (OUI) and Oracle Clusterware to perform a rolling upgrade to apply a patchset release of Oracle Clusterware. This allows you to shut down and patch RAC instances one or more at a time while keeping other RAC instances available online. You can also upgrade and patch clustered Oracle ASM instances in rolling fashion. This feature allows the clustered ASM environment to continue to function while one or more ASM instances run different software releases and you are doing the rolling upgrade of the Oracle ASM environment. Many of these rolling-upgrade features were available in releases prior to RAC 12c, but they are easier to do with the GUI in Oracle 12cR1.

In order to apply the rolling upgrade for Oracle RAC software, the Oracle RAC home must be on a local file system on each RAC node in the cluster, not in a shared file system. There are several types of patch for an Oracle RAC database: interim patch, bundle patch, patch set upgrades (PSU), critical patch update (CPU), and diagnostic patch. Before you apply a RAC database patch, check the readme to determine whether or not that patch is certified for the rolling upgrade. You can also use the Opatch utility to check if the patch is a rolling patch:

$ opatch query –all <Patch_location> | grep rolling

If the patch is not a rolling patch, it will show the result “Patch is a rolling patch: false”; otherwise, it will show “Patch is a rolling patch: true.”

You can use the OPatch utility to apply individual patches, not the patchset release to the RAC software. If the upgrade can be performed using the rolling fashion, follow these steps to perform the rolling upgrade:

1. Shut down the instance on one RAC node
2. Shut down the CRS stack on this RAC node
3. Apply the patch to the RAC home on that RAC node
4. Start the CRS stack on the RAC node
5. Start the RAC instance on the RAC node
6. Repeat steps 1 to 4 on each of the other RAC nodes in the cluster

There is a special type of interim patch or diagnostic patch. These patches contain a single shared library, and do not require shutting down the instance or relinking the Oracle binary. These patches are called online patches or hot patches. To determine whether a patch is an online patch, check if there is an online directory under the patch and if the README file has specified this patch to be online patchable. You can use the Opatch tool to apply an online patch without shutting down the Oracle instance that you are patching. For example, Patch 10188727 is an online patch, as shown in the patch directory:

$ cd <PATCH_TOP>/10188727
$ ls
etc/ files/online/README.txt

You also can query if the patch is an online patch by going to the patch direcory and running the following command:

$ opatch query -all online

If the patch is an online patch, you should see something like this in the result for this command: “Patch is an online patch: true.” You should not confuse this result with the query result of a rolling patch result, "Patch is a rolling patch: true."

You should be aware that very few patches are online patches. Usually, online patches are used when a patch needs to be applied urgently before the database can be shut down. It is highly recommended that at the next database downtime the all-online patches should be rolled back and replaced with offline version of the patches. Refer to MOS note ID 761111.1 for all the best practices when using online patches.

For those patches that are not certified for the rolling upgrade, if you have a physical standby configuration for the database, you can use the Oracle Data Guard SQL apply feature and Oracle 11g Transient Logical standby feature to implement the rolling database upgrade between the primary database and standby database and thus reduce database upgrade downtime. In this case, the database can be either a RAC or a non-RAC single-node database. Refer to MOS note ID 949322.1 for a detailed configuration of this method.

Oracle RAC One Node to Achieve HA

Oracle RAC One Node is a single-node RAC database. It provides an alternative way to protect the database against both unplanned and planned downtime. Unlike Oracle RAC Database with multiple database instances to provide an active-active cluster database solution, Oracle RAC One Node database is an active-passive cluster database. At any given time, only one database instance is running on one node of the cluster for the database. The database instance will be failed over to another node in the cluster in case of failure of the RAC node. This database instance can also be relocated to another node. This relocation is called online migration, as there is no database service downtime during the relocation. This online migration eliminates the planned downtime of maintenance.

Oracle RAC One Node is based on the same Grid Infrastructure as Oracle RAC Database. Oracle Clusterware in the Grid Infrastructure provides failover protection for Oracle RAC One Node. Since Oracle RAC One Node runs only one database instance, you can scale up the database server only by adding more CPU and memory resources to the server instead of scaling out by running multiple database instances. If the workloads expand beyond the capacity of a single server, you can easily upgrade the RAC One Node database to a fully functional multi-node Oracle RAC. Compared with Oracle RAC, Oracle RAC One Node has a significant advantage in its software license cost. Chapter 14 discusses Oracle RAC One Node technology in detail.

RAC Scalability

Oracle RAC offers an architecture that can potentially increase database capacity by scaling out the database across multiple server nodes. In this architecture, the multi-node cluster combines the CPU and memory computing resources of multiple RAC nodes to handle the workloads. Oracle cache fusion makes this scale-out solution possible by coordinating shared access to the database from multiple database instances. This coordination is managed through communication over the high-speed interconnect. One of the key components of the coordination is GCS data block transfer between database instances. Heavy access to the same data block by multiple database instances leads to high traffic of the data transfer over the interconnect. This can potentially cause interconnect congestion, which easily becomes a database performance bottleneck. The following considerations may help maintain RAC database scalability:

1. Segregate workloads to different RAC nodes to reduce the demand for sharing the same data block or data object by multiple RAC nodes. This segregation can also be done through application affinity or instance affinity. For example, for the Oracle E-Business application, we can assign each application module to a specific RAC instance so that all applications that access the same set of tables are from the same instance.
2. Reduce potential block transfers between instances. One way is to use a big cache value and NOORDER option for the sequence creation in a RAC database. This will ensure that each instance caches a separate range of sequence numbers and the sequence numbers are assigned out of order by the different instances. When these sequence numbers are used as the index key values, different key values of the index are inserted depending on the RAC instance in which the sequence number is generated. This creates instance affinity to the index Leaf blocks, and helps reduce pinging of the index Leaf blocks between instances.
3. Reduce the interconnect network latency by using a high bandwidth, high-speed network such as InfiniBand or 10GB Ethernet.
4. Constantly monitor the interconnect traffic and RAC cluster wait events. You can either monitor these cluster wait events on the Clusterware cache coherence page of Oracle Enterprise Manager, or check if there are any of the GCS-related wait events shown on the Top 5 Timed Events of AWR report.

KHDB =
 (DESCRIPTION =
  (ADDRESS = (PROTOCOL = TCP)(HOST = kr720n1-vip)(PORT = 1521))
  (ADDRESS = (PROTOCOL = TCP)(HOST = kr720n2-vip)(PORT = 1521))
 (LOAD_BALANCE =  yes)
  (CONNECT_DATA =
   (SERVICE_NAME =khdb.dblab.com)
   )
  )
 )

SQL> show parameter _listener
 
NAME                   TYPE    VALUE
---------------------- ------- ------------------------------
local_listener         string  (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(
                               (PROTOCOL=TCP)(HOST=172.16.9.171)
                               (PORT=1521))))
remote_listener        string  kr720n-scan:1521

Consolidating Database Services with Oracle RAC

In the traditional corporate computing model, one infrastructure is usually built for one application, with little or no resource sharing among applications. Not only does this model result in low efficiency and poor utilization of resources, it also makes it very difficult to reassign resources to adapt to the rapid pace of business change. Many systems have to preallocate large amounts of system resources in their capacity planning to cover peak demand and future growth. Today’s IT departments, under increasing pressure to provide low-cost, flexible computing services, have to adapt to the idea that multiple applications services can be consolidated to share a pool of the computing resources: servers, networks, and storage. Grid computing originated from the concept of the electricity utility grid, and the recent Private Cloud is also based on this idea. These models have the following characteristics:

1. Consolidation: Many applications with different workloads are consolidated in the same infrastructure.
2. Dynamic resource sharing: All resources in the infrastructure are shared and can be dynamically reassigned or redistributed to applications services as needed.
3. High Availability: Applications within the shared infrastructure can be failed over or migrated across physical resources. This provides a virtualized infrastructure service that is independent of the physical hardware and also protects applications against unplanned system outages and planned system maintenance.
4. Scalability: Allows adding more physical resources to scale the infrastructure.

Consolidation and resource sharing dramatically increase resource utilization, and reduce hardware and system costs. There are cost savings both in capital expenses and operating expenses, as you not only need to purchase less hardware and software, but you can spend less on management and ongoing support costs. Consolidation and resource sharing also confer the benefits of high availability, flexibility, and scalability on application services.

Oracle RAC, along with Oracle Clusterware and ASM, provides the key technologies to implement this shared resource infrastructure for database consolidation.

• Oracle Clusterware and ASM provide infrastructure which consists of a pool of servers, along with storage and a network for database consolidation.
• Oracle Clusterware and RAC provide high availability and scalability for all databases that share this infrastructure.
• Oracle RAC features such as Instance Caging, Database Resource Manager, and Quality of Service enable the efficient use of shared resources by databases.
• The enhancement introduced by Oracle 12c Clusterware, like the policy-based approach for Clusterware management, allows for dynamic resource reallocation and prioritization of various applications’ workloads consolidated in the shared cluster infrastructure.

Figure 1-5 shows an example of such a shared resources Grid Infrastructure based on a 16-node Oracle 11gR2 RAC that consolidates more than 100 Oracle E-Business suite databases running on various versions of Oracle Database from 10gR2 to 11gR2.

• Each database may run on one to three RAC nodes with the capability to expand to more nodes.
• Each database instance can be failed over or relocate to other nodes.
• Each RAC node may run multiple RAC database instances from different databases using different versions of Oracle RAC Database binaries.
• Multiple versions (10g2-11gR2) of Oracle RAC Database are based on a single Oracle 11gR2 Grid Infrastructure.
• It is required to ping CRS on each of 16 nodes for pre-11gR2 databases

MOS note ID 1439551.1 “Oracle (RAC) Database Consolidation Guidelines for Environments Using Mixed Database Versions” discusses some principles and guidelines for how to consolidate multiple versions of Oracle Database on the same Oracle RAC and Oracle Grid Infrastructure 11g release 2 or higher. The discussions include considerations for including pre-11g Release 2 databases and how to set up the number of the LMS processes and how to set the CPU count to manage CPU resources for those pre-11g R2 Database instances. The related whitepaper on the Database consolidation topic is Oracle whitepaper “Best Practices for Database Consolidation in Private Clouds,” which can be found at the following URL: www.oracle.com/technetwork/database/focus-areas/database-cloud/database-cons-best-practices-1561461.pdf.

As one of the most important new features introduced in Oracle 12c, the pluggable database provides a better solution to consolidate multiple database services. In Chapter 4, we explain in detail how the pluggable database feature works in the Oracle RAC 12c environment and how to implement the multitenancy of database services by consolidating multiple pluggable databases into a single container database running on Oracle RAC 12c.

Cost of Ownership

One of the possible reasons that IT departments are looking at Oracle RAC is to reduce the cost of ownership of the database infrastructure. This cost saving is relative, depending on what you are comparing. The cost of Oracle RAC implementation includes three parts: hardware infrastructure cost, Oracle software cost, and management cost. The hardware stack consists of multiple servers, redundant networks, and shared storage. The software cost mainly includes Oracle RAC license and the Oracle Database license. For Oracle Database Enterprise Edtion, the Oracle RAC license is separate from Oracle Database license, while for Oracle Database standard edition, the Oracle Database license already includes the Oracle RAC license which you don’t have to pay for separately.

One of the limitations of Oracle Standard Edition is that the total number of CPU sockets of all the servers in the cluster can not go beyond 4. A CPU socket is a connection that allows a computer processor to be connected to a motherboard. A CPU socket can have multilple CPU cores. For example, a Dell R820 server has four CPU sockets while a Dell R720 server has two sockets. Since each socket can have 8 cores, an R820 server can have up to 4 * 8 = 32 CPU cores, and a Dell R720 server can have up to 16 CPU cores. Using Oracle Standard Edition with a maximum capacity of 4 CPU sockets, you can make a two-node Oracle RAC cluster with Dell R720 servers, but only a one-node cluster with a Dell R820 server. For Oracle Enterprise Edition, the RAC license can be based on the total number of processors. This is based on the total cores of the servers in the cluster. For example, for a two-node Oracle RAC configuration using Dell R720s with 8 core CPU sockets, the total number of the CPU cores can be 2 * 2 * 8 = 32.

Management staff cost is related to the cost of training and attracting individuals with the skills needed (system admins, network admins, and DBAs) to manage it. The hardware and software costs include the initial purchase cost as well as the ongoing support cost. Although this cost is higher than a simple database solution like a single-node MS SQL server, the RAC solution is cheaper than typical complex mission-critical databases running on big SMP servers, as Oracle RAC is mainly implemented on Linux and industry-standard low-cost commodity hardware. In the last decade, these industry-standard servers running Linux have become much cheaper, and offer a powerful and reliable solution widely accepted for enterprise systems.

Another cost-saving factor is that Oracle RAC can be implemented as a shared resource pool to consolidate many databases. This can significantly reduce the costs of hardware, software, and management by reducing the number of systems. In the Oracle E-Business database consolidation example mentioned in the last section, 100 databases were consolidated onto a 16-node RAC. The number of database servers was reduced from 100 or more to 16. The reduction led to huge savings in hardware, software, and management. As already mentioned, long-term operating costs are also cut by reducing the need for support, maintenance, and even powering and cooling 100 systems in a data center for the entire life cycle of the environment. For full details of this example, refer to my technical presentation at Oracle OpenWorld: http://kyuoracleblog.files.wordpress.com/2012/09/ebs_dbs_on_11gr2_grid_oow2011_session8945.pdf.

High Availability Considerations

Oracle RAC provides HA of the database service by reducing unplanned and planned downtime caused by server failure. But RAC itself doesn’t protect the database against other failures, such as storage failure, data corruption, network failure, human operation error, or even data center failure. To provide complete protection against these failures, additional measures need to be taken. Oracle MAA (Maximal Availability Architecture) lists the guidelines and related Oracle technologies needed to protect databases against those failures.

During the deployment of RAC, it is critical to follow HA practices to ensure the stability of the RAC. The most important hardware components that Oracle RAC relies on are the private network and shared storage. The private network should be based on a redundant network with two dedicated switches. Chapter 9 discusses the RAC network in detail. The shared storage access should be based on multiple I/O paths, and the storage disk drives should be set up with a RAID configuration and Oracle ASM disk mirroring to ensure redundancy. Chapter 5 discusses storage best practices in detail.

In theory, Oracle RAC protects the database service against failure of up to N-1 servers (where N is the total number of servers). In reality, if all of the N-1 servers fail, the workloads of the entire clusterware will be on the only surviving node, and the performance will definitely suffer unless each server leaves N-1/N headroom. For example, for a four-node RAC, leaving 3/4 (75%) headroom would not be realistic. A realistic approach is to ensure that each server in the cluster can handle the failed-over workload in case of single server failure. This requires each server to leave only 1/N headroom. And the bigger N is, the less headroom is needed. The worst case is a two-node RAC, where each server needs to reserve 1/2 (50%) headroom. For a four-node RAC, only 1/4 = 25% headroom is needed.

Scalability Considerations

Oracle RAC provides database scalability. With the addition of each extra RAC node, the cluster is expected to increase database performance capability: handling larger workloads or more concurrent users, performing more TPS (transactions per second) for OLTP, or reducing the average transaction/query response time. However, many RAC databases may not show linear scalability when adding more RAC nodes. This is because there are many other factors that are related to database scalability:

1. Poor database design and poorly tuned SQL queries can lead to very costly query plans that may kill database throughput and significantly increase query response time. Poorly tuned queries will run just as badly (or even worse) in RAC compared to a single-node database.
2. There may be quite costly performance overhead caused by Oracle cache fusion, and excessive wait time on data blocks transferring on interconnects between RAC nodes during query executions and database transactions. These wait events are called cluster wait events. Cache fusion overhead and cluster wait events may increase when multiple RAC instances access the same data blocks more frequently. A higher number of RAC nodes also contributes to cluster waits and slows down the interconnect. The number of RAC nodes is limited by the bandwidth of the interconnect network, which is less of an issue with the introduction of high-speed networks such as InfiniBand and 10-40GB Ethernet.
3. In some database environments with I/O-intensive workloads, most performance bottlenecks are on storage I/O with lower CPU utilization. Such environments will not scale well just by adding more RAC nodes. Therefore, it is important to understand the workload characteristics and potential performance bottlenecks before we opt for the scalable solution. Storage performance capacity is measured in IOPS (I/O operations per second) for OLTP workloads and throughput MB/second for DSS workloads. Since the speed of hard disks is limited by physics (drive seek time and latency), they tend to impose an upper limit on IOPS and throughput. One way to scale storage performance is to add more disk drives and stripe the data files with either RAID or Oracle ASM striping. Another option is to move frequently accessed data (hot data) to solid disk drives (SSDs). SSDs provide much higher IOPS, especially for the random small I/O operations which dominate OLTP workloads, as SSDs have no moving parts and hence no mechanical delays. Using SSDs is a very viable option to scale storage IOPS performance for OLTP workloads. For DSS workloads, one option is based on the building block concept. Each building block is composed of a RAC node plus additional storage and network based on the balance between CPU processing power and storage throughput for a DSS/Data warehouse–type workload. Scalability is based on the building block instead of just a server.

RAC or Not

When IT organizations need to decide whether to deploy an Oracle RAC as their database architecture, IT architects and DBAs need to make decisions based on many factors.

1. The High availability SLA: How much database downtime is acceptable for both unplanned and planned downtime? Without RAC, planned downtime for hardware and software maintenance may vary from a few minutes to as much as a few hours. And the unplanned time for hardware and software problems can also vary from minutes to hours. If a database server is completely lost, it will take a longer time to rebuild it, although some downtimes, like the complete loss of the server, may occur only rarely, Are these potential downtimes acceptable to the business according to the SLA? If not, can downtime prevention justify the cost of the RAC deployment? And furthermore, a loss of the entire database system including the storage may take hours or days to recover from the backup. Does this justify a Disaster Recovery (DR) solution which consists of a completely duplicated system in another data center? Some mission-critical databases may be equipped with the combination of RAC and Data Guard DR solution to protect the database from server failure as well as storage and site failure. However, the cost and technical complexity need to be justified by business needs.
2. Scalability Requirement: What kind of workloads are there and where is the performance bottleneck: CPU intensive and/or storage I/O intensive? If there is no need to scale out CPU/memory resources or if we can scale up by adding additional CPUs or memory, Oracle RAC One Node (instead of multiple RAC RAC) may be a good way of providing HA without the need to pay for an RAC license. RAC One Node also has the flexibility to be easily upgraded to a full RAC solution any time there is a need to scale out to multi-node RAC in the future.
3. Database Consolidation Requirements: If the organization has a business requirement to consolidate many database services together, a multi-node RAC can offer significant advantages in terms of cost of ownership wjile providing HA and scalability to all the databases.
4. If the organization decides to deploy the RAC solution, it should fulfil the hardware requirements and follow configuration and management best practices to ensure that the RAC provides all the potential advantages.
5. Many companies have carried out successful migration of their mission-critical databases from big SMP Unix machines to multi-node Oracle RAC clusters based on lower-cost industry-standard commodity X86-64 servers running Linux. In the last decade, these servers have advanced significantly in terms of reliability as well as processing power. Practical experience has shown that the new architecture can provide a highly available and scalable infrastructure for enterprise-level applications.