After a failover, the applications that were running on the failed instance will be running on one or more of the surviving instances. OPS instances must be able to accommodate this additional workload. You need to design your OPS environment so that any instances that you plan to use for failover purposes have enough extra memory and CPU resources to handle the additional workload that results when a failover occurs. Otherwise, a failover might result in a performance bottleneck. Another approach to this issue is to design your applications so that the workload on nodes designated as failover nodes is reduced when a failover occurs. You could, for example, stop nonessential reports from running in order to free up resources for mission-critical transaction systems.

Redistributing applications in an OPS environment during a failover has implications for application partitioning. After the workload has been redistributed, the data access patterns of the nodes will be different. If you don’t properly plan for this situation, the surviving nodes may experience increased pinging as a result of contention for the same database objects. This will have an adverse impact on performance. This issue is not a factor in a two-node OPS setup, because in that setup, when one node fails, only one node remains to access the database. However, in an OPS environment with several nodes, your failover planning should take into consideration the application partitioning issues discussed in Chapter 11.

In a two-node OPS environment, you can use a new feature of Oracle8i to configure one OPS node as the primary instance and the other as the secondary instance. You activate this feature by adding the following initialization parameter to the parameter files for both of the instances:

ACTIVE_INSTANCE_COUNT = 1

The instance that first mounts the database becomes the primary instance, and it (and only it) registers with the Net8 listeners. During normal operation, only the primary instance will accept connection requests from clients. The instance that starts last becomes the secondary instance and does not register with the Net8 listeners. For this reason, no client connections can be directed to the secondary instance. The Integrated Distributed Lock Manager (IDLM) monitors both instances. If the IDLM detects the failure of the primary instance, it notifies the secondary instance to take over. The secondary instance then registers itself with the Net8 listeners and accepts client connections. With this setup, contention between nodes is never a problem, because at any one time only one node is accessing the shared database. Application partitioning is not required.

With application failover, server-side database recovery and reconnections to the backup instance are transparent to end users. However, these activities have a performance impact on the surviving OPS instances, and they may cause client applications to stop or be delayed during failover. Failover performance is affected by the following:

It’s the cluster manager’s job to detect node failure, and the SMON process of a surviving instance detects instance failure. The cluster manager detects node failure by monitoring a periodic heartbeat signal generated by each of the nodes. When a node fails, the heartbeat from that node is lost. The cluster manager will eventually detect this loss of heartbeat communication and recognize that a node failure has occurred. The time required for the detection of a failed node depends on the threshold value set for the heartbeat communication. It’s usually on the order of a few seconds.

After a node failure has been detected, the IDLM needs to reconfigure itself to run on the surviving instances. IDLM redistributes lock information among the surviving nodes. This process is referred to as lock remastering. The time required for lock remastering depends on the number of PCM locks allocated to various datafiles. This is one of the tradeoffs involved in PCM lock allocation. On the one hand, a higher number of locks provides better protecton against false pings, but on the other hand, a high number of locks delays recovery from a node failure.

Instance recovery is another task that must take place after a node failure. One of the surviving instances will automatically start instance recovery for the failed instance, so that no data from committed transactions is lost. Instance recovery involves rolling forward through the redo logs and applying all the changes made by the failed instance. The time required for this phase depends on the amount of redo that needs to be applied. You can minimize the amount of redo, and consequently the time needed for recovery, by having checkpoints occur frequently.

Not only does instance recovery involve the application of committed changes, it also involves the rollback of all uncommitted transactions that were in process on the failed instance. The time required for rollback depends on both the number of uncommitted transactions and the amount of data that these transactions modified. A new Oracle8 feature called deferred transaction recovery reduces the impact of this rollback activity by allowing new transactions to take place during the rollback phase of instance recovery, as long as those new transactions do not need to access rows involved in the uncommitted transactions being rolled back. If this happens (i.e., a new transaction needs to access rows involved in the uncommitted transactions), then only new transactions have to wait for the rollback of the failed transaction. A further improvement in Oracle8i, referred to as fast-start rollback, allows a new transaction to roll back only the rows it needs (rather than roll back the full transaction) when those rows are involved in the uncommitted transaction. This Oracle8i feature reduces recovery time when a long-running transaction is involved with the new transaction.

Another factor that impacts failover performance is the number of database connections to the failed instance. If the number of failed connections is large, then you might experience a performance impact as a result of all those connections reconnecting to the surviving instance(s) at the same time. With dedicated server connections, it is necessary to spawn a shadow process for each connection request. This has a significant performance impact. In an MTS configuration, the impact is much less because dispatcher processes are used to connect new clients to preexisting shared server processes.

In later sections, we’ll explain that it’s possible to preconnect to a backup instance before failure of the primary instance. In this case, failover is quicker because connections to the backup instance already have been established. However, preconnecting takes up computing resources in the backup node during normal operations. Response time after a failover, however, still depends upon the capacity of the backup node to process the redirected workload of the failed instance.