CHAPTER

22

Virtualization Summary and Best Practices

This chapter summarizes the key concepts and tasks presented in this book relating to the Oracle virtualization platform. This chapter uses two approaches: the minimalist approach (sometimes confused with “least expensive”) and the realist approach, whereby benefits of virtualization are realized using recommended best-practice methods without breaking the bank. For example, although it is true only one virtual machine server is required to deploy guest VMs, and although such a configuration may be deemed acceptable for demonstration or discussion purposes, a single-VM-server configuration cannot realistically meet even the most basic availability and performance requirements. In addition, a single Oracle VM Server configuration is also limited to the scaling capability of just that machine (that is, the number of CPU sockets, memory slots, expansion slots, and so on). Meanwhile, a recommended best practice is to have three or more virtual machine servers. Again, depending on the objective, three, five, seven, or nine OVS machines may be required in the long run, but the initial deployment requirements are satisfied with five servers.

Each approach (minimalist or realist) will be followed by a summary of post-deployment, management, and operations best practices. These include the concepts and tasks in support of backup and recovery, monitoring and tuning, and capacity planning, which will primarily focus on storage volume, network traffic, and process performance, including virtualization administrative tasks such as VM migrations and patching.

Oracle VM Resource Basics

The importance of planning could be emphasized at the beginning of every section of every chapter and it would still not be emphasized enough. The following sections summarize the key concepts when standing up a new Oracle VM environment. Each section is based on the primary computing environment resource—namely, storage, networking, memory, and CPU.

Storage

The source of storage from the perspective of the guest VM is either storage presented by the repository, storage presented directly to the guest VM by a SAN or a file server, or storage residing locally on an Oracle VM server. Generally speaking, all storage in an Oracle VM environment is based on SAN storage, file server storage, or storage residing locally on an Oracle VM server. The main difference is whether or not additional maintenance and operations are required when you’re growing the storage used by the guest virtual machine. Regardless of the source (SAN, NFS, or disk attached locally to the Oracle VM server), if the guest VM is using storage presented through a repository, and the repository has available space, then the guest VM storage can grow dynamically without additional SAN, NFS, or VM server local disk administration. If, however, the guest VM is using storage presented directly from the SAN, NFS, or local disk (as opposed to presented through a repository), then additional administration is required to grow the LUN.

Repository-Based Storage

The storage used by an Oracle VM storage repository is based on SAN storage, file server storage, or storage residing locally on an Oracle VM server. Although there are performance implications between using network-based storage versus locally attached storage, it is more important to understand the maintenance and operational issues when you’re considering using either storage method. Figure 22-1 shows information about physical storage that is locally attached to the Oracle VM server. In this case, the storage is a 44TB array defined as a single LUN. The Oracle VM server detects the storage and lists it as available. This storage can be used when you’re defining the repository.

FIGURE 22-1.    Repository using locally attached storage

Using locally attached storage, however, limits the VM migration capabilities regardless of the way the storage is presented to the guest VM (that is, through a repository or directly) and restricts any VM using the locally attached storage to operate strictly on the VM server where the locally attached storage resides. This limitation is somewhat resolved with the release of Oracle VM 3.4; however, there are still some limitations that have an impact on live migration and are covered in greater detail in the release notes.

The key point is that guest VMs using repository-based storage have the flexibility to grow the storage allocated to them dynamically from storage available in the storage repository VM without having to allocate additional storage LUNs from the SAN or grow a file system. Repository-based storage is therefore a general best practice to use as a source of storage for guest VMs.

Network-Based Shared Storage

Network-based shared storage presented to the Oracle VM servers either is a storage area network (SAN) using the iSCSI protocol or is based on file server sharing (NAS) using NFS exports/shares. In either case, a virtual disk used by a guest VM is defined at the storage layer with a specific size, and it requires manual intervention to grow if additional space is needed. Figure 22-2 shows the Storage tab of the VM manager GUI, which lists a 2TB LUN that was created in the SAN server “prodsan1” and is currently being used by the repository “PHYrepo1.”

FIGURE 22-2.    Storage tab showing SAN storage

This LUN could have instead been presented directly to a guest VM, but that would have been a 2TB dedication. Because the LUN has been presented to the repository PHYrepo1, the 2TB is available to any VM and the storage can be allocated based on the VM’s need. For example, if an Oracle Linux guest VM (OLVM1) were to be created and needed two disks (for example, 20GB for the root file system and 80GB for everything else), that guest virtual machine (once created) would use a total of 100GB out of the 2TB allocated to the repository, and the repository would still have 1.9TB available for other virtual machines. Meanwhile, if the storage requirements of OLVM1 were to increase, then the disks of that guest virtual machine could be increased to whatever size is necessary (provided it’s less than 1.9TB), and no other administration would be required because the storage is available in the storage repository. If the two discs created in this example were allocated as individual LUNs, then a system administrator would be required to grow each of the LUNs at the operating system before the disks could be grown in the guest virtual machine.

There may also be performance benefits between using physical LUNs versus virtual disks out of a storage repository. However, each environment and situation should be tested to determine whether the performance benefits outweigh the additional administration required to manage the shared storage as opposed to using repository-based storage.

Locally Attached Storage

A given VM server may have storage that is locally attached and is therefore only available to processes running on that server. With the release of Oracle VM version 3.4, locally attached storage can be migrated, but other limitations generally prohibit the use of locally attached storage. For these reasons, use of locally attached storage is discouraged. However, with that said, locally attached storage does work well for the minimalist approach because with that approach the Oracle VM architecture very likely has only one VM server, so live migration activities are very likely not an issue. Figure 22-3 shows locally attached storage for VM usage.

FIGURE 22-3.    Locally attached storage for VM usage

Networking

Generally speaking, there are two primary considerations in the Oracle VM network environment: the physical network components and the virtual network components, also known as network channels. Figure 22-4 shows the Networking tab in the Oracle VM Manager GUI. From the minimalist perspective, there will be one physical network interface card (NIC) per machine, and all Oracle VM network channels use the single NIC. This is an acceptable configuration from the perspective of an operational environment. However, even from a minimalist perspective, in production there should be at least two physical NIC adapters per network path, and these should be bonded to provide at least a minimum sense of availability (in most cases, this provides performance benefits as well).

FIGURE 22-4.    Networking tab

Oracle VM Network Channels

The purpose of the Oracle VM network channel is to provide a dedicated path in support of specific Oracle VM networking operations. Ideally, each of the network channels would have one or more dedicated physical NIC network connections. Whereas the minimalist approach may have only one NIC (or at least a bonded pair of NIC adapters) to be shared by all network channels, a realist would approach the configuration with the following considerations:

   Storage and Virtual Machine network channels    These network channels are used to provide the path for the virtual machine virtual network (communications between Oracle VMs and from the Oracle VMs to the Internet and resources on the local network). It is a recommended practice to define VLAN virtual adapters for the storage and network, as well as to manage virtual machine communications. Figure 22-5 shows an example of the VLAN adapter definition screen. In this example, two VLAN interfaces have been defined. They are on two separate bond ports of two Oracle VM servers. In this example, no addressing was chosen for the VLAN interfaces, but they could’ve easily been defined as Static or Dynamic. An IP address and subnet mask would be required if Static were chosen, and DHCP would be expected to provide an IP address if Dynamic were chosen. Ultimately, CIDR (classless inter-domain routing) addressing may be used for the VLAN adapter definitions to manage traffic and access to specific address ranges, as well as to provide additional security.

FIGURE 22-5.    VLAN adapter definition

   Server Management and cluster heartbeat network channels    These two network channels may be combined, given the notion that the cluster heartbeat is a high-priority virtual network, and the traffic on this network should not (if at all possible) ever be degraded or impeded because the cluster heartbeat is used to monitor the overall health of a given Oracle VM server. If a problem is detected for a given Oracle VM server on the cluster heartbeat network, that particular Oracle VM server may be “fenced” and the associated virtual machines will be migrated to another Oracle VM server in the cluster. The Server Management network channel is used by the agents running on the Oracle VM servers.

   Live Migrate network channel    As the name implies, this network channel is used during Oracle guest VM migration operations. Migration operations occur manually or indirectly. During live migration, the memory semaphores of an Oracle guest VM are moved from one VM server to another. This operation may also be performed for a stopped guest virtual machine, in which case there are no memory semaphores to move because the guest VM is not currently running. Indirectly, however, if the agent running on an Oracle VM server detects unacceptable latency or an outage of the heartbeat, then the agent spawns the request to migrate running guest VMs to another VM server in the cluster. It is therefore a recommended best practice to dedicate a physical network path to live migration operations.

Memory and CPU

The minimalist approach to memory is simple: pack the Oracle VM server with enough memory to support the guest VMs, plus a little more for the hypervisor. This is not far from reality. Although it is not required to have the same amount of memory on each of the VM servers, it is recommended that all the servers in a given cluster are of the same “family” in terms of CPU, and any given VM server should conceivably be able to handle the load of all guest VMs. Realistically, this should never be expected, especially for some larger implementations. However, there are no rules to the size and speed of memory, save one: more is better.

Again, although it is true no given Oracle VM server should be expected to run all Oracle VMs simultaneously, the consideration must be made that in the event of a failure of one or more Oracle VM servers, the remaining Oracle VM servers should be able to sustain the operations of the guest virtual machines currently running. At best, there should be sufficient memory in a single Oracle VM server to run the mission-critical guest VMs. This is the minimalist approach. Realistically, however, a given Oracle VM server or several Oracle VM servers in a cluster should have sufficient memory to support the mission-critical guest virtual machines as well as a few additional servers, for example, servers used for administration and maintenance (Oracle Enterprise Manager RMAN catalog database, etc.) Also keep in mind that the memory resources are not “shared” across the Oracle VM servers (which would really be nice).

Finally, the amount of memory on a given VM server needs to be sufficient to run not only the guest VMs, but also to support the requirements of the dom0 hypervisor. Figure 22-6 is an example of a guest VM showing memory and CPU cores. Rule of thumb: add up the memory required by each guest VM and then add another 4 to 8GB.

FIGURE 22-6.    VM server-based guest VMs showing memory and CPU cores

With respect to CPU requirements, as with memory, the number of available CPU cores on a given Oracle VM server should be based on the number of CPU cores required by the virtual machines, plus one or two CPU cores for the hypervisor dom0. For example, if there are 10 guest machines, and each guest VM requires two virtual CPUs, then the VM server should have a minimum of 22 cores available: (two cores per guest VM × 10 guest VMs) + two cores for the hypervisor dom0. Again, this is the minimalist approach. Realistically, because there should be at least three Oracle VM servers, and the load should be spread more or less equally across the VM servers, then each Oracle VM server should have approximately eight CPU cores. Keep in mind this does not mean 12 sockets. In today’s world, a 24-core equipped machine may fit on just three CPU sockets.

In short, both the minimalist and realist approaches to CPU and memory are basically the same: more is better (that is, more available physical resources), and ultimately the virtual requirement must be equal to or less than the physical resources available. The difference between the two approaches (minimalist and realist) with respect to CPU and memory resources is a factor of capacity rather than performance; in other words, regardless of the CPU core “speed” (in terms of hertz) or the cores-to-socket ratio, the minimum number of CPUs “to go into the plan” should be equal to or greater than the number of virtual machines. Similarly, for the memory, the total physical memory should be equal to or greater than the maximum that will be allocated (or allowed) to the respective guest virtual machines. Again, a resource’s speed is not the limiting factor.

Oracle VM and Resource Virtualization

The discussion of computer system virtualization is often framed from a data center and hosted operating system perspective with concepts and components related to “X as a Service.” When presenting a recipe, the chef first discusses or explains the ingredients needed. In the case of virtualization, the ingredients equate to the resources defined, configured, integrated, and finally deployed and managed in support of virtualizing the application and database systems. These resources are then presented as a service to the cloud customers. These ingredients, or resources, include storage, networking, memory, and CPU.

   Storage    Shared storage that is defined for the Oracle VM repository is a general best practice. An NFS-mounted file system can be used for the repository and may be easier to manage than a LUN carved out of SAN storage, but the SAN storage may perform better. Ultimately, the guest VM use of storage from the repository is easier to manage because the storage is served out of the repository and managed by Oracle VM. Storage that is carved out of a SAN LUN must be manually adjusted in the SAN to accommodate an increase in storage requirements. If space from a storage repository were used, then the same requirement increase can be handled within Oracle VM alone, without a change in the SAN—the difference being that the storage repository must have been defined with the extra storage to start with.

   Networking    The minimalist approach is to run all network channels out of a single NIC or a single bonded pair. However, the realist approach is to dedicate physical resources to a logical grouping of the network channels such that the cluster heartbeat is the most volatile for a clustered server pool configuration, and each of the remaining network channels should be configured with VLANs. CIDR addressing (at the VLAN adapter) may also be used based on the deployment requirements. It should be noted that although the Oracle VM networking facilities provide virtual switching and networking capabilities, the physical network must also be configured with the appropriate VLAN characteristics required by the Oracle VM network.

   Memory and CPU    Simply put, more is better. There should be one or two more CPU cores than required by the sum of all guest VMs running at a given point in time. Regarding memory, there should be 4 to 8GB more memory than required by the sum of all guest VMs running at a given point in time.

The Oracle VM Start to Finish

This section covers a hypothetical deployment of an Oracle VM technology stack. After the Oracle VM Manager, Oracle VM server, guest Oracle VM templates, and the requisite Oracle Linux software have been acquired, the concepts and tasks for deploying an Oracle VM installation begin with planning the configuration, with the objective of sustaining some basic guest VMs.

The source of the installation and upgrade software should be the Oracle eDelivery site “Oracle Software Delivery Cloud” whenever possible. Although there are many other sources of software, including the Oracle Technology Network, only the eDelivery site is guaranteed to contain production-ready bundles. The URL for this site is https://edelivery.oracle.com, and it contains the Oracle VM software and templates required for most deployments. Figure 22-7 shows a partial Oracle software-delivery site listing of Oracle VM Server software. If the term “template” would have been included in the search, the result would have included 38 products, each of which require further refinement to drill down to the specific required template.

FIGURE 22-7.    Oracle software-delivery site listing VM software

In preparation for our hypothetical Oracle VM environment deployment, the following software should be downloaded:

   Oracle VM Manager

   Oracle VM Server

   Oracle Linux template (optional)

   Oracle Linux server software (optional)

   Other operating system(s), such as Windows, CentOS, and so on

Storage, Pools, Servers, and Repositories

The downloaded software will be used to establish the environment. As discussed previously in this book, it is a good practice (if not a best practice) to establish a local yum repository. Although it is not required, the local yum repository can be used to store the entire set of Oracle VM software. It is mentioned here purely as a suggestion, but it is not required. In fact, there is no “yum install” command to deploy the Oracle VM Manager or VM Server software. The best practice is to install the VM Manager into a supported bare-metal-based operating system using the process documented in this book, along with the Oracle VM documentation. The yum repository is an option for storing and deploying subsequent patches as they become available. Oracle VM patches can be downloaded and deployed into the local yum repository so that the software is available not only to the Oracle VM Manager and Oracle VM Server machines running the Oracle VM environment, but also to the guest VMs deployed into the Oracle VM environment.

Deploying Oracle VM

The deployment of Oracle VM begins with planning. The Oracle VM Manager is the first major component to deploy. As such, the following serves to clarify some points:

   OVM Manager should not be installed on the same server as Enterprise Manager. The OVM Manager installation cannot use the WebLogic server that is installed with Oracle Enterprise Manager. The OVM Manager installation requires its own technology stack.

   Oracle VM Manager is an application that is installed on a normal Linux operating system such as Oracle Linux or Redhat Linux 5.8+ (including 6 and 7), and it can be a physical or virtual machine. (However, you should learn the process first before complicating it with virtualizing the virtualization management tools.)

   The Oracle VM Manager doesn’t need anything special for networking. It just needs enough network to allow connectivity; nothing else is needed.

   The Oracle VM Manager does not need access to SAN or NAS storage used by the OVM servers (this is an especially misunderstood aspect of storage).

   The Oracle VM Manager GUI is the best place to manage the Oracle VM environment. Very few operations (by design) are performed from the command line outside the Oracle VM Manager. Unless directed specifically by Oracle Support, you should perform all Oracle VM management operations from within the Oracle VM Manager environment.

Assuming enough thought and planning have been given to the storage and networking, a typical deployment will have storage served to three or more Oracle VM servers through multiple repositories. It is advisable, if not a best practice, to group the Oracle VM guest virtual machines of a similar type, business system, or role into a given repository or group of repositories, which makes backing up the guest VMs and their associated data much simpler and more logical. In other words, you should create each repository with the intent of putting logically grouped VMs into it. This practice applies not only to repositories, but also to the server pools and network-creation tasks. Figure 22-8 shows an example of logically grouped VM repositories, where there are multiple repositories for guest virtual machines based on physical disks and another repository for guest VMs targeted for cloud deployment.

FIGURE 22-8.    Logically grouped VM repositories

With respect to the planning of the server pools, the most important restriction to remember is that the clustering is only available at creation time. A nonclustered, stand-alone server pool cannot be dynamically “clustered.” It must be re-created as a clustered server pool. The converse is also true: that is, a clustered server pool cannot be made into a nonclustered, stand-alone server pool but rather must be dropped and re-created. The main difference between clustered and nonclustered server pools is the OCFS2 file system used to support the clustered server pool. The crux of the biscuit: it is perfectly acceptable (and is a recommended best practice) to create the server pool as a clustered server pool with only one server and then add servers down the road as the Oracle VM environment grows.

The first major task is to install the Oracle VM Manager on a machine. Again, generally speaking, it is this author’s opinion that you should run Oracle products in and around other Oracle products. In other words, it is a recommended best practice to deploy Oracle Fusion middleware, for example, within an Oracle Linux environment, and if you’re virtualizing the environment, to do so with Oracle virtualization products. This may be the most obvious of general best practices, but it is also no doubt the most difficult to address, especially if the given environment is running another flavor of virtualization product.

Clustered server pools are defined at the time of server pool creation. A server pool is created using one or more Oracle VM servers; the recommended number of Oracle VM servers is three (or more), but it should also be an odd number of Oracle VM servers (three, five, seven, and so on). In a larger environment, it is also a best practice to logically group the Oracle VM components. Logical groupings include the following:

   Logically grouped VMs within a repository    Having multiple repositories is a common best practice. Each repository is a related collection of VMs and the related storage and associated components (ISOs, templates, and assemblies).

   VM servers within a server pool    Some large deployments will have 13 or more VM servers; grouping the Oracle VM servers and running specific guest VMs in a given server pool is akin to separating hardware environments in a data center.

   Guest VM networks Having multiple guest VM networks helps to isolate traffic based on the processing requirements and system’s objective. Figure 22-9 displays an example of multiple VM machine and storage networks. Separating networks also provides additional security by isolating guest VM communications in one group from the communications of another group of guest VMs. In the example, the “Customer” network supports the guest VMs of a “customer” group of VM guests, whereas the “vmnet” network supports a network for development activities, including functional/systems integration testing, QA (quality assurance), and training.

FIGURE 22-9.    Networking tab showing multiple channels for storage, virtual machines, and Live Migrate

Summary

This chapter summarized the most important basic best practices of the Oracle VM environment. Namely, you should logically group guest virtual machines into repositories that can be placed into a backup strategy whereby the repository backups capture the logical grouping of guest VMs, including the guest VM, the configuration, and the data presented through the repository. Additionally, repository-based storage is presented in comparison to storage offered by the SAN or file server directly to the guest VM.

With respect to networking, a similar concept prevails as a best practice: to logically group the traffic used by the guest virtual machines into VLANs and furthermore to control the grouping of the traffic (while adding a layer of security) through CIDR addressing schemes.

This chapter also showed that some of the best practices allow for a minimalist approach as a proof of concept, while leaving open the opportunity to grow into a more realistic deployment using Oracle VM software for virtualizing the data center in the cloud.