8. Choosing a Virtualization Technology

The preceding chapters described different forms of server virtualization, each with its own strengths and limitations. But which is the right combination to use?

No one answer applies equally well in all situations. Each form of virtualization has both strengths and weaknesses, which must be weighed against each other. One approach might provide great value in one situation, but offer little value in another scenario. Some virtualization solutions can be combined, making it possible to benefit from the advantages of more than one virtualization technology at the same time.

This chapter discusses potential advantages and disadvantages of layering virtualization technologies. It also provides guidelines for selecting the right virtualization technologies for different requirements.

A key component of virtualization is workload isolation. This takes three forms: hardware, software, and security isolation. Hardware isolation prevents a hardware failure from affecting more than one virtual environment (VE). Software isolation prevents a software failure from affecting multiple environments; it also provides separate namespaces for files, users, and processes. Security isolation permits the establishment of separate security contexts, with separate user identities, roles, and authentication and authorization mechanisms for each VE. The effect is containment of intrusions to just one VE. Although security isolation often focuses on web-facing workloads exposed to public networks and on systems housing sensitive financial, medical, or government data, it is important to secure all environments in a computing infrastructure to prevent attacks against the weakest link. Thus, organizations are well advised to architect and integrate security on a systematic basis, rather than reserving it for “high-value” targets.

Those features providing the most isolation can add significantly to the costs of a system. Nevertheless, business-critical workloads are most in need of comprehensive isolation and can often justify the extra expense of supporting independent VEs. Any additional expenses may be offset by the savings provided by consolidation.

Different forms of virtualization offer different levels of software compatibility. With some approaches, there are few or no incompatibilities: Software runs essentially the same in a virtualized environment as it does in a non-virtualized system. With others, the virtualization infrastructure performs some operations on behalf of a VE that are implemented differently than the corresponding operations on non-virtualized systems, or functionality is limited in some way. These considerations were discussed in previous chapters and will be reviewed later in this chapter as needed. In all cases, incompatibilities should be analyzed and software retested to ensure that no problems will arise in system functioning.

Recertification of software products, if necessary, should be performed by the software vendor. In such cases, in-house testing should be minimal. Software developed in-house should be analyzed or retested if it is to be used on the types of virtualization that affect program behavior.

Most forms of virtualization create at least some level of performance overhead. This overhead stems from the time that hypervisor spends running on a CPU to prevent unwanted interactions between VEs; inefficiencies caused by emptying the CPU cache after one guest runs for a time slice, before the next guest runs on that CPU; and virtual I/O, which performs many additional steps to prevent prohibited actions. The precise amount of overhead depends on the type of virtualization and on the workload. CPU-intensive workloads are the least affected because they require the least effort from the virtualization technology. The simplest case is containers (e.g., Oracle Solaris Zones) because their processes interact with the operating system kernel just as they would in a non-virtualized environment. No additional code or effort is required on the part of other components, so there is no performance overhead. In other simple cases, a VE gets a time slice on a CPU without further assistance from the virtualization technology. By comparison, I/O-intensive workloads experience performance overhead in some virtualization solutions due to resource contention and an increased number of state-changing operations, especially when I/O devices are shared. Latency-sensitive workloads should not be installed in VEs that suffer from such performance overhead without testing to ensure they can still meet their performance objectives.

VE scalability is a related performance topic. Some types of virtualization can host only a limited number of VEs with acceptable performance. Most can theoretically run dozens or even hundreds of VEs. In practice, however, most of those solutions do not scale well: They may need terabytes of RAM, they may be limited by the scalability limits of the hardware architecture to which they are tied, or their current implementation may create so much performance overhead that they can scale to only 10–20 VEs. Others can effectively handle more than 100 VEs. This effect is often mitigated by the common practice of consolidating physical systems from old, slow, low-utilization platforms onto new processors. In such circumstances, the modest resource requirements and the significant increase in processor speed may compensate for the possible overhead. To maximize the cost savings achieved from consolidation, an organization should choose the most scalable virtualization solution.

The resource requirements for many workloads change over either short or long periods of time. An ability to dynamically change the amount of resources available to those VEs is necessary to maintain appropriate throughput and response times.

Most applications are able to run on more than one version of an operating system. Some are tied to a specific version of the OS or to a specific version of an OS package or a patch level. Such applications can benefit from the freedom to run different VEs at different OS levels, thereby permitting applications with different OS requirements to coexist on the same server. Also, VEs with independent OS levels can use different patch windows for updating system software. Virtual environments that share a common OS software level may need a synchronized patch window. Unfortunately, this strategy may be difficult to realize on a practical level when different applications and different application owners share the same physical system.

Similarly, while many applications can run on different operating systems, others are available only for a particular OS. The ability to run different operating systems at the same time (e.g., Microsoft Windows and Apple Mac OS) also has value, and in some situations is an absolute necessity.

Some virtualization features enhance business agility and operational flexibility—key considerations that are discussed in Chapter 7, “Automating Virtualization.” Dynamic data centers, whose workloads regularly grow and shrink, will benefit the most from these features. Cloning, for example, is the ability to create an exact copy of a VE. It accelerates provisioning when many similar workloads will exist, and can even be performed as a programmed response to a request for a new instance of an application. Likewise, VE mobility can be used to move an application from one computer to another. This capability is very valuable in dynamic data centers where workloads grow and shrink regularly, and it facilitates ongoing availability of applications during maintenance periods for physical server hardware.

Virtualization has enabled the rapid proliferation of cloud computing environments. Both public and private clouds are possible because of the ease with which VEs can be created. This computing model greatly reduces the effort involved in managing a data center, and it increases business agility by shortening the time from making a decision to run an application and having a suitable environment in which to run it.

Architects choosing virtualization technologies to meet an application’s requirements must take all of the previously mentioned factors into consideration, mapping requirements onto those combinations of technologies that will best satisfy them. The following sections describe the virtualization technologies under discussion in this context.

Fault isolation. Software service and application failures in one zone have no effect on other zones.

Security. Zones remain securely isolated from one another. This separation permits extremely granular privilege assignments. Further, features are provided to configure immutable zones—that is, “locked-down” environments that are protected from modification. Tamper-proof execution environments are very difficult to achieve with other virtualization methods.

A high level of compatibility with non-zone Solaris environments. Zones have some functional limitations. For example, a zone cannot modify a physical device’s tuning parameters.

Native performance. The use of zones does not affect application performance because zones require no hypervisor or other virtualization software. The code paths of the OS and applications are unchanged, so there is no performance overhead. This outcome is typical for OSV, but unlike that for virtual machine implementations.

Maximum scalability, permitting the largest number of VEs per computer. The theoretical maximum is 8191 zones per OS instance. Production systems with more than 100 running zones exist in data centers.

Highest resource granularity. This characteristic permits fine-grained assignment of CPU capacity, RAM use, and allocation of other system resources.

Low overhead. Besides a lack of performance overhead (mentioned earlier), native zones require the smallest amount of additional RAM, and they do not waste resources by reserving them and leaving them idle.

More predictable performance than hypervisors. Because only one scheduler is making scheduling decisions, performance is more consistent.

Independence from processor family. Zones are available on any SPARC-, Intel-, or AMD-based computer that runs Solaris 11. This platform independence lets planners select a virtualization architecture without being locked into a chip family.

Centralized observability. Solaris tools enable you to examine all activities of all processes in all zones without buying or learning new tools. This is not possible with other virtualization methods.

Coexistence with Solaris versions. Solaris 11 environments can coexist with Solaris 10 VEs.

No extra license fees needed. Zones are simply a feature of the OS. Some hypervisors require a fee for their use, or a fee for support in addition to the support fees for the operating systems.

Business agility and operational efficiency. Zones’ features and flexibility exceed those of hardware partitions, owing to their advanced features such as VE cloning, mobility, live migration, and dynamic resource management.

Solaris kernel zones differ from native zones, leading to slightly different advantages for the former:

Solaris version isolation. Applications that require different versions of Solaris can run in different kernel zones, unlike native zones.

Fault isolation. Software fault isolation is greater than that of native zones.

A high level of compatibility with non-zone Solaris environments. Kernel zones provide even more consistency with non-zoned environments. System tuning decisions may be made for each kernel zone.

Low overhead. Kernel zones have low computational and memory overhead, albeit not as low as native zones. Each kernel zone uses more RAM than a native zone, and some virtual operations incur a performance penalty.

Solaris Zones were introduced with Solaris 10 in 2005, and have been continually enhanced and widely adopted since then. They permit robust, scalable, and highly flexible virtualization without committing to a particular chip architecture.

The benefits of Solaris Zones can be combined with the benefits of other virtualization technologies, thereby enabling the creation of highly functional and flexible virtualization architectures. When an Oracle Solaris system is hosted in a physical domain, logical domain, or other type of virtual machine, deploying application environments inside zones nested within those virtual environments provides additional benefits. Security is enhanced by zones’ ability to provide a locked-down environment. Zones can be quickly and easily cloned, which simplifies the provisioning of additional virtual environments. Perhaps most importantly, the low resource requirements and scalability make it possible to locate many isolated virtual environments within the same domain or virtual machine—with the end result being better scalability than would be possible with domains or hypervisor technology alone.

One way to realize the most benefits from this combination is to deploy a new domain or virtual machine when a virtual environment needs a dedicated operating system instance, and to deploy Solaris Zones within the virtual environment for Solaris virtual environments with compatible OS software levels. That combination exploits the advantages of both forms of virtualization.

Strong fault isolation. A software failure in one domain cannot affect another domain, and domains remain isolated from failure of CPU threads and other hardware resources assigned to other domains. The fault isolation characteristics of domains are better than those of both software-based hypervisors and OSV, as they permit redundancy with multiple service domains. Nevertheless, this level of fault isolation is not as good as that offered by hardware partitioning, especially when shared I/O resources are in use. Even so, the level of fault isolation available with logical domains is appropriate for almost all workloads.

Security isolation. A security penetration or compromise in one domain has no effect on other domains. Logical domains can be used to store sensitive data.

An extremely high level of compatibility with non-domained environments. Only a few incompatibilities exist, such as the specific behavior of a virtual disk or network device, as documented in the current version of the domains reference material. Except for those differences, software performs the exact same steps as on a stand-alone system.

Native CPU performance compared to non-domained environments for many workloads. I/O performance may suffer under shared I/O configurations, which can reduce the scalability for some workloads. Because CPU hardware threads are assigned to each domain, scalability is limited by the number of available threads on specific server models. This virtual machine technology premiered in 2007, on SPARC computers with 4 to 8 cores and 4 threads per core, for a maximum of 32 addressable CPU threads. At most 32 domains were possible, including the control domain. Newer products utilizing the SPARC M7 chip, such as the SPARC M7-16 server, now provide 32 cores with threads per core for every M7 processor. Now environments can run as many as 128 logical domains per system. This exclusive assignment of CPUs eliminates the need to time-slice guest environments onto CPUs, thereby avoiding the performance penalty experienced with other hypervisors.

A large number of completely separate Oracle Solaris instances. Each domain can have its own patch levels and maintenance windows without conflict with or interference from other domains. This approach compares favorably to most OS virtualization configurations, which may require keeping patch levels synchronized across some or all guests.

High resource granularity. Oracle VM Server for SPARC permits fine-grained assignment of CPU threads and RAM. The granularity associated with this approach is more flexible than with hardware partitions but less flexible compared to OS virtualization.

No extra license fees needed. The logical domains technology is included with all SPARC systems. Some hypervisors require a fee for their use or support in addition to acquisition and support fees for the operating systems and hardware.

Excellent support for business agility and operational efficiencies. Advanced features, such as domain cloning and mobility, support both flexibility and cost-effectiveness.

Oracle VM Server for SPARC has proved to be one of the most popular features of the SPARC server product line, with rapid uptake in its adoption and feature growth occurring since its introduction. This technology is widely used for server consolidation, especially when replacing multiple small, horizontally scaled servers from previous SPARC product families.

Best fault isolation. No failure in one domain or in its infrastructure can affect another. Other solutions carry at least some risk that a failure occurring in one VE will affect other VEs, perhaps even causing all of them to halt.

Security isolation. A security penetration or compromise in one domain has no effect on other domains.

Complete compatibility in every respect with non-domained environments. There is no concern about incompatibility or non-supportability of software.

Native performance compared to non-domained environments. There is no virtualization layer. The CPU code path of an application is not elongated by virtualization, and no additional firmware is needed to assist with virtualization. This advantage, compared to software hypervisors, is greatest for I/O-intensive applications. Nevertheless, domain scalability remains limited by hardware capabilities. The highest scalability is provided by SPARC M6-32 systems, which currently can have a maximum of 24 physical domains.

Completely separate Oracle Solaris instances. Each instance can potentially have its own OS version, patch levels, and maintenance windows without any conflict or interference. Management of a single domain is the same as management of a single system. This is similar to hypervisor solutions, but is often an advantage compared to OS virtualization.

Reconfigurability. On the SPARC M6-32 and M7-16, the sizes of the physical domains can be changed to meet business requirements. Although upgrading from a two-CPU SPARC T7-2 to a four-CPU SPARC T7-4 requires replacing the entire computer, physical domains on the high-end servers can be reconfigured from a 2-CPU partition to a 4-CPU partition, or even larger—up to a 32-CPU partition.

No extra license fees needed. The physical domains technology is included with all domain-capable systems. Some hypervisors require a fee for their use, or a fee for support in addition to support fees for the guest operating systems.

Well-established and accepted history. The SPARC product line benefits from two decades of partitioning development and deployment experience.

Physical domains are widely deployed in enterprise environments, especially for large applications. In such applications, ability to scale and the highest degree of isolation and availability are necessary for a relatively small number of vertically scaled virtualized environments.

Type 2 hypervisor. The user benefits from the rich set of tools supported on the underlying operating system, and use of that host OS does not suffer from any performance degradation because it is not running on top of a hypervisor.

Fault isolation. No software failure in one virtual machine can affect another VE, although a failure of the hypervisor may halt all of the VEs.

Security isolation. A security penetration or compromise of one guest has no effect on other guests. Given that most hypervisors are managed across a network, however, this creates an opportunity for an intruder to access a hypervisor directly; penetration of the hypervisor could theoretically give access to each guest, as demonstrated in the VENOM vulnerability.

High resource granularity compared to hardware partitions, permitting fine-grained assignment. Hypervisors often permit oversubscription of CPU threads and RAM for enhanced consolidation ratios.

Coexistence of multiple operating environments that run on the x86 platform. Use of VirtualBox provides an opportunity to leverage infrastructure investments for more heterogeneous OS mixtures. In contrast, most OS virtualization implementations support only guests of the same OS type, version, and patch level. Solaris is an exception: It supports Solaris 8 Containers and Solaris 9 Containers in Solaris 10 systems, and Solaris 10 Zones in Solaris 11 systems. All of these are VEs that mimic different versions of Solaris.

Are full fault and service isolation needed with no single points of failure?

What are the availability requirements of the workloads being virtualized?

Is there additional budget for licensing add-on virtualization technologies?

Are there business needs for automation and sophisticated management such as cloud provisioning, cloning, and mobility?

Does the solution require a large number of virtualized environments with highly granular resource assignment, or are only a few environments required?

Are workloads dynamic in nature, requiring the ability to easily change resource assignments?

Is maximum performance needed? This consideration is important for workloads that are I/O intensive or make many system calls, because some virtualization technologies introduce performance penalties for those operations.

Is an automated process needed for migrating from legacy unconsolidated environments or older operating system levels to a more current system?

While platform architecture may drive virtualization choices, this is not a one-way process: The virtualization capabilities of each platform can influence platform selection as well. For example, standardizing on Solaris makes it possible to leverage Solaris Zones regardless of the chip architecture employed. This approach provides a solution without extra costs, which takes advantage of the best resource granularity and scalability of the available technologies, and which makes it possible to use either SPARC or x86 systems. Conversely, selecting SPARC makes it possible to leverage the built-in domaining capabilities of modern SPARC processors without incurring additional licensing expenses.

The outcome may be different if the requirements and preferences are reversed. For example, a requirement for live migration limits your choices to logical domains and Solaris kernel zones. That requirement would not limit the features that could be used with logical domains, whereas a preference for hardware isolation would remove kernel zones from the list, and would require the aforementioned configuration restrictions.

After an initial technology choice has been made, you can apply additional features offered with the technology. Continuing with our example, suppose there is a requirement to consolidate and run unmodified Solaris 8 or 9 binaries on recent SPARC servers: this could be accomplished for legacy workloads by running Solaris 8 or Solaris 9 Containers in a Solaris 10 guest domain. Alternatively, existing Solaris 8, Solaris 9, or Solaris 10 systems could be consolidated into guest domains using the logical domains physical to virtual (P2V) tool; alternatively, they could be converted into branded or native zones, which in turn run in a domain. If there are a large number of low-utilization environments to consolidate, then multiple host systems could be consolidated into zones in the same Solaris guest domain, with the fine-grained resource controls available with Solaris Zones then being leveraged.

If the highest priorities include vertical scale, complete hardware redundancy, and complete fault isolation for separate Oracle Solaris instances, then physical domains are the only choice.

If the highest priorities include granular, dynamic, resource allocation for multiple Solaris instances, but you also prefer separate hardware for each VE, then use logical domains.

If the maximum number of virtual instances with the most flexible control of resources is important, then use Solaris Zones as the primary virtualization technology.

If you want to maximize workload density inside physical domains or logical domains, deploy applications in zones within each domain. This approach increases operational flexibility without adding cost, and it permits multiple Solaris virtual environments to reside within a single domain, perhaps using different versions of Solaris. You can also use Solaris Zones to harden the workload’s operating environment against intruders.

When using logical domains with physical domains, you should assign only a single domain configurable unit to a physical domain. This minimizes the number of logical domains that will be affected by a board failure in a physical domain.

If you require a mix of operating systems for developers, VirtualBox is an excellent choice.

Mission-critical workloads should be placed in the lowest-risk environments. Consolidation adds risk, while isolation reduces it, with isolation being a primary factor in determining the level of risk. Hard partitions are strongly suggested, and perhaps required.

High-scale workloads need high-scale hardware. x86 systems may not scale high enough, while SPARC systems are ideal. A high-scale operating system, such as Oracle Solaris, will also play a key role.

Because large workloads tend to have stricter availability requirements, large computer systems typically have better RAS (reliability, availability, and serviceability) characteristics, of which availability is the most important. SPARC M6-32, M7-8, and M7-16 servers excel here, offering full hardware redundancy and excellent system reliability. The ability to service many components while the servers are running is another reason why large workloads run best on the high-end servers.

Consolidated systems benefit from dynamic resource flexibility. The flexibility of physical domains will meet this need. A mid-range or high-end SPARC system with 256 or more CPU cores is the ideal solution in this scenario. The amount of RAM in the system should be slightly more than the sum of RAM needed by all of the workloads. If you are consolidating on a SPARC server that uses a SPARC M7 processor, consider populating all the DIMM slots on the CPU boards. This will enable the DIMM sparing feature, delivering greater uptime in the event a memory DIMM completely fails.

Very small: 0.5 CPU core, 512MB RAM, 4 GB disk: 1× cost units

Small: 1 CPU core, 1 GB RAM, 16 GB disk: 2× cost units

Medium: 4 CPU cores, 2 GB RAM, 64 GB disk: 4× cost units

Large: 8 CPU cores, 8 GB RAM, 128 GB disk: 10× cost units

Very large: 16 CPU cores, 16 GB RAM, 256 GB disk: 25× cost units

These factors will help you choose the virtualization best solution:

The highest density of VEs per server requires the greatest efficiency—that is, the least amount of RAM and disk space per VE, and the least performance overhead per VE. Zones provide the highest density of virtualization solutions and are a good choice in such a case. With applications running, the Oracle Solaris kernel will use hundreds of megabytes of RAM, but each VE will consume only dozens of megabytes, plus the memory needed for the web server applications. A single-CPU server, either x86 or SPARC, can easily handle 50 small customers, or two very large customers.

System memory is a significant portion of the cost of the system. Maximizing customer density means reducing the RAM needs as much as possible and, therefore, reducing acquisition costs. This consideration also leads to the choice of Solaris Zones. We will ignore the memory needed for the web server software, which is independent of virtualization technology. Instead of a minimum of 1 GB RAM being required for each of 50 VEs, which is typical in a hypervisor environment, only approximately 100MB is needed for each zone. Instead of a total of 50 GB RAM, plus memory for the applications, the system will need only 50 GB RAM plus application space.

Costs are reduced through the use of virtual devices, which share physical infrastructure over multiple clients.

Agility is provided by dynamic resource allocation, which makes it possible to add and remove CPU, memory, and I/O devices in a running domain.

Legacy support is enhanced by the ability to run Solaris 10 in guest domains on recent servers that are supported by Solaris 11 and later for physical instances.