Chapter 11

Managing Resource Allocation

The idea that we can take a single physical server and host many VMs has a great deal of value in today's dynamic datacenter environments, but let's face it: There are limits to how many VMs can run on a VMware ESXi host. To make the most of your virtualization platform, you must understand how key resources—memory, processors, disks, and networks—are consumed by the VMs running on the host and how the host itself consumes resources. The method an ESXi host uses to arbitrate access to each of these resources is a bit different. This chapter discusses how an ESXi host allocates these resources and how you can change the way these resources are allocated.

In this chapter, you will learn to

• Manage VM memory allocation
• Manage CPU utilization
• Create and manage resource pools
• Control network and storage I/O utilization
• Utilize flash storage

Reviewing Virtual Machine Resource Allocation

One of the most significant advantages of server virtualization is the ability to allocate resources to a VM based on the machine's actual performance needs. In traditional physical server environments, a server is often provided with more resources than it really needs because it was purchased with a specific budget in mind and the server specifications were maximized for the budget provided. For example, does a Dynamic Host Configuration Protocol (DHCP) server really need dual processors, 32 GB of RAM, and 146 GB mirrored hard drives? In most situations, the DHCP server will most certainly underutilize those resources. In the virtual world, you can create a VM better suited for the role of a DHCP server. For this DHCP server, then, you would assemble a VM with a more suitable 2 GB or 4 GB of RAM (depending on the guest OS), access to a single CPU, and 20 GB to 40 GB of disk space, all of which are provided by the ESXi host on which the VM is running. Then, you can create additional VMs with the resources they need to operate effectively without wasting valuable memory, CPU cycles, and disk storage. Correctly allocating resources based on the anticipated need of the guest OS and the applications inside a VM is the essence of right-sizing your VMs, which we discussed in Chapter 9, “Creating and Managing Virtual Machines.” Right-sizing your VMs allows you to achieve greater efficiency and higher consolidation ratios (more VMs per physical server).

Even when you right-size your VMs, though, as you add more, each VM places additional demand on the ESXi host, and the host's resources are consumed to support the VMs. At a certain point, the host will run out of resources. What does ESXi do when it runs out of resources? How does ESXi handle it when the VMs are asking for more resources than the physical host can actually provide? How can you guarantee that a guest OS and its applications get the resources they need without being starved by other guest OSes and their applications?

Fortunately, VMware vSphere offers a set of controls that are designed to do exactly that: to guarantee access to resources when necessary, to curb or control the use of resources, and to enable prioritized access to resources when available resources are low. Specifically, vSphere offers three controls for controlling or modifying resource allocation: reservations, limits, and shares.

While the behavior of these mechanisms varies based on the resource, the basic idea behind these mechanisms is as follows:

Reservations Reservations act as guarantees of a particular resource. You would use reservations when you want to ensure that, no matter what else is going on, a specific VM is absolutely assured to have access to a particular amount of a given resource.

Limits Limits restrict the amount of a given resource that a VM can use. VMs already have some limits simply by how they are constructed—for example, a VM configured to have a single virtual CPU (vCPU) is limited to using only that single vCPU. The Limit feature within vSphere grants you even greater granularity over how resources are utilized. Depending on the resource to which the limit is being applied, the specific behavior of ESXi will change. This is discussed in detail later in this chapter under each resource's specific section.

Shares Shares establish priority. When an ESXi host comes under contention and must decide which VM gets access to which resources, shares are used to determine priority. VMs with higher shares assigned will have higher priority, and therefore greater access, to the ESXi host's resources.

Figure 11.1 shows these three mechanisms displayed in the properties of a VM.

Throughout the rest of this chapter, we discuss how one or more of these three mechanisms—reservations, limits, and shares—are applied to control or modify resource allocation across all four major resources in a vSphere environment: memory, CPU, storage, and network.

FIGURE 11.1 Reservations, limits, and shares offer more fine-grained control over resource allocation.

The first VM resource we'll examine is memory. In many instances, memory is the first resource to come under constraints, so taking a look at memory first is warranted.

Working with Virtual Machine Memory

Let's start with a discussion of how memory is allocated to a VM. Later we'll discuss how you as an administrator can use reservations, shares, and limits to help control or modify how VMs consume memory.

When you create a new VM through the Web Client, you are asked how much memory the VM should have. The Web Client suggests a default value based on the selected guest OS (the selected guest OS in this case is Windows Server 2008 R2), as shown in Figure 11.2.

The amount of memory you allocate on this screen is the amount the guest OS will see—in this example, it is 4,096 MB. This is the same as when you build a physical system and put a set of four 1,024 MB memory sticks into the system board. If you install Windows Server 2008 R2 in this VM, Windows will report 4,096 MB of RAM installed. Ultimately, this is the amount of memory the VM “thinks” it has and the maximum amount of memory the guest OS will ever be able to access. Like a physical system with four 1,024 MB DIMMs installed, this VM will never be able to use more than 4,096 MB of physical RAM.

Let's assume you have an ESXi host with 16 GB of physical RAM available to run VMs (in other words, the hypervisor is using some RAM and there's 16 GB left over for the VMs). In the case of our new VM, it will comfortably run, leaving approximately 12 GB for other VMs (there is some additional overhead that we discuss later, but for now let's assume that the 12 GB is available to other VMs).

FIGURE 11.2 The memory configuration settings for a VM indicate the amount of RAM the VM “thinks” it has.

What happens when you run three more VMs, each configured with 4 GB of RAM? Each of the additional VMs will request 4 GB of RAM from the ESXi host. At this point, four VMs will be accessing the physical memory, and you will have allocated all 16 GB of memory to the VMs. ESXi has now run out of a critical resource (memory).

What happens when you launch a fifth VM? Will it run? The short answer is yes, and some of the key technologies that enable administrators to overcommit memory—that is, to allocate more memory to VMs than is actually installed in the VMware ESXi host—are quite advanced. Because these technologies are integral to understanding how memory allocation works with VMware ESXi, let's take a look at these technologies and how they work.

Understanding ESXi Advanced Memory Technologies

VMware ESXi supports a number of different technologies for advanced memory management. As a result of these advanced memory management technologies, at the time of this writing, VMware ESXi is the only commercially available hypervisor on the market capable of performing memory overcommitment in a manner that is guest OS agnostic.

VMware ESXi employs five different memory-management technologies to make sure the physical server's RAM is utilized as efficiently as possible: idle memory tax, transparent page sharing, ballooning, memory compression, and swapping.

While it doesn't cover everything, for anyone interested in more in-depth and detailed information on some of these memory-management technologies, we strongly recommend reading Memory Resource Management in VMware ESX Server by Carl A. Waldspurger, available online at the following location:

http://www.waldspurger.org/carl/papers/esx-mem-osdi02.pdf

IDLE MEMORY TAX

Before VMware ESXi actively starts making changes to relieve memory pressure, it ensures that VMs do not actively horde memory by “charging” more for the idle memory. Up to 75 percent of the memory allocated to each VM can be borrowed to service another VM by IMT. This setting is configurable on a per-VM basis within the Advanced VM settings (see Chapter 9), although in most cases this is not necessary and not recommended unless there is a specific requirement. Inside the guest OS, VMware Tools will use its balloon driver to understand which memory blocks are allocated but idle and, therefore, available to be used elsewhere. The balloon driver is also used in a more active fashion, which we will explain shortly.

TRANSPARENT PAGE SHARING

The next memory-management technology ESXi uses is transparent page sharing (TPS), in which identical memory pages are shared among VMs to reduce the total number of memory pages consumed. The hypervisor computes hashes of the contents of memory pages to identify pages that contain identical memory. If it finds a hash match, it compares the matching memory pages to exclude a false positive. Once the pages are confirmed to be identical, the hypervisor will transparently remap the memory pages of the VMs so they are sharing the same physical memory page. This reduces overall host memory consumption. Advanced parameters are available to fine-tune the behavior of the page-sharing mechanisms.

Normally, ESXi works on 4 KB memory pages and will use transparent page sharing on all memory pages. However, when the hypervisor is taking advantage of hardware offloads available in the CPUs—such as Intel Extended Page Tables (EPT) Hardware Assist or AMD Rapid Virtualization Indexing (RVI) Hardware Assist—then the hypervisor uses 2 MB memory pages, also known as large pages. In these cases, ESXi will not share these large pages, but it will compute hashes for the 4 KB pages inside the large pages. In the event that the hypervisor needs to invoke swapping, the large pages will be broken into small pages, and having the hashes already computed allows the hypervisor to invoke page sharing before they are swapped out.

BALLOONING

We mentioned previously that ESXi's memory-management technologies are guest OS agnostic, meaning that the guest OS selection doesn't matter. This is true; any supported guest OS can take advantage of all of ESXi's memory-management functionality. However, these technologies are not necessarily guest OS independent, meaning that they operate without interaction from the guest OS. While transparent page sharing operates independently of the guest OS, ballooning does not.

Ballooning involves the use of a driver—referred to as the balloon driver—installed into the guest OS. This driver is part of VMware Tools and gets installed when VMware Tools is installed. Once installed into the guest OS, the balloon driver can respond to commands from the hypervisor to reclaim memory from that particular guest OS. The balloon driver does this by requesting memory from the guest OS—a process calling inflating—and then passing that memory back to the hypervisor for use by other VMs.

Because the guest OS can give up pages it is no longer using when the balloon driver requests memory, it's possible for the hypervisor to reclaim memory without any performance impact on the applications running inside that guest OS. If the guest OS is already under memory pressure—meaning the amount of memory configured for that VM is insufficient for the guest OS and its applications—it's very likely that inflating the balloon driver will invoke guest OS paging (or swapping), which will impair performance.

MEMORY COMPRESSION

From vSphere 4.1, VMware added another memory-management technology to the mix: memory compression. When an ESXi host gets to the point that hypervisor swapping is necessary, the VMkernel will attempt to compress memory pages and keep them in RAM in a compressed memory cache. Pages that can be successfully compressed by at least 50 percent are put into the compressed memory cache instead of being written to disk and can then be recovered much more quickly if the guest OS needs that memory page. Memory compression can dramatically reduce the number of pages that must be swapped to disk and thus can dramatically improve the performance of an ESXi host that is under strong memory pressure. There is a configurable amount of VM memory used for the compression cache (by default 10 percent), but this starts at zero and grows as needed when VM memory starts to be swapped out. Compression is invoked only when the ESXi host reaches the point that VMkernel swapping is needed.

SWAPPING

There are two forms of swapping involved when you examine how memory is managed with VMware ESXi. There is guest OS swapping, in which the guest OS inside the VM swaps pages out to its virtual disk according to its own memory-management algorithms. This is generally due to memory requirements that are higher than available memory. In a virtualized environment, this would translate into a VM being configured with less memory than the guest OS and its applications require, such as trying to run Windows Server 2008 R2 in only 1 GB of RAM. Guest OS swapping falls strictly under the control of the guest OS and is not controlled by the hypervisor.

The other type of swapping involved is hypervisor swapping. In the event that none of the previously described technologies trim guest OS memory usage enough, the ESXi host will be forced to use hypervisor swapping. Hypervisor swapping means that ESXi is going to swap memory pages out to disk in order to reclaim memory that is needed elsewhere. ESXi's swapping takes place without any regard to whether the pages are being actively used by the guest OS. As a result, and due to the fact that disk response times are thousands of times slower than memory response times, guest OS performance is severely impacted if hypervisor swapping is invoked. It is for this reason that ESXi won't invoke swapping unless it is absolutely necessary, as a last resort after all other memory management techniques have been tried.

The key thing to remember about hypervisor swapping is that you want to avoid it if at all possible; there is a significant and noticeable impact to performance. Even swapping to SSD (Solid State Drive) is considerably slower than directly accessing RAM.

Although these advanced memory-management technologies allow ESXi to allocate more memory to VMs than there is actual RAM in the physical server, they do not help guarantee memory or prioritize access to memory. Even with these advanced memory-management technologies, at some point it becomes necessary to exercise some control over how the VMs access and use the memory allocated to them. This is where a VMware vSphere administrator can use reservations, limits, and shares—the three mechanisms described previously—to modify or control how resources are allocated. Next, we'll describe how these mechanisms are used to control memory allocation.

Controlling Memory Allocation

Like all physical resources, memory is a finite resource. The advanced memory-management technologies in ESXi help with the efficient use of this finite resource by making it go farther than it normally would go. For finer-grained control over how ESXi allocates memory, though, administrators must turn to the use of the three mechanisms listed previously: reservations, shares, and limits. Figure 11.3 shows these three settings in the Virtual Machine Properties dialog box for a VM.

The steps for editing a Reservation, Limit, or Shares value for either memory or CPU are the same. Storage I/O and network I/O are handled a bit differently, so we'll discuss those in the appropriate sections later in this chapter. Storage I/O is covered in the section “Controlling Storage I/O Utilization,” and network I/O is discussed in the section “Regulating Network I/O Utilization.”

Perform the following steps to edit a VM's memory or CPU Reservation, Limit, or Shares value:

1. Use the Web Client to connect to a vCenter Server instance.
2. Navigate to either the Hosts And Clusters or VMs And Templates view, then drill down through the inventory to find the VM to be edited.
3. Select the VM, and select the Edit Settings option in the content area.
4. On the Virtual Hardware tab, click the triangle next to the CPU or Memory item.
5. Adjust the Shares, Reservation, and Limit values as desired.

   FIGURE 11.3 vSphere supports the use of reservations, shares, and limits for controlling memory allocation.

   

Now that you've seen how to adjust the Reservation, Limit, and Shares values, we'll take a detailed look at the specific behaviors of how these mechanisms apply to memory usage and allocation.

USING MEMORY RESERVATIONS

The memory reservation is an optional setting for each VM. You can see in Figure 11.3 that the default memory reservation is 0 MB (the equivalent of no memory reservation at all). You can adjust this value, but what exactly does this value do? What impact does adjusting the memory reservation have?

The memory reservation amount specified in the VM settings is the amount of actual, real physical memory that the ESXi host must provide to this VM for the VM to power on. A VM with a memory reservation is guaranteed the amount of RAM configured in its Reservation setting. As mentioned, the default is 0 MB, or no reservation. In the previous example, configuring the VM with 4 GB of RAM and the default reservation of 0 MB means the ESXi host is not required to provide the VM with any physical memory. If the ESXi host is not required to provide actual RAM to the VM, then where will the VM get its memory? In the absence of a reservation, the VMkernel has the option to provide VM memory from VMkernel swap.

VMkernel swap is the hypervisor swapping mechanism referred to previously when discussing the various memory-management techniques that ESXi employs. VMkernel swap is implemented as a file with a .vswp filename extension that is created when a VM is powered on. These per-VM swapfiles created by the VMkernel reside, by default, in the same datastore location as the VM's configuration file and virtual disk files (although you do have the option of relocating the VMkernel swap, which we'll explain later in the section “Configuring Swap to Host Cache”). In the absence of a memory reservation—the default configuration—this file will be equal in size to the amount of RAM configured for the VM. Thus, a VM configured for 4 GB of RAM will have a VMkernel swapfile that is also 4 GB in size and stored, by default, in the same location as the VM's configuration and virtual disk files.

In theory, this means a VM could get its memory allocation entirely from VMkernel swap—or disk—resulting in VM performance degradation because disk access time is several orders of magnitude slower than RAM access time.

Just because a VM without a reservation could potentially get all its memory from VMkernel swap, does this mean that a VM will actually get all of its memory from swap when ESXi host RAM is available? No. ESXi attempts to provide each VM with all the memory it requests, up to the maximum amount configured for that VM. Obviously, a VM configured with only 4,096 MB of RAM cannot request more than 4,096 MB of RAM. However, when an ESXi host doesn't have enough RAM available to satisfy the memory needs of the VMs it is hosting and when technologies such as transparent page sharing, the balloon driver, and memory compression aren't enough, the VMkernel is forced to page some of each VM's memory out to the individual VM's VMkernel swapfile.

Is there a way you can control how much of an individual VM's memory allocation can be provided by swap and how much must be provided by real physical RAM? Yes. This is where a memory reservation comes into play. Recall that we said a memory reservation specifies the amount of real, physical RAM that the ESXi host must provide the VM. By default, a VM has a memory reservation of 0 MB, which means that ESXi is not required to provide any real, physical RAM. This means potentially all of the VM's memory could be paged out to the VMkernel swapfile if necessary.

Let's look at what happens if you decide to set a memory reservation of 1,024 MB for this VM, shown in Figure 11.4. How does this change the way this VM gets memory?

In this example, when this VM is started, the ESXi host must provide at least 1,024 MB of physical RAM to support this VM's memory allocation. In fact, 1,024 MB of RAM is guaranteed for this VM. The host can provide the remaining 3,072 MB of RAM from either physical RAM or VMkernel swap, as shown in Figure 11.5. In this case, because some of the VM's RAM is guaranteed to come from physical RAM, ESXi reduces the size of the VMkernel swapfile by the amount of the reservation. Therefore, the VMkernel swapfile is reduced in size by 1,024 MB. This behavior is consistent with what we've shown you so far: With a reservation of 0 MB, the VMkernel swapfile is the same size as the amount of configured memory. As the reservation increases, the size of the VMkernel swapfile decreases in size correspondingly.

FIGURE 11.4 This memory reservation guarantees 1,024 MB of RAM for the VM.

FIGURE 11.5 The memory reservation reduces the potential need for VMkernel swap space by the size of the reservation.

This behavior ensures that a VM has at least some high-speed memory available to it if the ESXi host is running more VMs than it has actual RAM to support, but there's also a downside. If you assume that each of the VMs you start on this host has a 1,024 MB reservation and you have 8 GB of available RAM in the host to run VMs, then you will be able to launch only eight VMs concurrently (8 × 1,024 MB = 8,192 MB). On a more positive note, if each VM is configured with an initial RAM allocation of 4,096 MB, then you're now running VMs that would need 32 GB of RAM on a host with only 8 GB. ESXi uses the technologies described previously—transparent page sharing, the balloon driver, memory compression, and finally, VMkernel swap—to manage the fact that you, as the administrator, have allocated more RAM than is physically installed in the server.

There's one other side effect from using memory reservations that you must also understand. We mentioned previously that using a memory reservation guarantees physical RAM for the VM. This is true, but only as the guest OS in the VM requests memory. If you have a VM with a 1,024 MB reservation configured, then the ESXi host will allocate RAM to the VM on an as-needed basis, and the first 1,024 MB of RAM allocated to that VM is part of the reservation. RAM is allocated on demand; the presence of a reservation doesn't change that behavior. Once allocated, though, because this RAM is part of the memory reservation, it's locked to this VM—it won't be reclaimed via the balloon driver, and it won't be swapped out to disk or compressed. In a way, that's good; it underscores the fact that this memory is guaranteed to this VM. In a way, it's also bad, though, because the reserved memory, once allocated to a VM, can't be reclaimed for use by other VMs or for use by the hypervisor itself.

Like all the mechanisms described in this chapter, this means that you'll want to use memory reservations carefully and with a full understanding of the impact on the ESXi host's behavior and operation.

USING MEMORY LIMITS

If you refer back to Figure 11.3, you will also see a setting for a memory limit. By default, all new VMs are created without a limit, which means that the initial RAM you assigned to it during creation is its effective limit. So, what exactly is the purpose of the Limit setting? It sets the actual limit on how much physical RAM may be utilized by that VM.

To see this behavior in action, let's now change the limit on this VM from the default setting of Unlimited to 2,048 MB.

So, what is the effective result of this configuration? Here's how it breaks down:

• The VM is configured with 4,096 MB of RAM, so the guest OS running inside that VM believes that it has 4,096 MB of RAM available to use.
• The VM has a reservation of 1,024 MB of RAM, which means that the ESXi host must allocate 1,024 MB of physical RAM to the VM. This RAM is guaranteed to this VM.
• Assuming the ESXi host has enough physical RAM installed and available, the hypervisor will allocate memory to the VM as needed up to 2,048 MB (the limit). Upon reaching 2,048 MB, the balloon driver kicks in to prevent the guest OS from using any more memory beyond 2,048 MB. When the guest OS's memory demands drop below 2,048 MB, the balloon driver deflates and returns memory to the guest. The effective result of this behavior is that the memory the guest OS uses remains below 2,048 MB (the limit).
• The 1,024 MB “gap” between the reservation and the limit could be supplied by either physical RAM or VMkernel swap space. ESXi will allocate physical RAM if it is available.

The key problem with memory limits is that they are enforced without any guest OS awareness. If you have a VM configured for 4 GB of RAM, the guest OS inside that VM is going to think it has 4 GB of RAM with which to work, and it will behave accordingly. If you then place a 2 GB limit on that VM, the VMkernel will enforce that the VM only uses 2 GB of RAM. Fine—but it will do so without the knowledge or cooperation of the guest OS inside that VM. The guest OS will continue to behave as if it has 4 GB of RAM, completely unaware of the limit that has been placed on it by the hypervisor. If the working set size of the guest OS and its applications exceeds the memory limit, setting a limit will degrade the performance of the VM because the guest OS will constantly be forced to swap pages to disk (guest OS swapping, not hypervisor swapping).

In general, then, you should consider memory limits a temporary stop-gap measure when you need to reduce physical memory usage on an ESXi host and a negative impact to performance is acceptable. You wouldn't, generally speaking, want to overprovision a VM with RAM and constrain memory usage with a limit on a long-term basis. In that scenario, the VM will typically perform very poorly and would actually perform better with less RAM configured and no limit.

Working together, an initial allocation of memory, a memory reservation, and a memory limit can be powerful tools in efficiently managing the memory available on an ESXi host. But there is still one more tool to examine, and that's memory shares.

USING MEMORY SHARES

In Figure 11.3, there is a third setting called Shares that we have not yet discussed. The two mechanisms described to you already, memory reservations and memory limits, help provide finer-grained controls over how ESXi should or should not allocate memory to a VM. These mechanisms are always in effect; that is, a Limit setting is enforced even if the ESXi host has plenty of physical RAM available for the VM to use.

Memory shares are very different. The share system in VMware is a proportional share system that allows administrators to assign resource priority to VMs, but shares are only used when the ESXi host is experiencing physical RAM contention. In other words, the VMs on an ESXi host are requesting more memory than the host can provide. If an ESXi host has plenty of memory available, shares will not play a role. However, when memory is scarce and ESXi must decide which VM should be given access to memory, shares can establish a priority setting for a VM requesting memory that is greater than the VM's reservation but less than its limit. (Recall that memory under the reservation is guaranteed to the VM, and memory over the limit would not be allocated. Shares, therefore, affect only the allocation of memory between the reservation and the limit.) In other words, if two VMs want more memory than their reservation limit and the ESXi host can't satisfy both of them using RAM, then you can set share values on each VM so that one gets higher-priority access to the RAM in the ESXi host than the other.

Some would say that you should just increase the reservation for that VM. Although that might be a valid technique, it might limit the total number of VMs that a host can run, as indicated previously in this chapter. Increasing the configured amount of RAM also requires a reboot of the VM to become effective (unless you are running a guest OS that supports hot-add of memory and that feature has been enabled for the VM, as described in Chapter 9), but shares can be dynamically adjusted while the VM remains powered on.

One key part we must repeat is that shares come into play only when the ESXi host cannot satisfy the requests for memory. If the ESXi host has enough free memory to satisfy the requests from the VMs for memory, then it doesn't need to prioritize those requests. It has enough to go around. It's only when the ESXi host doesn't have enough to go around that decisions have to be made on how that resource should be allocated.

For the sake of this discussion, let's assume you have two VMs (VM1 and VM2), each with a 1,024 MB reservation and a configured maximum of 4,096 MB, and both are running on an ESXi host with less than 2 GB of RAM available to the VMs. If the two VMs in question have an equal number of shares (let's assume it's 1,000 each; we'll show you actual values shortly), then as each VM requests memory above its reservation value, each VM will receive an equal quantity of RAM from the ESXi host. Furthermore, because the host cannot supply all of the RAM to both VMs, each VM will swap equally to disk (VMkernel swapfile). This is assuming, of course, that ESXi cannot reclaim memory from other running VMs using the balloon driver or other memory-management technologies described previously. If you change VM1's Shares setting to 2,000, then VM1 now has twice the shares VM2 has assigned to it. This also means that when VM1 and VM2 are requesting the RAM above their respective Reservation values, VM1 gets two RAM pages for every one RAM page that VM2 gets. If VM1 has more shares, VM1 has a higher-priority access to available memory in the host. Because VM1 has 2,000 out of 3,000 shares allocated, it will get 67 percent; VM2 has 1,000 out of 3,000 shares allocated and therefore gets only 33 percent. This creates the two-to-one behavior described previously. Each VM is allocated RAM pages based on the proportion of the total number of shares allocated across all VMs. Figure 11.6 illustrates this behavior.

FIGURE 11.6 Shares establish relative priority based on the number of shares assigned out of the total shares allocated.

Even if you don't specifically assign shares to a VM, VMware vSphere automatically assigns shares to a VM when it is created. You can see the default Shares value back in Figure 11.3; it is equal to 10 times the configured memory value when the memory allocation is expressed in terms of MB—more accurately, by default 10 shares are granted to every MB assigned to a virtual machine. The VM shown in Figure 11.3 had 4,096 MB of RAM configured; therefore, its default memory Shares value was 40,960. This default allocation ensures that each VM is granted priority to memory on a measure that is directly proportional to the amount of memory configured for it.

It gets more difficult to predict the actual memory utilization and the amount of access each VM gets as more VMs run on the same ESXi host. Later in this chapter, in the section titled “Using Resource Pools,” we'll discuss more sophisticated methods of assigning memory limits, reservations, and shares to a group of VMs using resource pools.

We've talked about how VMware ESXi uses some advanced memory management technologies, but there is another aspect of virtualization that you must also consider: overhead. In the next section, we'll provide some information on the memory overhead figures when using ESXi.

EXAMINING MEMORY OVERHEAD

As they say, nothing in this world is free, and in the case of memory on an ESXi host, there is a cost. That cost is memory overhead. Several basic processes on an ESXi host will consume host memory. The VMkernel itself, various daemons (services) running on the ESXi host, and each VM that is running will cause the VMkernel to allocate some memory to host the VM above the initial amount that you assign to it. The amount of RAM allocated to power on each VM depends on the virtual CPU and memory configuration of each VM. VMware has improved the overhead requirements significantly over the last few versions of vSphere. To give you an indication of what they are for version 5.5, see Table 11.1. The values have been rounded to the nearest whole number.

TABLE 11.1: Virtual machine memory overhead

As you go about planning the allocation of memory to your VMs, be sure to keep these memory overhead figures in mind. You will want to include these overhead values in your calculations of how memory will be assigned and used, especially if you plan on using a number of VMs with large amounts of memory and a large number of virtual CPUs. As you can see in Table 11.1, the memory overhead in that situation could become fairly substantial.

SUMMARIZING HOW RESERVATIONS, LIMITS, AND SHARES WORK WITH MEMORY

Because the specific behavior of reservations, shares, and limits is slightly different for each resource, here's a quick review of their behavior when they are used for controlling memory allocation:

• Reservations guarantee memory for a particular VM. Memory isn't allocated until requested by the VM, but the host must have enough free memory to satisfy the entire reservation before the VM can be powered on. Therefore—and this makes sense if you think about it—you cannot reserve more memory than the host physically has installed. Once allocated to a VM, reserved memory is not swapped, nor is it reclaimed by the ESXi host. It is locked to that VM.
• Limits enforce an upper ceiling on the usage of memory. Limits are enforced using the balloon driver (if VMware Tools is installed) and—depending on the VM's working set size—could have a dramatic negative impact on performance. As the VM approaches the limit (a limit of which the guest OS is not aware), the balloon driver will inflate to keep VM memory usage under the limit. This will cause the guest OS to swap out to disk, which will typically degrade performance noticeably.
• Shares apply only during periods of host RAM contention and serve to establish prioritized access to host RAM. VMs are granted priority based on percentage of shares allocated versus total shares granted. During periods when the host is not experiencing memory contention, shares do not apply and will not affect memory allocation or usage.

We'll provide a similar summary of the behavior of reservations, limits, and shares when they are used to control CPU usage, which is the topic of the next sections.

Managing Virtual Machine CPU Utilization

When creating a new VM using the Web Client, you have two options when configuring the CPU. First, choose how many virtual CPUs you want in the VM and then decide how many cores should be allocated to each socket. Your selection determines how many sockets the VM has. These CPU settings effectively let the guest OS in the VM utilize between 1 and 64 virtual CPUs on the host system, depending upon the guest OS and the vSphere license.

When the VMware engineers designed the virtualization platform, they started with a real system board and modeled the VM after it—in this case it was based on the Intel 440BX chipset. The VM could emulate the PCI bus, which could be mapped to input/output devices through a standard interface, but how could a VM emulate a CPU? The answer was “no emulation.” Think about a virtual system board that has a “hole” where the CPU socket goes—and the guest OS simply looks through the hole and sees one of the cores in the host server. This allowed the VMware engineers to avoid writing CPU emulation software that would need to change each time the CPU vendors introduced new instruction sets. If there was an emulation layer, it would also add significant overhead, which would limit the performance of the virtualization platform by adding more computational overhead.

So, how many CPUs should a VM have? A VM that replaces a physical DHCP server that runs at less than 10 percent CPU utilization at its busiest point in the day surely does not need more than one virtual CPU. As a matter of fact, if you give this VM two virtual CPUs (vCPUs), then you might limit the scalability of the entire host. Here's why.

The VMkernel simultaneously schedules CPU cycles for multi-vCPU VMs. This means that when a dual-vCPU VM places a request for CPU cycles, the request goes into a queue for the host to process, and the host has to wait until there are at least two cores or hyperthreads (if hyperthreading is enabled) with concurrent idle cycles to schedule that VM. A relaxed co-scheduling algorithm provides a bit of flexibility in allowing the cores to be scheduled on a slightly skewed basis, but even so, it can be more difficult for the hypervisor to find open time slots on at least two cores. This occurs even if the VM needs only a few clock cycles to do some menial task that could be done with a single processor. Here's an example: Have you ever been stuck behind a truck with a wide load that takes up more than one lane? Normally traffic could flow around this slow-moving vehicle, but now traffic is held up because two lanes are occupied.

On the other hand, if a VM needs two vCPUs because of the load it will be processing on a constant basis, then it makes sense to assign two vCPUs to that VM—but only if the host has four or more CPU cores total. If your ESX host is an older-generation dual-processor single-core system, then assigning a VM two vCPUs will mean that the VM owns all of the CPU processing power on that host every time it gets CPU cycles. You will find that the overall performance of the host and any other VMs will be less than stellar. Of course, in today's market of multicore CPUs, this particular consideration is less significant than it was in previous hardware generations, but it is something to keep in mind.

Default CPU Allocation

Like the memory settings discussed previously, the Shares, Reservation, and Limit settings can be configured for CPU capacity as well.

When a new VM is created with a single vCPU, the total maximum CPU cycles for that VM equals the clock speed of the host system's core. In other words, if you create a new VM, it can see through the “hole in the system board,” and it sees whatever the core is in terms of clock cycles per second—an ESXi host with 3 GHz CPUs in it will allow the VM to see one 3 GHz core.

Figure 11.7 shows the default settings for CPU Reservation, Limits, and Shares.

FIGURE 11.7 By default, vSphere provides no CPU reservation, no CPU limit, and 1,000 CPU shares.

Setting CPU Affinity

In addition to shares, reservations, and limits, vSphere offers a fourth option for managing CPU usage: CPU affinity. CPU affinity allows an administrator to statically associate a VM to a specific physical CPU core. CPU affinity is generally not recommended; it has a list of rather significant drawbacks:

• CPU affinity breaks vMotion.
• The hypervisor is unable to load-balance the VM across all the processing cores in the server. This prevents the hypervisor's scheduling engine from making the most efficient use of the host's resources.
• Because vMotion is broken, you cannot use CPU affinities in a cluster where vSphere DRS isn't set to Manual operation.

Because of these limitations, most organizations don't use CPU affinity. However, if, for example, you find that you need to use CPU affinity to adhere to licensing requirements, you can configure your VM to use it.

Perform these steps to configure CPU affinity:

1. If it is not already running, launch the Web Client and connect to a vCenter Server instance or a stand-alone ESXi host.
2. Navigate to either the Hosts And Clusters or VMs And Templates view.
3. Right-click the VM for which you'd like to configure CPU affinity and select Edit Settings.
4. On the Virtual Hardware tab, click the triangle next to CPU.
5. In the Scheduling Affinity section, supply a list of the CPU cores this VM is allowed to access.

   For example, if you wanted the VM to run on cores 1 through 4, you could type 1–4.

6. Click OK to save the changes.

Rather than trying to use CPU affinity to guarantee CPU resources, you're far better off using reservations.

Using CPU Reservations

As you saw in Figure 11.7, the default CPU reservation for a new VM is 0 MHz (no reservation). Recall that a reservation is a resource guarantee. Therefore, by default, a VM is not guaranteed any CPU activity by the VMkernel. This means that when the VM has work to be done, it places its CPU request into the CPU queue so that the VMkernel can handle the request in sequence along with all of the other VMs' requests. On a lightly loaded ESXi host, it's unlikely the VM will wait long for CPU time; however, on a heavily loaded host, the time this VM might have to wait could be significant.

If you were to set a 1,024 MHz reservation, as shown in Figure 11.8, this would effectively make that amount of CPU available instantly to this VM if there is a need for CPU cycles.

FIGURE 11.8 A VM configured with a 1,024 MHz reservation for CPU activity is guaranteed that amount of CPU capacity.

Using a CPU reservation has one notable impact on the behavior of the ESXi host, and in this regard CPU reservations and memory reservations behave identically. The ESXi host must be able to satisfy the reservation by providing enough resources to meet the reservations. If each VM you create has a 1,024 MHz reservation and your host has 12,000 MHz of CPU capacity, you can power on no more than 11 VMs (1,024 MHz × 11 = 11,264 MHz), even if all of them are idle. Note that we said “power on” and not “create”—resources are allocated only when a VM is powered on, not created.

While a CPU reservation behaves like a memory reservation in this regard, a CPU reservation is very different than a memory reservation when it comes to “sharing” reserved CPU cycles. Recall from the previous section that reserved memory, once allocated to the VM, is never reclaimed, paged out to disk, or shared in any way. The same is not true of CPU reservations. Suppose you have a VM, creatively named VM1, that has a CPU reservation of 1,024 MHz. If VM1 is idle and not using its reserved CPU cycles, those cycles can be given to VM2. If VM1 suddenly needs cycles, VM2 doesn't get them anymore, and they are assigned to VM1.

So using a Reservation setting on CPU is similar to using a Reservation setting with memory, but it is also very different. You saw previously that using a Limit setting with memory had some significant drawbacks; what about CPU limits?

Using CPU Limits

In addition to a CPU reservation, you can set an option to place a limit on the amount of CPU allocated. This effectively limits the VM's ability to utilize a maximum number of clock cycles per second, regardless of what the host has available. Keep in mind that a VM with one single-core virtual CPU hosted on a 3 GHz, quad-processor ESXi host will see only a single 3 GHz core as its maximum, but as administrator you could alter the limit to prevent the VM from using the maximum core speed. For instance, you could set a 500 MHz limit on that DHCP server so that when it re-indexes the DHCP database, it won't try to take all of the 3 GHz on the processor that it can see. The CPU limit lets you throttle the VM with less processing power than is available on a core on the physical host. Not every VM needs to have access to the entire processing capability of the physical processor core.

The key drawback to using a CPU Limit setting is its performance impact on the guest OS and the applications running in that VM. The Limit setting is a true limit; the VM won't be scheduled to run on a physical CPU core more than the limit specifies, even if there are plenty of CPU cycles available. It's important, therefore, to understand the CPU processing needs of your VMs before arbitrarily setting CPU limits or you could find yourself significantly impacting performance.

Using CPU Shares

VMware vSphere's shares model, which lets you prioritize access to resources when resource contention occurs, behaves similarly for both memory and CPU. The shares for CPU will determine how much CPU is provided to a VM in the face of contention with other VMs needing CPU activity. All VMs, by default, start with an equal number of shares, which means that if VMs compete for CPU cycles on an ESXi host, each one gets serviced with equal priority. Keep in mind that this share value affects only those CPU cycles that are greater than the reservation set for the VM, and the share value applies only when the ESXi host has more requests for CPU cycles than it has CPU cycles to allocate. In other words, the VM is granted access to its reservation cycles regardless of what else is happening on the host, but if the VM needs more—and there's competition—then the share values come into play. If there is no CPU contention on the host and it has enough CPU cycles to go around, the CPU Shares value won't affect CPU allocation.

Several conditions have to be met for shares to even be considered for allocating CPU cycles. The best way to determine this is to consider several scenarios. For the scenarios we'll cover, assume the following details about the environment:

• The ESXi host includes dual, single-core, 3 GHz CPUs.
• The ESXi host has one or more VMs.

Scenario 1 The ESXi host has a single VM running. The shares are set at the defaults for any running VMs. Will the Shares value have any effect in this scenario? No. There's no competition between VMs for CPU time.

Scenario 2 The ESXi host has two idle VMs running. The shares are set at the defaults for the running VMs. Will the Shares values have any effect in this scenario? No. There's no competition between VMs for CPU time because both are idle.

Scenario 3 The ESXi host has two equally busy VMs running (both requesting maximum CPU capacity). The shares are set at the defaults for the running VMs. Will the Shares values have any effect in this scenario? No. Again, there's no competition between VMs for CPU time, this time because each VM is serviced by a different core in the host.

Scenario 4 To force contention, both VMs are configured to use the same CPU by setting the CPU affinity. The ESXi host has two equally busy VMs running (both requesting maximum CPU capacity). This ensures contention between the VMs. The shares are set at the defaults for the running VMs. Will the Shares values have any effect in this scenario? Yes! But in this case, because all VMs have equal Shares values, each VM has equal access to the host's CPU queue, so you do see any effects from the Shares values.

Scenario 5 The ESXi host has two equally busy VMs running (both requesting maximum CPU capacity with CPU affinity set to the same core). The shares are set as follows: VM1 is set to 2,000 CPU shares, and VM2 is set to the default 1,000 CPU shares. Will the Shares values have any effect in this scenario? Yes. In this case, VM1 has double the number of shares that VM2 has. This means that for every clock cycle that VM2 is assigned by the host, VM1 is assigned two clock cycles. Stated another way, out of every three clock cycles assigned to VMs by the ESXi host, two are assigned to VM1, and one is assigned to VM2. The diagram earlier in Figure 11.6 helps graphically reinforce how shares are allocated based on percentage of the total number of shares assigned to all VMs.

Scenario 6 The ESXi host has three equally busy VMs running (each requesting maximum CPU capabilities with CPU affinity set to the same core). The shares are set as follows: VM1 is set to 2,000 CPU shares, and VM2 and VM3 are set to the default 1,000 CPU shares. Will the Shares values have any effect in this scenario? Yes. In this case, VM1 has double the number of shares that VM2 and VM3 have assigned. This means that for every two clock cycles that VM1 is assigned by the host, VM2 and VM3 are each assigned a single clock cycle. Stated another way, out of every four clock cycles assigned to VMs by the ESXi host, two cycles are assigned to VM1, one is assigned to VM2, and one is assigned to VM3. You can see that this has effectively watered down VM1's CPU capabilities.

Scenario 7 The ESXi host has three VMs running. VM1 is idle while VM2 and VM3 are equally busy (each requesting maximum CPU capabilities, and all three VMs are set with the same CPU affinity). The shares are set as follows: VM1 is set to 2,000 CPU shares, and VM2 and VM3 are set to the default 1,000 CPU shares. Will the Shares values have any effect in this scenario? Yes. But in this case VM1 is idle, which means it isn't requesting any CPU cycles. This means that VM1's Shares value is not considered when apportioning the host CPU to the active VMs. In this case, VM2 and VM3 would equally share the host CPU cycles because their shares are set to an equal value.

Given these scenarios, if you were to extrapolate to an eight-core host with 30 or so VMs, it would be difficult to set Shares values on a VM-by-VM basis and to predict how the system will respond. The question then becomes, “Are shares a useful tool?” The answer is yes, but in large enterprise environments, you need to examine resource pools and the ability to set share parameters along with reservations and limits on collections of VMs. We'll introduce resource pools in the section “Using Resource Pools.” First, though, we'll summarize the behavior of reservations, limits, and shares when used to control CPU allocation and usage.

Summarizing How Reservations, Limits, and Shares Work with CPUs

The following list includes some key behaviors and facts surrounding the use of reservations, limits, and shares, when applied to controlling or modifying CPU usage:

• Reservations set on CPU cycles provide guaranteed processing power for VMs. Unlike memory, reserved CPU cycles can and will be used by ESXi to service other requests when needed. As with memory, the ESXi host must have enough real, physical CPU capacity to satisfy a reservation in order to power on a VM. Therefore, you cannot reserve more CPU cycles than the host is actually capable of delivering.
• Limits on CPU usage simply prevent a VM from gaining access to additional CPU cycles even if CPU cycles are available to use. Even if the host has plenty of CPU processing power available to use, a VM with a CPU limit will not be permitted to use more CPU cycles than specified in the limit. Depending on the guest OS and the applications, this might or might not have an adverse effect on performance.
• Shares are used to determine CPU allocation when the ESXi host is experiencing CPU contention. Like memory, shares grant CPU access on a percentage basis calculated on the number of shares granted out of the total number of shares assigned. This means that the percentage of CPU cycles granted to a VM based on its Shares value is always relative to the number of other VMs and the total number of shares granted, and it is not an absolute value.

As you can see, there are some key differences as well as a number of similarities between how these mechanisms work for memory when compared to how they work for CPU.

So far we've discussed two of the four major resource types (memory and CPU). Before we can move on to the third resource type—networking—we need to discuss the concept of resource pools.

Using Resource Pools

The settings for VM resource allocation (memory and CPU reservations, limits, and shares) are methods used to modify or control the allocation of resources to individual VMs or to modify the priority of VMs seeking access to resources. In much the same way as you assign users to groups and then assign permissions to the groups, you can leverage resource pools to make allocating resources to collections of VMs a less tedious and more effective process. In other words, instead of configuring reservations, limits, or shares on a per-VM basis, you can use a resource pool to set those values on a group of VMs all at once.

A resource pool is a special type of container object, much like a folder, in the Hosts And Clusters view. You can create a resource pool on a stand-alone host or as a management object in a DRS-enabled cluster. Figure 11.9 shows the creation of a resource pool.

If you examine the properties of the resource pool, you'll see two sections: one for CPU settings (Reservation, Limit, and Shares) and another section with similar settings for memory. When you apply resource settings to a resource pool, those settings affect all the VMs found within that resource pool. This provides a scalable way to adjust the resource settings for groups of VMs. Setting CPU and memory shares, reservations, and limits on a resource pool is very much like setting these values on individual VMs. The behavior of these values, however, can be quite different on a resource pool than on an individual VM.

To illustrate how to set shares, reservations, and limits on a resource pool, as well as to explain how these values work when applied to a resource pool, we'll use an example of an ESXi host with two resource pools. The resource pools are named ProductionVMs and DevelopmentVMs. Figure 11.10 and Figure 11.11 show the values that have been configured for the ProductionVMs and DevelopmentVMs resource pools, respectively.

FIGURE 11.9 You can create resource pools on individual hosts and within clusters. A resource pool provides a management and performance configuration layer in the vCenter Server inventory.

FIGURE 11.10 The ProductionVMs resource pool is guaranteed CPU and memory resources and higher-priority access to resources in the face of contention.

FIGURE 11.11 The Development VMs resource pool is configured for lower-priority access to CPU and memory in the event of resource contention.

Configuring Resource Pools

Before we can show you how resource pools behave with regard to resource allocation, you must first create and configure the resource pools. Use the resource pools, shown earlier in Figure 11.10 and Figure 11.11, as examples for creating and configuring resource pools.

To create a resource pool, simply right-click either an individual ESXi host or a cluster of ESXi hosts, and select New Resource Pool. In the Create Resource Pool dialog box, you'll need to supply a name for the new resource pool and set the CPU and Memory values as desired.

After you create the resource pool, you must move the VMs into it by clicking the VM in the inventory panel and dragging it onto the appropriate resource pool. The result is a hierarchy similar to that shown in Figure 11.12.

FIGURE 11.12 VMs assigned to a resource pool consume resources allocated to the resource pool.

In this particular example, you have two classifications of servers, production and development, and you've created a resource pool for each classification. The goal in this example is to ensure that if there's competition for a particular resource, the VMs in production should be assigned higher-priority access to that resource. In addition to that goal, you need to ensure that the VMs in development cannot consume more than 24 GB of physical memory with their running VMs. You don't care how many VMs run concurrently as part of the development group as long as they don't collectively consume more than 24 GB of RAM. Finally, you need to ensure that a minimum amount of resources are guaranteed for both groups of VMs.

To achieve your goal of guaranteeing resources for the production VMs, you will set the ProductionVMs resource pool to use the following settings (refer to Figure 11.10 earlier):

• CPU resources area: Shares value of High.
• CPU resources area: Reservation value of 11,700 MHz.
• CPU resources area: Expandable check box Reservation Type is deselected.
• CPU resources area: No CPU limit (Unlimited option in the Limit drop-down is selected).
• Memory resources area: Reservation value of 16,384 MB.
• Memory resources area: Expandable check box for Reservation Type is deselected.
• Memory resources area: No memory limit (Unlimited option in the Limit drop-down is selected).

Similarly, you will apply the following settings to the DevelopmentVMs resource pool (see Figure 11.11 earlier):

• CPU resources area: Reservation value of 2,925 MHz.
• CPU resources area: Expandable check box for Reservation Type is deselected.
• CPU resources area: Limit value of 11,700 MHz.
• Memory resources area: Reservation value of 4,096 MB.
• Memory resources area: Expandable check box for Reservation Type is deselected.
• Memory resources area: Limit value of 24,576 MB.

Again, setting the values on the DevelopmentVMs resource pool involves right-clicking the resource pool, selecting Edit Settings, and then setting the values you need.

Now that you have an example to work with, we'll explain what these settings will do to the VMs contained in each of the resource pools.

Understanding Resource Allocation with Resource Pools

In the previous section we walked you through creating a couple of resource pools called ProductionVMs and DevelopmentVMs. The values for these resource pools are illustrated in Figure 11.10 and Figure 11.11. The goal behind creating these resource pools and setting the values on them was to ensure that a certain level of resources would always be available to production VMs (those found in the ProductionVMs resource pool) and to limit the resources used by the development VMs (VMs found in the DevelopmentVMs resource pool). In this example, you used all three values—Shares, Reservation, and Limit—in an effort to accomplish your goal. Let's look at the behavior of each of these values when used on a resource pool.

MANAGING CPU USAGE WITH RESOURCE POOLS

First we'll examine the Shares value assigned to the resource pools for CPU usage. As you can see in Figure 11.10, the CPU shares for the ProductionVMs resource pool are set to High (8,000). Figure 11.11 shows the DevelopmentVMs CPU shares set to Low (2,000). The effect of these two settings is similar to that of comparing two VMs' Shares values for CPU—except in this case, if there is any competition for CPU resources between VMs in the ProductionVMs and DevelopmentVMs resource pools, the entire ProductionVMs resource pool and all the VMs in it would have higher priority. Figure 11.13 shows how this would break down with two VMs in each resource pool.

FIGURE 11.13 Two resource pools with different Shares values will be allocated resources proportional to their percentage of share ownership.

As you consider the information presented in Figure 11.13, keep in mind that the resource allocation occurs at each level. There are only two resource pools under the given ESXi host, so the CPU is allocated 80/20 according to its Shares value. This means that the ProductionVMs resource pool gets 80 percent of the CPU time while the DevelopmentVMs resource pool gets only 20 percent of the CPU time.

Now let's expand on Figure 11.13 and add the two VMs in each resource pool to get a more complete view of how Shares values would work with a resource pool. Within the resource pool the CPU Shares values assigned to the VMs, if any at all, come into play. Figure 11.14 shows how this works.

In Figure 11.14, there are no custom CPU shares assigned to the VMs, so they all use the default value of 1,000 CPU shares. With two VMs in the resource pool, this means each VM gets 50 percent of the resources available to the resource pool in which it is located (because each VM has 50 percent of the total number of shares assigned within the pool). In this example, this means 40 percent of the host CPU capacity will go to each of the two VMs in the ProductionVMs resource pool. If there were three VMs in each resource pool, then the CPU allocated to the parent resource pool would be split three ways. Similarly, if there were four VMs, then the CPU would be split four ways. You can verify this breakdown of resource allocation using the Monitor tab on the selected cluster, ESXi host, or resource pool. Figure 11.15 shows the Resource Allocation subsection for a cluster with the ProductionVMs and DevelopmentVMs resource pools. The CPU button is selected, meaning that the Web Client is showing you the breakdown of CPU allocation for the selected cluster.

FIGURE 11.14 The percentage of resources assigned to a resource pool via its Shares values is further subdivided according to the Shares values of the VMs within the pool.

FIGURE 11.15 The Resource Allocation tab can verify the allocation of resources to objects within the vCenter Server hierarchy.

Note that in the screenshot in Figure 11.15, both resource pools and VMs are directly in the root of the cluster (which, for all intents and purposes, is a resource pool itself). In this case, the sum of all the Shares values—for both resource pools as well as VMs—is used to calculate the percentage of CPU allocated.

Now that we've introduced you to the Resource Allocation tab, we need to discuss an important consideration about the use of resource pools. It's possible to use resource pools as a form of organization, as you would use folders. Some organizations and administrators have taken to using resource pools in this way to help keep VMs organized in a specific fashion. While this is possible, it's not recommended. The Resource Allocation tab helps show why.

Look at Figure 11.16, which shows the Resource Allocation tab for a cluster of ESXi hosts. In the root of this cluster are six VMs assigned a total of 7,000 shares. Because each of these VMs is using the default CPU Shares value (1,000 shares per vCPU), they each get equal access to the host CPU capacity—in this case, 14 percent per vCPU (the VMs with 2,000 shares and 28-percent shares have two vCPUs).

FIGURE 11.16 In the absence of custom CPU shares, CPU capacity is equally allocated to all VMs.

Now look at Figure 11.17. The only change here is that we've added a resource pool. I did not change any of the default values for the resource pool. Note that the resource pool has a default CPU Shares Value of 4,000, and note how the simple addition of this resource pool changes the default CPU allocation for the individual VMs from 14 percent per vCPU to only 9 percent per vCPU. The resource pool, on the other hand, now gets 36 percent. If you added a single VM to the resource pool, that one VM would get 36 percent of the host CPU capacity while other VMs only received 9 percent (or 18 percent for VMs with two vCPUs).

This unintended change on the resource allocation distribution is why we don't recommend using resource pools strictly for the purposes of organizing VMs. If you do insist on using resource pools in this way, be sure to understand the impact of configuring your environment in this manner. A better way to organize your environment, and one that won't impact the performance, is to use folders or tags within vCenter; for more information, see Chapter 3, “Installing and Configuring vCenter Server.”

FIGURE 11.17 The addition of a resource pool will, by default, alter the resource allocation policy even if you don't set any custom values.

The next setting in the resource pool properties to evaluate is CPU Reservation for the CPU. Continuing with the examples shown earlier in Figure 11.10 and Figure 11.11, you can see a CPU Reservation value of 11,700 MHz has been set on the ProductionVM's resource pool. The DevelopmentVMs pool has a CPU Reservation value of 2,925 MHz. (The ESXi hosts in the cluster hosting these resource pools have quad-core 2.93 GHz Intel Xeon CPUs, so this essentially reserves four cores on one server for the ProductionVMs resource pool and one core on one server for the DevelopmentVMs resource pool.) This setting ensures that at least 11,700 MHz of CPU time is available for all the VMs located in the ProductionVMs resource pool (or 2,925 MHz of CPU for VMs in the DevelopmentVMs resource pool). Assuming that the ESXi host has a total of 23,400 MHz CPU (8 × 2,925 MHz = 23,400 MHz), this means 8,775 MHz of CPU time is available on that host for other reservations. If one more resource pool was created with a Reservation value of 8,775 MHz, then the cumulative reservations on the system have reserved all available host CPU capacity (5,850 MHz × 4 = 23400 MHz). This configuration means the administrator will not be able to create any additional resource pools or any individual VMs with Reservation values set. Remember that the ESXi host or cluster has to have enough resource capacity—CPU capacity, in this case—to satisfy all reservations. You can't reserve more capacity than the host actually has.

Part of the CPU Reservation setting is the option to make the reservation expandable. An expandable reservation (noted as such by selecting the Expandable check box next to Reservation Type) allows a resource pool to “borrow” resources from its parent host or parent resource pool in order to satisfy reservations set on individual VMs within the resource pool. Note that a resource pool with an expandable reservation would “borrow” from the parent only to satisfy reservations, not to satisfy requests for resources in excess of the reservations. Neither of the resource pools has expandable reservations, so you will be able to assign only 5,850 MHz of CPU capacity as reservations to individual VMs within each resource pool. Any attempt to reserve more than that amount will result in an error message explaining that you've exceeded the allowed limit.

Deselecting the Expandable check box does not limit the total amount of CPU capacity available to the resource pool; it limits only the total amount of CPU capacity that can be reserved within the resource pool. To set an upper limit on actual CPU usage, you'll need to use a CPU Limit setting.

CPU Limit is the third setting on each resource pool. The behavior of the CPU limit on a resource pool is similar to its behavior on individual VMs, except in this case the limit applies to all VMs in the resource pool. All VMs combined are allowed to consume up to this value. In the example, the ProductionVMs resource pool does not have a CPU limit assigned. In this case, the VMs in the ProductionVMs resource pool are allowed to consume as many CPU cycles as the ESXi hosts in the cluster can provide. The DevelopmentVMs resource pool, on the other hand, has a CPU Limit setting of 11,700 MHz, meaning that all the VMs in the DevelopmentVMs resource pool are allowed to consume a maximum of 11,700 MHz of CPU capacity. With 2.93 GHz Intel Xeon CPUs, this is the approximate equivalent of one quad-core CPU.

For the most part, CPU shares, reservations, and limits behave similarly on resource pools and on individual VMs. The same is also true for memory shares, reservations, and limits, as you'll see in the next section.

MANAGING MEMORY USAGE WITH RESOURCE POOLS

In the memory portion of the resource pool settings, the first setting is the Shares value. This setting works in much the same way as memory shares worked on individual VMs. It determines which group of VMs will be the first to give up memory via the balloon driver—or if memory pressure is severe enough, activate memory compression or swap out to disk via hypervisor swapping—in the face of contention. However, this setting sets a priority value for all VMs in the resource pool when they compete for resources with VMs in other pools. Looking at the memory share settings in our example (ProductionVMs = Normal and DevelopmentVMs = Low), this means that if host memory is limited, VMs in the DevelopmentVMs resource pool that need more memory than their reservation would have a lower priority than an equivalent VM in the ProductionVMs resource pool. Figure 11.14, which we used previously to help explain CPU shares on resource pool, applies here as well. As with CPU shares, you can also use the Resource Allocation tab to explore how memory resources are assigned to resource pools or VMs within resource pools.

The second setting is the resource pool's memory reservation. The memory Reservation value will reserve this amount of host RAM for VMs in this resource pool, which effectively ensures that some actual RAM is guaranteed to the VMs. As explained in the discussion on CPU reservations, the Expandable check box next to Reservation Type does not limit how much memory the resource pool can use but rather how much memory you can reserve there.

With the memory Limit value, you set a limit on how much host RAM a particular group of VMs can consume. If administrators have been given the Create Virtual Machines permission, then the memory Limit value would prevent those administrators from running VMs that consume more than that amount of actual host RAM. In our example, the memory Limit value on the DevelopmentVMs resource pool is set to 24,576 MB. How many VMs can administrators in development create? They can create as many as they want.

Although this setting does nothing to limit creating VMs, it places a limit on running VMs. So, how many can they run? The cap placed on memory use is not a per-VM setting but a cumulative setting. Administrators might be able to run only one VM with all the memory or multiple VMs with lower memory configurations. Assuming that each VM is created without an individual memory Reservation value, the administrator can run as many VMs concurrently as they want. However, once the VMs consume 24,576 MB of host RAM, the hypervisor will step in and prevent the VMs in the resource group from using any additional memory. Refer back to the discussion of memory limits in the section titled “Using Memory Limits” for the techniques that the VMkernel uses to enforce the memory limit. If the administrator builds six VMs with 4,096 MB as the initial memory amount, then all four VMs will consume 24,576 MB (assuming no overhead, which we've already shown you isn't the case) and will run in real RAM. If an administrator tried to run 20 VMs configured for 2,048 MB of RAM, then all 20 VMs will share the 24,576 MB of RAM, even though their requirement is for 40,960 MB (20 × 2,048 MB)—the remaining amount of RAM would most likely be provided by VMkernel swap. At this point, performance would be noticeably slow.

If you want to clear a limit, select Unlimited from the Limit drop-down menu. This is true for both CPU limits as well as memory limits. By now you should have a fair idea of how ESXi allocates resources to VMs as well as how you can tweak those settings to meet your specific demands and workloads.

As you can see, if you have groups of VMs with similar resource demands, using resource pools is an excellent way of ensuring consistent resource allocation. As long as you understand the hierarchical nature of resource pools—that resources are allocated first to the pool at its level in the hierarchy, and then the VMs in the pool—then you should be able to use resource pools effectively.

So far you've seen how to control the use of CPU and memory, but those are only two of the four major resources consumed by VMs. In the next section, you'll see how to control network traffic through network resource pools.

Regulating Network I/O Utilization

The resource pools we've shown you so far can only be used to control CPU and memory usage. However, vSphere offers another type of resource pool, a network resource pool that allows you to control network utilization. Using network resource pools—to which are assigned shares and limits—you can control incoming and outgoing network traffic. This feature is referred to as vSphere Network I/O Control (NIOC).

When you enable vSphere NIOC, vSphere activates eight predefined network resource pools:

• NFS Traffic
• Management Traffic
• vMotion Traffic
• vSphere Storage Area Network Traffic
• vSphere Replication (VR) Traffic
• iSCSI Traffic
• Virtual Machine Traffic
• Fault Tolerance (FT) Traffic

All of these network resource pools are visible on the Resource Allocation tab of the vDS, as you can see in Figure 11.18.

FIGURE 11.18 Network resource pools on a vDS provide granular control of network traffic.

Two steps are involved in setting up and using NIOC. First, you must enable NIOC on that particular vDS. Second, you must create and configure network resource pools as necessary. The first of these steps is already complete if you create a brand-new vDS with a version set to 5.5.0—NIOC is enabled by default.

Perform the following steps to enable NIOC on an existing vDS:

1. If it is not already running, launch the Web Client and connect to a vCenter Server instance. Because NIOC relies on vDS and vDS is available only with vCenter, NIOC cannot be used when connected directly to an ESXi host.
2. Navigate to the Networking view using the navigation bar or the home screen.
3. Select the vDS for which you want to enable NIOC.
4. Right-click the vDS.
5. Click Edit Settings.
6. Select Enabled in the Network I/O Control drop down box, and then click OK.

This enables NIOC on this vDS. The Resource Allocation tab of the vDS object will note that NIOC is enabled, as shown in Figure 11.19.

Along with actually enabling NIOC, you can also modify existing network resource pools or create new resource pools, if you are using a vDS version 5.0.0 or above. For a version 4.1.0 vDS, you can neither create new network resource pools nor edit the existing network resource pools.

A network resource pool consists of three basic settings:

• The first value is Host Limit. This value specifies an upper limit on the amount of network traffic, in Mbps, that this network resource pool is allowed to consume. Leaving Unlimited selected means that only the physical adapters themselves limit the network resource pool.
• The second value is Physical Adapter Shares. Like the shares you used to prioritize access to CPU or RAM when there was contention, physical adapter shares in a network resource pool establish priority for access to the physical network adapters when there is network contention. As with other types of shares, this value does not apply when there is no contention.

  You can set this value to one of three predefined values, or you can set a custom value of up to 100. For the predefined values, Low translates to 25 shares, Normal equates to 50 shares, and High equals 100 shares.

• The third value is QoS Tag. The QoS (Quality of Service) tag is an 802.1p tag that is applied to all outgoing packets. Upstream network switches that are configured to recognize the 802.1p tags can further enhance and enforce the QoS beyond just the ESXi host.

FIGURE 11.19 vCenter Server provides a clear indication that NIOC is enabled for a vDS.

Figure 11.20 shows all three of the values for one of the predefined network resource pools, the Fault Tolerance (FT) Traffic network resource pool.

FIGURE 11.20 vSphere allows an administrator to modify the pre-defined network resource pools.

Administrators have the option of editing the predefined network resource pools or creating their own network resource pools.

Perform the following steps to edit an existing network resource pool:

1. If it is not already running, launch the Web Client and connect to a vCenter Server instance.
2. Navigate to the Networking view.
3. Select the vDS that contains the network resource pool you want to modify.
4. Click the Resource Allocation tab.
5. Select the network resource pool you want to edit and click the Edit pencil icon.
6. From the Network Resource Pool Settings dialog box, modify the Host Limit, Physical Adapter Shares, or QoS Tag values as desired.
7. Click OK to save the changes to the network resource pool.

You might prefer to leave the predefined network resource pools intact and create your own. Perform the following steps to create a new network resource pool:

1. If it is not already running, launch the Web Client and connect to a vCenter Server instance.
2. Navigate to the Networking view.
3. Select the vDS on which you want to create the new network resource pool.
4. Click the Resource Allocation tab.
5. Click the New Network Resource Pool plus icon. The New Network Resource Pool dialog box appears (Figure 11.21).

   FIGURE 11.21 Administrators have the option of creating new network resource pools for custom network traffic control.

   

6. Supply a name and description for the new network resource pool.
7. For Physical Adapter Shares, select a predefined option (Low, Normal, or High), or select Custom and enter a value from 1 to 100.
8. To set a limit, select the Unlimited check box (to uncheck it) and then enter a value for Host Limit. This value is entered in Mbps (megabits per second).
9. If you want to apply a QoS priority tag, select the value from the drop-down list.
10. Click OK to create the new network resource pool with the values you specified.

After you have at least one user-defined network resource pool, you have the option of mapping port groups to your network resource pool.

Perform the following steps to assign a port group to a user-defined network resource pool:

1. Launch the Web Client if it is not already running, and connect to a vCenter Server instance.
2. Switch to the Networking view.
3. Select the vDS that hosts the network resource pool you'd like to map to a port group.
4. Right-click the RPort Group you wish to map and select Edit Settings.
5. Within the settings, select the appropriate network resource pool, as shown in Figure 11.22.

   FIGURE 11.22 Users can map a port group to any user-defined network resource pool, and multiple port groups can be associated to a single network resource pool.

   

6. Click OK to save the changes and return to the Resource Allocation tab.

In large environments with lots of port groups, it might be a bit tedious to try to determine which port groups are mapped to which network resource pools. To help ease this administrative burden, vCenter Server offers an easy way to show all the port groups linked to a particular network resource pool:

• With a network resource pool selected, simply click the Distributed Port Groups tab near the bottom of the screen.
• The view will shift to show you the specific port groups associated with the selected network resource pool. You can see this displayed in Figure 11.23, which shows the port groups associated with the user-defined network resource pool named Custom Network Resource Pool. You'll notice that a fair amount of networking-specific detail—like VLAN ID, port binding, number of attached VMs, and so forth—is also included in this display for ease of use.

FIGURE 11.23 The Web Client provides a consolidated view of all the port groups associated with a network resource pool for reduced administrative overhead.

NIOC offers a powerful way to help ensure that all the various types of converged network traffic present in a VMware vSphere environment will coexist properly, especially as organizations move toward 10 Gigabit Ethernet and away from Gigabit Ethernet. Fewer faster connections means more consolidated traffic and therefore a greater need for controlling how that traffic coexists on the same physical medium.

We've taken you through three of the four major resources and shown you how VMware vSphere offers controls for managing the use of and access to them. Only one resource remains: storage.

Controlling Storage I/O Utilization

For vSphere, controlling memory or CPU allocation and utilization is relatively easy. The hypervisor can easily determine how busy the CPU is and if physical memory has been depleted. When resource contention occurs for these resources, not only is it easy to detect, but it's also easy to correct. If the CPU is too busy and there aren't enough CPU cycles to go around, then you don't schedule cycles for lower-priority VMs and you assign more cycles to higher-priority VMs. And how is this priority determined? Remember that Shares values are the mechanism that vSphere uses to determine priority. Likewise, if RAM becomes constrained, invoke the balloon drivers in the guest OSes and reclaim some memory, or slow down the rate of allocation to lower-priority VMs and increase the rate of allocation to higher-priority VMs. Not only does the hypervisor have complete visibility into the utilization of these resources, it also has complete control over the resources. Nothing gets scheduled on the CPU without going through the hypervisor, and nothing gets stored in RAM without the hypervisor knowing about it. This complete control is what enables vSphere to offer shares as a way of establishing priority and offer reservations (guaranteed access to resources) and also limits (caps on the usage of a resource). You read and learned about these mechanisms in previous parts of this chapter.

When you get to network utilization, things begin to change a little. The hypervisor has some visibility into the network; it can see how many Mbps are being generated and by which VMs. However, the hypervisor does not have complete control over network utilization. It can control inbound and outbound traffic, but traffic generated somewhere else in the network really can't be controlled. Given the nature of networking, it's pretty much a given that other workloads outside the control of VMware vSphere will be present, and vSphere can't control or influence them in any way. Even so, vSphere can offer shares (to establish priority) and limits (to enforce a cap on the amount of network bandwidth a VM can consume). This is Network I/O Control, and we discussed it in the previous section.

With regard to resource allocation and utilization, storage is similar in many ways to networking. Other workloads are likely to be present on the shared storage vSphere requires for so many features. These other workloads will be external to vSphere and can't be controlled or influenced in any way, and therefore vSphere isn't going to have complete control over the resource. It's also generally true that the hypervisor won't have as much visibility into the storage as it does with CPU and memory, making it more difficult to detect and adjust storage resources. There are two metrics, however, that vSphere can use to help determine storage utilization. The first is latency and the second is peak throughput. Using one of these two metrics to detect contention, vSphere can offer shares (to establish priority when contention occurs) as well as limits (to ensure that a VM doesn't consume too many storage resources). The feature that enables this functionality is called Storage I/O Control, or SIOC.

Longtime users of VMware vSphere (and VMware Infrastructure before that) are probably aware that you've been able to assign Shares values to disks for quite some time. The difference between that functionality and what SIOC offers is a matter of scope. Without SIOC, enabling shares on a VM's virtual disk is only effective for that specific host; the ESX/ESXi hosts did not exchange information about how many shares each VM was allocated or how many shares were assigned in total. This meant it was impossible to properly align the Shares values with the correct ratios of access to storage resources across multiple hosts.

SIOC addresses this by extending shares assignments across all hosts accessing a particular datastore. Using vCenter Server as the central information store, SIOC combines all the assigned shares across all the VMs on all the hosts and allocates storage I/O resources in the proper ratios according to the shares assignment.

In order to make this work, SIOC has a few requirements you must meet:

• All datastores that are SIOC-enabled must be managed under a single vCenter Server instance. vCenter Server is the “central clearinghouse” for all the shares assignments, so it makes sense that all the datastores and hosts be managed by a single vCenter Server instance.
• SIOC is supported on VMFS datastores connected via Fibre Channel (including FCoE) and iSCSI. NFS datastores are also supported. Raw device mappings (RDMs) are not supported.
• Datastores must have only a single extent. Datastores with multiple extents are not supported.

Assuming that your environment meets the requirements, then you can take advantage of SIOC. Configuring SIOC is a two-step process. First, enable SIOC on one or more datastores. Second, assign shares or limits to storage I/O resources on individual VMs.

Let's look first at enabling SIOC for a particular datastore.

Enabling Storage I/O Control

SIOC is enabled on a per-datastore basis. By default, SIOC is disabled for a datastore, meaning that you explicitly enable SIOC to take advantage of its functionality.

Perform the following steps to enable SIOC for a datastore:

1. Launch the Web Client, if it is not already running, and connect to a vCenter Server instance.

   SIOC is available only when connected to vCenter Server, not when you are connected to an individual ESXi host with the vSphere Client.

2. Navigate to the Storage view.
3. Select the datastore for which you want to enable SIOC.
4. Click the Manage Settings tab.
5. Select Edit (Figure 11.24).
6. In the Datastore Capabilities dialog box, select Enabled under Storage I/O Control.
7. Click OK.

SIOC is now enabled for the selected datastore; this is reflected in the Datastore Capabilities pane of the Web Client under the datastore manage tab, as you can see in Figure 11.25.

FIGURE 11.24 This dialog box allows you to manage the SIOC configuration of a specific datastore.

FIGURE 11.25 The status of SIOC for a datastore is displayed in the Web Client for easy reference.

Generally speaking, enabling SIOC using these steps is all you need to do to get started using SIOC to control use of storage I/O resources.

However, in some cases, you may need to adjust the configuration of SIOC in order to make it function properly for your specific array and array configuration. Previously in this section, we mentioned that vSphere has two metrics that could be used to detect contention: latency and peak throughput.

SIOC can use latency as the threshold to determine when it should activate and enforce Shares values for access to storage I/O resources. Specifically, when vSphere detects latency in excess of a specific threshold value (measured in milliseconds), SIOC is activated. Because of the vast differences in array architectures and array performance, VMware recognized that users might need to adjust this default congestion threshold value for SIOC. After all, a certain latency measurement might indicate congestion (or contention) on some arrays and configurations but not on others. Making the congestion threshold adjustable allows vSphere administrators to fine-tune the behavior of SIOC to best match their particular array and configuration.

The other metric is the one that's enabled by default in vSphere 5.5—Peak Throughput. SIOC uses a special tool called the IO Injector to test the characteristics of a storage array and perform two main checks. The first check is for overlaps in the underlying disk system of a datastore; this allows Storage DRS to make correct placement decisions for workload balancing, which is explained more in Chapter 12. The other check performed by the IO Injector—more relevant to this section—is a Peak Throughput check. Unlike checking for latency spikes, knowing the maximum capability of a particular datastore can ensure that SIOC kicks in before congestion (or contention) occurs. Just like latency, you can adjust this threshold with a percentage setting.

Perform the following steps to adjust the congestion threshold setting for SIOC on a particular datastore:

1. If it is not already running, launch the Web Client and connect to a vCenter Server instance.
2. Navigate to the Storage view.
3. Select the desired datastore from the inventory tree.
4. Select the datastore for which you want to enable SIOC.
5. Click the Manage Settings tab and click Edit.
6. The Edit Congestion Threshold dialog box appears. Enter the desired congestion threshold setting, in percentage or milliseconds, and then click OK.

Once you have enabled SIOC on one or more datastores and you have (optionally) adjusted the congestion threshold per your storage vendor's recommended values, you can start setting storage I/O resource values on your VMs.

Configuring Storage Resource Settings for a Virtual Machine

SIOC provides two mechanisms for controlling the use of storage I/O by VMs: shares and limits. These mechanisms operate in exactly the same way here as with other resources; the Shares value establishes a relative priority as a ratio of the total number of shares assigned, while the Limit value defines the upper ceiling on the number of I/O operations per second (IOPS) that a given VM may generate. As with memory, CPU, and network I/O, vSphere provides default settings for disk shares and limits. By default, every VM you create is assigned 1,000 disk shares per virtual disk and no IOPS limits.

If you need different settings than the default values, you can easily modify either the assigned storage I/O shares or the assigned storage I/O limit.

ASSIGNING STORAGE I/O SHARES

Modifying the default storage I/O Shares value is done on the VM's Edit Settings dialog box, just as it's done for modifying memory allocation or CPU utilization. Figure 11.26 shows the disk shares for a VM.

FIGURE 11.26 Storage I/O shares are modified from the same area as other resource allocation settings are modified.

Perform the following steps to modify the storage I/O Shares value for a VM:

1. Launch the Web Client, if it is not already running, and connect to a vCenter Server instance.
2. Navigate to either the Hosts And Clusters or VMs And Templates view.
3. Right-click the specific VM for which you'd like to change the storage I/O settings and select Edit Settings from the context menu.
4. Click the triangle next to the hard disk. This displays the dialog box shown previously in Figure 11.26.
5. For each virtual disk, click in the Shares drop-down to change the setting from Normal to Low, High, or Custom, as shown in Figure 11.27.

   FIGURE 11.27 You must change the setting to Custom if you want to assign an arbitrary storage I/O Shares value.

   

6. If you selected Custom in step 5, click in the Shares value drop-down and supply a custom storage I/O Shares value.
7. Repeat steps 4, 5, and 6 for each virtual disk associated with this VM.
8. Click OK to save the changes and return to the Web Client.

The selected virtual disks belonging to this VM will now receive a proportional allocation of storage I/O resources based on the Shares value whenever SIOC detects contention (or congestion) on the datastore. (Keep in mind that vSphere can use latency or peak bandwidth, as specified in the congestion threshold described previously, as the trigger for activating SIOC.) As with all other Shares values, SIOC enforces Shares values only when contention for storage I/O resources is detected. If there is no contention—as indicated by low latency or bandwidth values for that datastore or datastore cluster—then SIOC will not activate.

CONFIGURING STORAGE I/O LIMITS

You can also set a limit on the number of IOPS that a VM is allowed to generate. By default this value is unlimited. However, if you feel that you need to set an IOPS limit, you can set the IOPS limit in the same place you would set storage I/O shares.

Perform these steps to set a storage I/O limit on IOPS:

1. If it is not already running, launch the Web Client and connect to a vCenter Server instance.
2. Navigate to the Hosts And Clusters or VMs And Templates view.
3. Right-click a VM and select Edit Settings.
4. Click the triangle next to the disk on which you'd like to set an IOPS limit.
5. Click in the Limit - IOPS drop-down and type in a value for the maximum number of IOPS that this VM will be allowed to generate against this virtual disk.
6. Repeat steps 4 and step 5 for each virtual disk assigned to this VM.
7. Click OK to save the changes and return to the Web Client.

Like the limits you apply to memory, CPU, or network I/O, the storage I/O limits are absolute values. The hypervisor will enforce the assigned storage I/O limit, even when there is plenty of storage I/O available.

Setting these storage I/O resource values on a per-VM basis is fine, but what about when you need to have some sort of consolidated view of what settings have been applied to all the VMs on a datastore? Fortunately, vCenter Server and the Web Client provide a way to easily see a summary of the various settings.

VIEWING STORAGE I/O RESOURCE SETTINGS FOR VIRTUAL MACHINES

In the Datastores And Datastore Clusters view, you can view a list of all the datastores managed by a particular vCenter Server instance. You enabled SIOC from this view previously, in the section “Enabling Storage I/O Control,” and here you can get a consolidated view of all the storage I/O settings applied to the VMs on a datastore.

On the Virtual Machines tab of a selected datastore in the Datastores And Datastore Clusters view, vCenter Server provides a list of all the VMs on that datastore. If you scroll to the right using the scroll bar at the bottom of the Web Client window, you will see three SIOC-specific columns:

• Shares Value
• Limit–IOPs
• Datastore % Shares

Figure 11.28 shows these three columns on a datastore for which SIOC has been enabled. Note that the default values for the VMs on the selected SIOC-enabled datastore have not been modified.

FIGURE 11.28 The Virtual Machines tab of a datastore provides a useful summary view of storage-related information for all the VMs on that datastore.

As you can see in Figure 11.28, vCenter Server has used the assigned Shares values to establish relative percentages of access to storage I/O resources in the event of contention. This consistent behavior makes the complex task of managing resource allocation a bit easier for vSphere administrators.

So far we've discussed shared storage where the features and settings are all array agnostic—Fibre Channel, iSCSI, or NFS—it really doesn't matter how it's presented to your ESXi hosts. In the next section, we'll introduce some features that are solely for use with local SSD-based storage.

Utilizing Flash Storage

As physical to virtual ratios increase and environments become denser, the need for faster storage is obvious. Flash storage is quickly becoming an industry standard, and it's not uncommon for both servers and SANs to ship with flash-based storage. Traditionally, SANs have used smaller portions of flash storage as caches to increase the response times to slower spindles, but as SAN sizes increase, so does the need for expensive flash cache. Depending on the scenario, a more cost-effective solution can be to load up servers (either rack mount or blade) with local SSD or PCIe-based flash storage.

vSphere 5.5 offers two completely different ways of using local flash-based host storage, whether it be SSD drives or PCIe cards. vFlash Cache (vFC) is a feature that acts as a buffer for I/O on a per-VM basis. The other feature, Swap to Host Cache, is used to allocate local flash disks as swap space. These features are not mutually exclusive; they can be enabled at the same time and can even be backed by the same SSD, but they work in completely different ways.

ENABLING VFLASH CACHE FOR VIRTUAL MACHINES

vFlash caching is a resource type that can be allocated on a per-VM basis just like CPU or memory. In fact, it can be allocated down to an individual virtual disk (VMDK) level. However, a VM does not need to have vFlash allocated to function. Just like CPU and Memory allocations, no vFlash resources are consumed and the contents of the VM cache are flushed when the VM is powered off. Also just like CPU and Memory allocation, provided you have enough resources configured on each host in a cluster, vFlash Cache is compatible with HA, vMotion, and therefore DRS.

The following instructions outline how to enable vFlash Cache:

1. If it is not already running, launch the Web Client and connect to a vCenter Server instance.
2. Navigate to the Hosts And Clusters or VMs And Templates view.
3. With a host selected, click Manage Settings Virtual Flash Resource Management.
4. Within this screen, click the Add Capacity button in the top-right corner of the content area.
5. Select the SSD from the list, as shown in Figure 11.29. Click OK to close the dialog box. The SSD is now allocated to the vFlash Cache resource pool.

   FIGURE 11.29 New to vSphere 5.5 you can now add SSDs to the vFlash Cache resource capacity.

   

6. Navigate to the VM that you wish to allocate vFlash resources to, select the Actions dropdown menu, and click Edit Settings.
7. From within the Edit Settings dialog box, expand the Hard Disk area of the VM properties and allocate an amount of resources in GB or MB, as shown in Figure 11.30.
8. Click OK to close the dialog box and commit the configuration change to the VM.

FIGURE 11.30 Allocating vFlash Cache to a V M is as simple as allocating it as you would CPU or memory.

As you can see, configuring and allocating vFlash Cache is a fairly simple task. What won't be so easy is working out which VM hard disks benefit from allocating vFlash Cache and how much to allocate. In Chapter 13, “Monitoring VMware vSphere Performance,” we discuss monitoring vSphere Performance in great detail. The use of performance graphs and resxtop will be of great benefit when you start to baseline workloads for vFlash Cache inclusion. Depending on the workloads that your environment runs, you may find that vFlash Cache is of little benefit. In a low I/O but high memory overcommit, you may find that disk I/O is increasing because the hosts need to swap out memory to disk, just as described earlier in this chapter. In this scenario, allocating flash-based resources to swap would be a good idea.

CONFIGURING SWAP TO HOST CACHE

Swap to Host Cache, or “Swap to SSD,” is a feature that allocates a portion of local SSD storage to be the location of a VMs swapfile. As discussed earlier in this chapter, the VM swapfile is very different from the OS swap/page file.

One very important point to note—this feature is beneficial only within environments that have memory contention to the point that the hypervisor has to swap unreserved VM memory to disk. Based on the figures explained earlier in this chapter, accessing memory swapped to disk (even SSD) is still significantly slower than accessing it from RAM. That being said, if your environment fits within this category, there are significant performance benefits from enabling this feature.

To configure the host for Swap to Cache, a VMFS datastore that has been identified as being backed by an SSD needs to be present. The following steps outline how to enable this feature:

1. If it is not already running, launch the Web Client and connect to a vCenter Server instance.
2. Navigate to the Hosts And Clusters view.
3. Select the appropriate host.
4. Click the Manage Storage tab.
5. Select Host Cache Configuration as shown in Figure 11.31.

   

   FIGURE 11.31 The Host Cache Configuration feature is designed specifically for SSD datastores.

6. Highlight the appropriate SSD-backed VMFS datastore and click the Edit pencil icon.
7. Finally, ensure the Allocate Space For Host Cache is checked. The size is customizable if you would like to use this datastore for other purposes.
8. Click OK to finalize the configuration.

Once Swap to Host Cache is enabled, the host proceeds to fill the allocated size on the data-store with .vswp files (see Figure 11.32). These .vswp files will be used instead of the “regular” .vswp files that are usually found within the same datastore as the other VM files. Something to keep in mind here is that this setting needs to be enabled on every host within a cluster because it is not cluster aware. If a VM is vMotioned to a host that does not have Swap to Host Cache enabled, the normal datastore-based swapfiles will be used.

FIGURE 11.32 As soon as a data-store has been enabled for Host Cache, it is filled with pre-allocated files.

Throughout this chapter, we've shown you how to use reservations, shares, and limits to modify the resource allocation and resource utilization behaviors of VMware vSphere. In the next chapter, we'll show you some additional tools for balancing resource utilization across groups of servers.

The Bottom Line

Manage virtual machine memory allocation. In almost every virtualized datacenter, memory is the resource that typically comes under contention first. Most organizations run out of memory on their VMware ESXi hosts before other resources become constrained. Fortunately, VMware vSphere offers advanced memory-management technologies as well as extensive controls for managing the allocation of memory and utilization of memory by VMs.

Master It To guarantee certain levels of performance, your IT director believes that all VMs must be configured with at least 8 GB of RAM. However, you know that many of your applications rarely use this much memory. What might be an acceptable compromise to help ensure performance?

Master It You are configuring a brand-new large-scale VDI environment but you're worried that the cluster hosts won't have enough RAM to handle the expected load. Which advanced memory management technique will ensure that your virtual desktops have enough RAM without having to use the swapfile?

Manage CPU utilization. In a VMware vSphere environment, the ESXi hosts control VM access to physical CPUs. To effectively manage and scale VMware vSphere, administrators must understand how to allocate CPU resources to VMs, including how to use reservations, limits, and shares. Reservations provide guarantees to resources, limits provide a cap on resource usage, and shares help adjust the allocation of resources in a constrained environment.

Master It A fellow VMware administrator is a bit concerned about the use of CPU reservations. She is worried that using CPU reservations will “strand” CPU resources, preventing those reserved but unused resources from being used by other VMs. Are this administrator's concerns well founded?

Create and manage resource pools. Managing resource allocation and usage for large numbers of VMs creates too much administrative overhead. Resource pools provide a mechanism for administrators to apply resource allocation policies to groups of VMs all at the same time. Resource pools use reservations, limits, and shares to control and modify resource allocation behavior, but only for memory and CPU.

Master It Your company runs both test/development workloads and production workloads on the same hardware. How can you help ensure that test/development workloads do not consume too many resources and impact the performance of production workloads?

Control network and storage I/O utilization. Memory, CPU, network I/O, and storage I/O make up the four major resource types that VMware vSphere administrators must effectively manage in order to have an efficient virtualized datacenter. By applying controls to network I/O and storage I/O, administrators can help ensure consistent performance, meet service-level objectives, and prevent one workload from unnecessarily consuming resources at the expense of other workloads.

Master It Name two limitations of Network I/O Control.

Master It What are the requirements for using Storage I/O Control?

Utilize Flash Storage Flash storage is more popular than ever, and vSphere 5.5 has introduced the new vFlash Cache feature to sit alongside the Swap to Host Cache feature. This new resource type benefits environments that need maximum performance.

Master It You have a VM that has a large I/O requirement. Which flash feature should you configure and why?