This chapter describes our work at VMware to support HPC applications. The first section describes in detail many of the values identified by customers of using virtualization in HPC environments. The second section shows a few examples of how virtualization is deployed in an HPC environment, and the third discusses various aspects of performance, starting with an examination of some core aspects of performance, and then turning to throughput applications and performance for parallel-distributed Message Passing Interface (MPI) applications. It also includes pointers to several technical publications that will be of interest to those considering virtualizing their HPC workloads.
The majority of HPC systems are clusters, which are aggregations of compute nodes connected via some interconnect, such as Ethernet or InfiniBand (IB). Clusters can range in size from a small handful of nodes to tens of thousands of nodes. HPC clusters exist to run HPC jobs, and the placement of those jobs in the cluster is handled by a distributed resource manager (DRM). DRM is the middleware that provides the ability for HPC users to launch their HPC jobs onto an HPC cluster in a load-balanced fashion.
Users typically use command-line interfaces to specify the characteristics of the job or jobs they want to run, the DRM then enqueues those requests and schedules the jobs to run on the least loaded, appropriately configured nodes in the cluster. There are many DRMs available, both open source and commercial. Examples include Grid Engine (Univa), LSF (IBM), Torque, Portable Batch System (PBS), and Slurm. DRMs are also called batch schedulers. IB is a high-bandwidth, low-latency interconnect, commonly used in HPC environments to boost the performance of code/applications/jobs and to increase filesystem performance. IB is not Ethernet, it does not use TCP or any of the standard networking stack, and it is currently usable in a virtual environment only via VM direct path I/O (passthrough mode).
The purpose of this chapter is not to explain how virtualization works. x86 virtualization has evolved from its invention at Stanford in the late 1990s using a purely software-based approach to the current situation in which both Intel and AMD have added successively more hardware support for virtualization, such as CPU, memory, and I/O. These hardware enhancements, along with increasingly sophisticated virtualization software, have greatly improved the performance for an ever-growing number of workloads. This is an important point to make because HPC people have often heard that the performance of HPC applications is very poor when they get virtualized with some cases but run extremely well, even very close to native performance, in most of the cases. Massive consolidation isn't appropriate for HPC environments, so the values of virtualization for HPC can be utilized elsewhere.
We will now learn the use cases for virtualization in HPC that have been identified by customers and through our own research. Because HPC includes such a wide variety of workloads and environments, some of these will resonate more with specific customers than others. HPC clusters host a single, standard OS and application stack across all hosts as uniformity enables us to schedule jobs easily by limiting the options in these environments for different use cases, such as multiple user groups needing to be served from a single shared resource. Because these traditional clusters can't satisfy the needs of multiple groups, they encourage the creation of specialized islands of compute scattered across an organization, which is inefficient and expensive.