In this chapter, we will take the point introduced in Chapter 1, VM – It Is Not What You Think!, further. We will explain why the software-defined data center (SDDC) is much more than a virtualized data center.
We will cover the following topics:
- What exactly is a software-defined data center?
- SDDCs and the cloud.
- A comparison between a physical and a virtual data center.
In Chapter 1, VM – It Is Not What You Think!, we covered how a VM differs drastically from a physical server. Now, let's take a look at the big picture, which is at the data-center level. A data center consists of three major functions—compute, network, and storage. Security is not a function on its own, but a key property that each function has to deliver. We use the term "compute" to represent processing power, namely, CPU and memory. In today's data centers, compute is also used when referencing converged infrastructure, where the server and storage have physically converged into one box. The industry term for this is Hyper-Converged Infrastructure (HCI). You will see later in the book that this convergence impacts how you architect and operate an SDDC.
VMware has moved to virtualizing the network and storage functions as well, resulting in a data center that is fully virtualized and thus defined in the software. The software is the data center. This has resulted in the term "SDDC". This book will make extensive comparisons with the physical data center. For ease of reference, let's call the physical data center the hardware-defined data center (HDDC).
In SDDC, we no longer define the architecture in the physical layer. The physical layer is just there to provide resources. These resources are not aware of one another. The stickiness is reduced and they become a commodity. In many cases, the hardware can even be replaced without incurring downtime on the VMs running on top.
The next diagram shows one possibility of a data center defined in software. I have drawn the diagram to state a point, so don't take this as the best practice for SDDC architecture. In the diagram, there are many virtual data centers (I have drawn three due to space constraints). Each virtual data center has its own set of virtual infrastructure (server, storage, network, and security). They are independent of one another.
A virtual data center is no longer contained in a single building bound by a physical boundary. Although bandwidth and latency are still limiting factors in 2016, the main thing here is that you can architect your physical data centers as one or more logical data centers. You should be able to, with just a few clicks in VMware Site Recovery Manager (SRM), automatically move thousands of servers from data center 1 to data center 2; alternatively, you can perform Disaster Recovery (DR) from four branch sites to a common HQ data center.
An example SDDC
In our example, the virtual data centers run on top of two physical data centers. Large enterprises will probably have more than this (whether it is outsourced or not is a different matter). The two physical data centers are completely independent. Their hardware is not dependent on each other:
- The compute function: There is no stretched cluster between two physical sites. Each site has its own vCenter. There is no need to protect vCenter with DR.
- The network function: There is no stretched VLAN between two physical sites. You do not have to worry about a spanning tree or broadcast storm hitting multiple data centers. The physical sites can even be on different networks. Site 1 might be on the
10.10.x.xnetwork, while site 2 might be on20.20.x.x. - The storage function: There is no array-based replication. Replication can be done independently using a storage protocol (FC, iSCSI, or NFS) and Virtual Machine Disk (VMDK) type (thick or thin). vSphere has built-in host-based replication via TCP/IP, simply named vSphere Replication. It can replicate individual VMs, and it provides finer granularity than replication based on a Logical Unit Number (LUN) . You might decide to keep the same storage vendor and protocol, but that's your choice, not something forced upon you.
I have drawn two vendors for each layer to show you that the hardware does not define the architecture. It is there to support the function of that layer (for example, compute). So, you can have 10 vSphere clusters, of which three could be using Vendor 1 and seven could be using Vendor 2.
We are taking the shared-nothing architecture approach. This is a good thing, because you can contain the failure domain. Ivan Pepelnjak, an authority on data center networking architecture, stated the following:
"Interconnected things tend to fail at the same time."
You can find more details at http://blog.ipspace.net/2012/10/if-something-can-fail-it-will.html. While you are on the site, check out http://blog.ipspace.net/2013/02/hot-and-cold-vm-mobility.html.
In 2016, cloud computing is set to rise even further, with adoption accelerating. The SDDC fits better in the cloud strategy than HDDC, as it's not bound by the rigidity of physical hardware. You can certainly incorporate a public cloud in your SDDC architecture. Your cloud extends from private to public, hence creating a hybrid cloud.
You have three options when it comes to the deployment of your hybrid cloud.
The first choice is that all data centers are on-premises. They can be on your premises or on the managed hosting company's.
The second choice is a hybrid of on-premises and off-premises data centers. The off-premises data centers do not have to be pure cloud ones. They can be managed vSphere environments, where you are not paying for VMware licenses. There are at least three sub-models for off-premises VMware, as follows:
- Pure cloud: This is a multi-tenant, shared environment. You pay per VM per month. You own neither VMware licenses nor any hardware.
- Managed VMware (shared): You have a dedicated vSphere cluster, but you don't manage vSphere. You do not dictate the version of vCenter. You may or may not have your own LUN. You pay per cluster per month, and it gives you unlimited VMs. You own no VMware licenses nor any hardware.
- Managed VMware (dedicated): You have a dedicated vCenter. You may or may not manage it. You can dictate your vCenter version, which can be useful since, as you know, some features need the latest version of vCenter. You may own VMware licenses and some hardware.
The third choice is to have both off-premises:
- Just like the second option, you have sub-choices here. You do not have to choose from the same provider. You can have two VMware cloud partners since both use the same architecture, providing you with compatibility.
- You may choose the same VMware cloud partner if you want to leverage their global presence and private network. There are customers who do not want their data going over a public network, even though it may be encrypted.
We've covered SDDCs a fair bit. What do they look like? For those who are not familiar with VMware products, let's take a quick tour.
The next screenshot shows what vCenter looks like in vSphere, the foundation of vCloud Suite:
The vCenter 5.5 overall UI with the Summary tab
I will zoom in to a part of the screenshot as it's rather small. The left-hand part of the screenshot, shown next, shows that there are three vCenter servers, and I've expanded each of them to show their data centers, clusters, hosts, and VMs:
Let's take a closer look at the following screenshot to better understand what is displayed.
The left part of the screenshot, shows that there are three vCenter servers, with their Data Centers, Clusters, Hosts, and VMs.
From here, we can tell that we no longer need another inventory management software, as we can see all objects, their configurations, and how they relate to one another. It is clear from here how many data centers, clusters, ESXi hosts, and VMs we have.
We also get more than static configuration information. Can you see the live or dynamic information is that presented here?
We are able to see the current state of each component, such as the powered state (on/off) or object status (in error, warning, or clear). This is not the type of information you get from CMDBs or inventory management systems.
vCenter 5.5 – the Summary tab
You will notice from the preceding screenshot that we get warnings and alerts, so this is a live environment. We also get information on the capacity and health. In the upper-right corner of the screen, you can see the data center's CPU, MEMORY, STORAGE capacity, and their usage. In the vSphere Replication box, you can see the VM's replication status. For example, you can see that it has 7 outgoing replications and 3 incoming replications. In the middle of the screen, you can see Health State, which, by the way, comes from vRealize Operations. In the Infrastructure Navigator box, you get to see which applications are running, such as the Application Server and Database Server. This information also comes from vRealize Operations. So, many management functions are provided out of the box. These functions are an integral part of vCloud Suite.
Many of us virtualization engineers see a cluster as the smallest logical building block in vSphere. We treat it as one physical computer. We perform capacity management at the cluster level and not at the host level. This is because a VM moves around within a cluster with DRS and storage DRS. In a virtual data center, we think in terms of a cluster and not a server.
Let's take a look at the cluster called SDDC-DR-Workload-Cluster, shown in the next screenshot. We can tell that it has 3 Hosts, 24 processors (that's cores, not sockets or threads), and almost 140 GB of RAM (about 4 GB is used by the three instances of vmkernel). We can also tell that it has Enhanced vMotion Compatibility (EVC) mode enabled, and it is based on the Intel® "Nehalem" Generation. This means we can add an ESXi host running a newer Intel processor (for example, Skylake) live inside the cluster and perform vMotion across the CPU generations.
vSphere 5.5 Cluster – Summary tab
In the top-right corner, we can see the capacity used, just like we can see it at the vCenter level. In a sense, we can drill down from the vCenter level to the cluster level.
We can also see that vSphere High Availability (HA) and Distributed Resource Scheduler (DRS) are turned on. DRS is set to fully automated, which is recommended as you do not want to manually manage the ESXi hosts one by one. There are whole books on vSphere Cluster, as there are many settings for this feature. My favorite is by Duncan Epping and Frank Denneman, which is available at http://www.yellow-bricks.com/my-bookstore/.
The ramification of this is that the data center management software needs to understand vSphere well. It has to keep up with the enhancements in vSphere and vCloud Suite. A case in point: vSphere 5.5, in the update 1 release, added Virtual SAN, a software-defined storage integrated into vSphere. Virtual SAN has been updated several times, demonstrating the rapid changes in SDDC.
Notice Health State. Again, this information comes from vRealize Operations. If you click on it, it will take you to a more detailed page, showing charts. If you drill down further, it will take you to vRealize Operations.
Now look at the the Infrastructure Navigator area. It is useful to know which applications are running on your cluster. For example, if you have a dedicated cluster for Microsoft SQL Server (as you want to optimize the license) and you see SQL in another cluster (which is not supposed to run the database), you know you need to move the VM. This visibility is important because as an infrastructure team, you typically do not have access to go inside the VM. You do not know what's running on top of Windows or Linux.
We covered compute. Let's move on to network. The next screenshot shows a distributed virtual switch. As you can see, the distributed switch is an object at the data-center level. Therefore, it extends across clusters. In some environments, this can result in a very large virtual switch with more than 1,000 ports. In the physical world, this would be a huge switch indeed!
A VM is connected to either a standard switch or a distributed switch. It is not directly connected to the physical network interface card (NIC) in your ESXi host. The ESXi host's physical NICs become the virtual switch's uplinks instead, and I recommend you have at least two 10 Gigabit Ethernet ports. This means that the traditional Top-Of-Rack (TOR) switch has been entirely virtualized. It runs completely as software, and the following screenshot is where you create, define, and manage it. This means that the management software needs to understand the distributed vSwitch and its features. As you will see later, vRealize Operations understands virtual switches and treats networking as a first-class object.
A vSphere 5.5 distributed switch
The previous screenshot also shows that the virtual switch has six port groups and two uplinks. Let's drill down to one of the port groups, as shown in the next screenshot. A port group is a feature that is optional in physical switches, but mandatory in virtual ones. It lets you group a number of switch ports and give them a common property. You can also set policies. As shown in the Policies box, there are many properties that you can set. A port group is essential to managing all the ports connected to a switch.
A vSphere 5.5 distributed-switch port group
In the top-right corner, you see the capacity information, so you know how many ports you configured and how many ports are being used. This is where virtual networking is different from virtual compute and virtual storage. For compute and storage, you need to have the underlying physical resources to back them up. You cannot have a VM with a 32-core vCPU if the underlying ESXi has less than 32 physical threads. vSphere will let you create it but not power it on. Virtual networks are different. A network is an interconnection; it is not a "node" like compute and storage. It is not backed by physical ports. You can increase the number of ports to basically any number you want. The entire switch lives in memory! You power off the ESXi and the switch is no more.
In the Infrastructure Navigator widget, you will again see the list of applications. Infrastructure Navigator is embedded into the vSphere Web UI, making you feel as if it's a single application, like a single pane of glass. Over the past several releases of VMware products, they have been becoming one integrated suite, and this trend is set to continue.
Let's now move to storage. The next screenshot shows a vSphere 5.5 datastore cluster. The idea behind a datastore cluster is similar to that of a compute cluster.
A vSphere 5.5 datastore cluster
Let's use an example since it's easier to understand. Say you have a cluster of eight ESXi hosts, with each host sporting two sockets, 36 cores, and 72 threads. The entire cluster has 288 physical cores and 576 physical threads. In this cluster, you run 200 VMs, giving you a 25:1 consolidation ratio. Based on the guideline that Intel Hyper-Threading gives around a 25-percent performance boost, you can use 360 cores as the max thread count. This gives you around 1.8 cores per VM, which is possible assuming your VMs have two to four vCPUs and around 50-percent utilization. These 200 VMs are stored in four datastores, that is, around 50 VMs per datastore.
With the compute node, you need not worry about where a VM is running in that cluster. When you provision a new VM, you do not necessarily need to specify which host will run it. You let DRS decide. As the workload goes up and down, you do not want to manage the placement on an individual ESXi host for 200 VMs. You let DRS do the load balancing, and it will use vMotion automatically. You treat the entire cluster as if it were a single giant box.
With the storage node, you can do the same thing. When you provision a new VM, you do not specify a datastore for it. If you want to specify it manually, you need to check which datastore has the most amount of space and the least amount of IOPS. The first piece of information is quite easy to check, but the second one is not!
This is the first benefit of a datastore cluster. It picks a datastore based on both capacity and performance. The second benefit is based on the ongoing operation. As time passes, the VM grows at different rates in terms of both capacity and IOPS. Just like the well-known DRS, storage DRS monitors this and makes recommendations for you. The major difference here is the amount of data to be migrated:
- In vMotion, we normally migrate somewhere between 1 GB to 10 GB of RAM, as the kernel only copies the used RAM (and not the configured RAM).
- In storage vMotion, we potentially copy 100 GB of data. This takes a lot longer and hence has a greater performance impact. As such, storage DRS should be performed a lot less frequently, perhaps once a month.
A datastore cluster helps in capacity management, as you basically treat all the datastores as one. You can easily check key information about the datastore cluster, such as the number of VMs, total storage, capacity used, and largest amount of free space you have.
As usual, vRealize Operations, via its Infrastructure Navigator component, provides information about which applications are running in a datastore cluster. This is handy information in a large environment, where you have specific datastores for specific applications.
We covered all the three elements—compute, storage, and network. How are they related? The next screenshot shows the relationship between the key objects managed by vCenter.
It's handy information in a small environment. If you have a large environment, maps such as the one shown in the next screenshot really become much more complex! In this map, we only have three ESXi hosts and seven datastores, and we already have to hide some relationships. Notice that I did not select the Host to VM and VM to Datastore relationship options, because the diagram got complicated when I did. These can be added to the maps easily to give an even more detailed view of the datacenter.
The point of sharing the screenshot is to tell you that you really have your data center in software with the following characteristics:
- You have your VM as the consumer. You can show both powered-on and powered-off VMs.
- You have your compute (ESXi), network (port group), and storage (datastore) as the provider. You can show the relationships between your compute, network, and storage.
- You have the information about the network, storage, and compute your VM is connected to.
Think about it: how difficult would it be to have this type of relationship mapped in a large physical data center? We have all heard comments from customers that they do not know exactly how many servers they have, which network they are connected to, and which applications run on a box. A powered-off server is even harder to find! Even if you can implement a datacenter management system, which can give you the map, one or two years later, you cannot be sure that the map is up to date. The management system has to be embedded into the platform. In fact, it's the only point of entry to the virtual platform. It cannot be a separate, detached system.
vSphere Maps
The last point I'd like to bring up is that an SDDC is a world in itself. It's not simply your data center that has been virtualized. Look at the following table. It lists some of the objects in vSphere. We have not included NSX, Virtual SAN, or vRealize Suite objects here. These objects do not have their physical equivalents. If they do, they have different properties, generate different events, and are measured by different counters. Plus, all these objects have some relationship with one another. You need to look at vCloud Suite in its entirety to understand it well.
vSphere objects and their relationships
The downside of an SDDC is that the upgrade of such a "giant machine" is a new project for IT. It has to be planned and implemented carefully because it is as good as upgrading the data center while the servers, storage, and network are all still running. Using a physical-world analogy, it's like renovating your home while living in it.