4

Design of vSAN Storage Policies

In the previous chapter, you learned about the design of vCenter Server for the VxRail 7.x system, including an internal vCenter Server instance with an external DNS and an internal DNS, an external vCenter Server instance with an external DNS, and an internal vCenter Server instance with a customer-supplied virtual distributed switch. Now, you understand the advantages and disadvantages of each kind of VxRail deployment.

VMware vSAN is a key component of the VxRail system. It can automatically build up a local vSAN datastore across the VxRail cluster during VxRail initialization. VMware vSAN can provide different features based on your choice of vSAN license editions on the VxRail cluster – for example, Standard edition, Advanced edition, and Enterprise edition. You can non-disruptively expand the capacity and performance by adding nodes to a VxRail cluster (scaling out) or increase the capacity by adding disks to each node (scaling up). The vSAN storage policy is a good tool that can deliver different storage requirements on the VxRail 7.x system and it can also deliver the data recovery services for each virtual machine (VM).

This chapter will provide an overview of VMware vSAN on the VxRail 7.x system, including the vSAN architecture, vSAN components, vSAN disk groups, vSAN storage policies, and vSAN HCI Mesh.

This chapter includes the following topics:

• An overview of VMware vSAN on VxRail
• An overview of vSAN objects and components
• VMware vSAN storage policies
• VMware vSAN HCI Mesh

An overview of VMware vSAN on VxRail

VMware vSAN is a Software-Defined Storage (SDS) solution that is used in collaboration with the VMware vSphere hypervisor. The VMware vSAN solution can provide shared storage for VMs. In Figure 4.1, vSAN is a vSphere cluster feature that virtualizes the local physical storage in the pool of the Hard Disk Drive (HDD) and Solid-State Drive (SSD) on the ESXi hosts in a cluster, building them into a unified data store. Each ESXi host requires a minimum disk group that includes a SSD and HDD. vSAN can provide enterprise storage that is powerful, flexible, and user-friendly. This solution can be deployed in different environments – for example, the core, the edge, and the cloud. When you build the VxRail cluster on day one, the vSAN datastore is created automatically.

The vSAN datastore includes the following characteristics:

• All nodes in the VxRail cluster must be connected to a vSAN Layer 2 or Layer 3 network. It supports a maximum of a 1 millisecond (ms) Round-Trip Time (RTT) for VxRail standard clusters between all nodes in the cluster.
• Using a vSphere Distributed Switch (vDS) and Network IO Control (NIOC) is recommended.
• VMware vSAN is a single vSAN datastore accessible to all nodes in the VxRail cluster.
• You can create different service levels for each VM or Virtual Machine Disk (VMDK) per VM using vSAN storage policies.
• vSAN supports Hybrid mode and All-Flash mode on the VxRail cluster.
• The Hybrid and All-Flash disk groups cannot mix in the same VxRail cluster.
• vSAN does not support Raw Device Mapping (RDM).
• A vSAN datastore can be supported for mounting by a maximum of five vSAN client clusters.
• vSAN Stretched Clusters and two-node configurations are not supported in combination with HCI Mesh.

In Figure 4.1, there is a VxRail cluster with four nodes in this environment and one disk group installed on each VxRail, including one SSD for the cache tier and two capacity disks for the capacity tier. The vSAN datastore’s capacity depends on the number of capacity disks per VxRail node and the number of VxRail nodes in the cluster. For example, if the VxRail cluster includes four nodes and each node installs two 2 TB capacity disks, the approximate storage capacity is 16 TB (4 x 4 TB). vSAN supports two configurations – Hybrid and All-Flash mode:

Figure 4.1 – The architecture of VMware vSAN on the VxRail 7.x system

Important Note

In the initial deployment, the first three VxRail nodes in a cluster must be identical models. VxRail Hybrid and All-Flash nodes cannot mix in a VxRail cluster. VxRail S Series does not support the All-Flash model.

In Figure 4.2, you can configure a disk group with either Hybrid or All-Flash configurations in the VxRail cluster. Each VxRail model can support a maximum of four disk groups, each of which must have one flash device and one or more capacity devices (up to five devices). The number of supported disk groups depends on the VxRail model; for example, VxRail E Series supports a maximum of two disk groups, or VxRail E Series supports a maximum of four disk groups. In the Hybrid disk group, 70% of the available cache is used for frequently-read data, and 30% of the available cache is used for write buffering. In the All-Flash disk group, the flash devices are used in a two-tier format for caching and capacity, and 100% of the available cache is used for write buffering:

Figure 4.2 – The difference between the Hybrid disk group and the All-Flash disk group

Important Note

The SSD and NVMe devices support the cache tier on the VxRail cluster. The SSD, NVMe, SAS, SATA, and vSAS devices support the capacity tier on the VxRail cluster.

In a VMware vSAN 7.x cluster, you can also configure a disk group with either Hybrid or All-Flash configurations manually, but each vSAN node can support a maximum of five disk groups, each of which must have one flash device and one or more capacity devices (up to seven devices). Compared to the VxRail cluster, the maximum disk group is different from the VMware vSAN cluster. A VxRail node can only support up to four disk groups and the capacity tier supports up to six capacity devices. In each VxRail model, the disk group configuration rules are specific to vSAN configurations. For the disk group upgrades on each VxRail Series, we will discuss the details in Chapter 5, Design of Cluster Expansion.

The next section discusses the overview of vSAN objects and components.

An overview of vSAN objects and components

When you create VMs in a VxRail cluster, you must specify the vSAN storage policy (we will discuss the details in the next section) for each VM. It will automatically create each object’s vSAN component (that is, the VMDK) and allocate it to each VxRail node when you apply the vSAN storage policy to the VM. vSAN manages and stores data called objects. Each object on the vSAN datastore includes data, metadata, and a unique ID. The most common vSAN objects are the VDMKs. In traditional storage, you need to define the storage pools, the RAID level protection, and the hot spare. Then, you can create VMs or build your application into this storage. All storage configurations must be completed on day one. The vSAN architecture is different from traditional storage. vSAN objects are made of components determined by vSAN storage policies. You can change or update the Failure Tolerance Method (FTM) and Failures To Tolerate (FTT) of the VM at any time if the storage resource requirement is fulfilled in the VxRail cluster. In an example in Figure 4.3, there is a VM allocated to the VxRail cluster with four nodes and it is assigned the vSAN storage policy with FTM = RAID-1 and FTT = 1 in this VM. The vSAN object (the VMDK) included has three components, two replicas, and one witness. Each component is allocated to each VxRail node. Each vSAN object has the following state:

• Active: The vSAN object is accessible.
• Absent: The vSAN object is inaccessible, but no explicit error codes were detected.
• Degraded: The vSAN object is inaccessible with explicit error codes detected.
• Active-Stale: The sequence numbers are not consistent.

The level of FTT defines the level of resilience. The FTM defines the data layout approach. In this example, the fault tolerance method is set to RAID-1 mirroring, and FTT are set to 1:

Figure 4.3 – The sample configuration of vSAN objects

Now, we will discuss the architecture of different FTMs in the next section, including RAID-1, RAID-5, and RAID-6.

Failures to tolerate with RAID-1 mirroring

This section will discuss an overview of FTMs with RAID-1 on the VxRail cluster. In Figure 4.4, there is a VxRail cluster with five nodes (VxRail Node A, B, C, D, and E) in this scenario. And one VM with one 100 GB VMDK configured with RAID-1 and FTT = 1. There are three vSAN components: two Replica components and one Witness component. Each vSAN component is allocated to each node. Applying this vSAN storage policy to the VM will mirror the Replica component (the 100 GB VMDK) from VxRail Node A into VxRail Node B. The Witness component is used to determine a quorum:

Figure 4.4 – The diagram of vSAN objects in RAID-1 and FTT = 1

If you choose the vSAN policy with RAID-1 and FTT = 1, you need to consider the following requirements and limitations:

• The minimum number of nodes required is three in the same VxRail cluster – four nodes are the recommended configuration.
• It supports the VxRail Hybrid and All-Flash configuration.
• It supports vSAN Stretched Clusters.
• It supports the VMware vSAN standard, advanced, enterprise, and enterprise plus editions.
• It allows one node failure in the VxRail cluster if FTT is set to one – four nodes is the recommended configuration.
• It allows two-node failure in the VxRail cluster if FTT is set to two – six nodes is the recommended configuration.
• It allows three-node failure in the VxRail cluster if FTT is set to three; eight nodes is the recommended configuration.

According to this scenario, you have learned the architecture of FTMs with RAID-1 on the VxRail cluster. We will discuss the RAID-5 erasure coding scenario in the next section.

Failures to tolerate with RAID-5 erasure coding

This section will discuss the overview of FTMs with RAID-5 on the VxRail cluster. In Figure 4.5, there is a VxRail cluster with five nodes (VxRail Node A, B, C, D, and E), and one VM with one 100 GB VMDK configured with RAID-5 and FTT = 1. There are four vSAN components, three replicas, and one Parity component. Each vSAN component is allocated to each node when you apply this vSAN storage policy to the VM – after then, it separates into three replica components (100 GB VMDK) and one parity of the VM. Each component is stored across four hosts (VxRail Node A, B, C, and D):

Figure 4.5 – The diagram of vSAN objects in RAID-5 and FTT = 1

If you choose the vSAN policy with RAID-5 and FTT = 1, you need to consider the following requirements and limitations:

• The minimum number of nodes is four in the same VxRail cluster, and five nodes are the recommended configuration.
• It only supports the VxRail All-Flash configuration.
• It supports vSAN Stretched Clusters.
• It supports the VMware vSAN advanced, enterprise, and enterprise plus editions.
• It allows one node failure in the VxRail cluster if FTT is set to one – five nodes are the recommended configuration.
• A space reduction of 30% is guaranteed with RAID-5.

With this scenario, you have learned about the architecture of FTMs with RAID-5 on the VxRail cluster. We will discuss the RAID-6 erasure coding scenario in the next section.

Failures to tolerate with RAID-6 erasure coding

This section will discuss the overview of FTMs with RAID-6 on the VxRail cluster. In Figure 4.6, there is a VxRail cluster with seven nodes (VxRail Node A, B, C, D, E, F, and G) and one VM with one 100 GB VMDK configured with RAID-6 and FTT = 2. There are six vSAN components, four replicas, and two Parity components. Each vSAN component is allocated to each node when you apply this vSAN storage policy to the VM – then it separates into four replica components (100 GB VMDK) and two Parity components. Each component is stored across six hosts (VxRail Node A, B, C, D, E, and F).

Figure 4.6 – The diagram of vSAN objects in RAID-6 and FTT = 2

If you choose the vSAN policy with RAID-6 and FTT = 2, you need to consider the following requirements and limitations:

• The minimum number of nodes is six in the same VxRail cluster, and seven nodes are the recommended configuration.
• It only supports the VxRail All-Flash configuration.
• It supports vSAN Stretched Clusters.
• It supports the VMware vSAN advanced, enterprise, and enterprise plus editions.
• It allows two-node failure in the VxRail cluster if FTT is set to two; seven nodes are the recommended configuration.
• It guarantees a space reduction of 50% with RAID-6.

Important Note

Both RAID-5 and RAID-6 are only supported by the All-Flash disk group in the VxRail cluster. And the vSAN license edition must be Advanced, Enterprise, or Enterprise Plus.

With this scenario, you have learned about the architecture of FTMs with RAID-6 on a VxRail cluster. We will discuss vSAN storage policies in the next section.

VMware vSAN storage policies

The vSAN storage policy is used to define the VM storage requirements of performance and availability. After vSAN storage is applied to a VM, a number of vSAN component replicas and copies are created automatically based on the vSAN storage policy. vSAN supports the following RAID level protection. Table 4.1 includes a description of each supported RAID level on vSAN:

Table 4.1 – Descriptions of FTMs on the vSAN storage policies

When you create the vSAN storage policy based on your storage requirements for a VM and its VMDKs, you can define the different storage rules, including encryption services, space efficiency, and storage tiers.

In Figure 4.7, you can go to the Storage rules tab in the Create VM Storage Policy wizard if you want to enable storage rules on your vSAN storage policies. You can select the following storage rules:

• Encryption services is used to define the encryption rules for the VM:
  • Data-At-Rest encryption: Enables encryption on the VM
  • No encryption: Does not enable encryption on the VM
  • No preference: Makes the VM compatible with both Data-At-Rest encryption and No encryption
• Space efficiency is used to save storage space for the VM:
  • Deduplication and compression: Enables both deduplication and compression on VMs. Both features are only available on the vSAN All-Flash cluster.
  • Compression only: Enables compression on the VM. This feature is only available on the vSAN All-Flash cluster.
  • No space efficiency: Does not enable the space efficiency feature on the VM.
  • No preference: Makes the VM compatible with all of the preceding options.
• Storage tier is used to define the storage tier on the VM:
  • All Flash: This is supported by the vSAN All-Flash environment.
  • Hybrid: This is supported by the vSAN Hybrid environment.
  • No preference: Makes the VM compatible with All-Flash and Hybrid environments.

In the Storage rules tab, you can define all of these features in the Create VM Storage Policy wizard:

Figure 4.7 – The storage rules in the Create VM Storage Policy wizard

You can also define advanced policy rules for the VM. You can choose from the following options:

• Number of disk stripes per object: This defines the number of stripes for each vSAN object (or VMDK). The default setting is one; if you change it to two, the vSAN object is striped across two physical disks.
• IOPS limit for object: This defines the IOPS for the drive.
• Object space reservation: This defines the space type of the vSAN object, Thick provisioning or Thin provisioning.
• Flash read cache reservation: This defines the flash read cache for each vSAN object.
• Disable object checksum: If you enable this option, the storage object will not calculate the checksum information.
• Force provisioning: If you enable this option, the vSAN object can be provisioned if the vSAN storage policy cannot satisfy the resources available in the cluster.

In the Advanced Policy Rules tab, you can define all of the preceding features in the Create VM Storage Policy wizard, as shown in Figure 4.8:

Figure 4.8 – The advanced policy rules in the Create VM Storage Policy wizard

So far, we have had an overview of the vSAN storage policy. We will discuss a scenario involving vSAN storage policies on the VxRail cluster in the next section.

vSAN storage policy scenario

This section will discuss the vSAN component’s behavior on the VxRail cluster after updating the different vSAN storage policies, including RAID-1 and RAID-5. In the following scenarios, one disk group is required per node. In Figure 4.9, there is a VxRail cluster with five nodes and a VM with a 100 GB VMDK running on this cluster. The vSAN storage policy configures the following storage rules and applies them to the VM. There are five vSAN components, four Replica components (50 GB), and one Witness component in this scenario. In VxRail Node A, two components are allocated in Disk 1 and Disk 2. In VxRail Node B, two components are allocated in Disk 1 and Disk 2. The Witness component allocates the disk on VxRail Node C. The vSAN storage policy includes the following parameters:

• The FTM is RAID-1.
• FTT is 1.
• Stripe Width (SW) is 2.
  

Figure 4.9 – The vSAN storage policy with FTM = RAID-1, FTT = 1, and SW = 2

Important Note

If the vSAN objects are greater than 255 GB in size, vSAN automatically divides them into multiple components.

The following is the summary of the vSAN components if you apply the vSAN storage policy with FTM = RAID-1, FTT = 1, and SW = 2:

• The five vSAN components are created in the VxRail cluster.
• The size of each vSAN component is 50 GB and the witness component is not included.
• One node can fail in the VxRail cluster.
• Each vSAN component is written onto each disk.

According to the scenario in Figure 4.9, you now understand the behavior of the vSAN components. Now, we will update the SW variable to 3 in the vSAN storage policy:

Figure 4.10 – The vSAN storage policy with FTM = RAID-1, FTT = 1, and SW = 3

The following is the summary of the vSAN components if you apply the vSAN storage policy with FTM = RAID-1, FTT = 1, and SW = 3:

• The seven vSAN components are created in the VxRail cluster.
• The size of each vSAN component is around 33 GB and the Witness component is not included.
• One node can fail in the VxRail cluster.
• Each vSAN component is written onto each disk.

According to the scenario in Figure 4.10, you now understand the number of vSAN components will be changed to seven because the SW parameter changes to 3 in the vSAN storage policy. Now, we will discuss another scenario.

In Figure 4.11, there are eight vSAN components if you FTM = RAID-5 and FTT = 2, including six Replica components (16.5 GB) and two Parity components (16.5 GB):

Figure 4.11 – The vSAN storage policy with FTM = RAID-5, FTT = 1, and SW = 2

The following is the summary of the vSAN components if you apply the vSAN storage policy with FTM = RAID-5, FTT = 1, and SW = 2:

• The eight vSAN components are created in the VxRail cluster.
• The size of each vSAN component is around 16.5 GB and there is no witness component in RAID-5.
• One node can fail in the VxRail cluster.
• Each vSAN component is written onto each disk.

According to the preceding scenarios, you have learned about the behavior of the vSAN components in the vSAN storage policies with RAID-1 and RAID-5.

Table 4.2 is a summary of FTT and FTMs on the vSAN storage policy:

Table 4.2 – The summary of FTMs and FTT on the vSAN storage policy

Important Note

You can refer to this link for details of the stripe width improvements in vSAN 7 update 1: https://blogs.vmware.com/virtualblocks/2021/01/21/stripe-width-improvements-in-vsan-7-u1/.

The last section will discuss the benefits of VMware vSAN HCI Mesh.

VMware vSAN HCI Mesh

VMware HCI Mesh was introduced with vSphere 7.0 Update 2 and allowed you to remotely connect to one or more VMware vSAN datastores from another vSAN cluster. In Figure 4.12, there are three VxRail clusters (one server and two clients). Two VxRail clusters (Client) are remotely connected to a vSAN datastore from the VxRail cluster (Server). vSAN HCI Mesh can be used for balancing storage resources by migrating VMs to other vSAN clusters using vSphere Storage vMotion.

Now, compute clusters can connect to the remote vSAN datastores. You can define the different storage tier vSAN storage policies (for example, RAID-1, RAID-5, deduplication, encryption, and All-Flash) and apply them to the different tier VMs in the vSAN cluster. HCI Mesh brings together multiple independent clusters for a native, cross-cluster architecture that disaggregates compute and storage resources and enables efficient utilization of storage capacity. HCI Mesh is used to share the storage resources of a VxRail cluster with other VxRail clusters or other independent clusters:

Figure 4.12 – The sample configuration of vSAN HCI Mesh on the VxRail cluster

Now, we will discuss two scenarios to learn about the benefits of VMware vSAN HCI Mesh on the VxRail cluster.

In Figure 4.13, the system administrator needs extra storage for deploying new VMs into the vSphere cluster in Remote Data Center A. But there are no storage resources available in the vSphere cluster and they cannot add other storage resources to the vSphere cluster. They can add new vSAN nodes into the existing VxRail All-Flash cluster in Headquarter Data Center and expand the storage capacity of the vSAN datastore, then remotely share the storage resources to the vSphere cluster in Remote Data Center A:

Figure 4.13 – Scenario 1 regarding the benefits of VMware vSAN HCI Mesh

In this scenario, the system administrator can easily share the storage resources in the vSphere cluster, and service interruption is not required for this configuration.

In Figure 4.14, Remote Data Center A needs to shut down due to maintenance. How do we minimize the service interruption of the VMs running in the vSphere cluster? The system administrator can migrate the VMs with VMware vMotion into the VxRail cluster in Remote Data Center B:

Figure 4.14 – Scenario 2 regarding the benefits of VMware vSAN HCI Mesh

In this scenario, you can easily move the VM onto the two platforms across two data centers and service interruption is not required for this configuration.

According to these two examples, you now understand the benefits of VMware vSAN HCI Mesh and how to offload the daily operation in the virtualization environment.

Please refer to Table 4.3 for the hardware and software requirements for each scenario in Figure 4.13 and Figure 4.14:

Table 4.3 – The details of each scenario in Figures 4.13 and 4.14

To plan for vSAN HCI Mesh on the VxRail 7.x system, you can refer to the examples in Figure 4.13 and Figure 4.14.

Summary

In this chapter, you got an overview of the design of VMware vSAN on the VxRail 7.x system, including FTMs, FTT, and SWs. You now understand the relationships of vSAN objects and components on the different vSAN storage policies – for example, RAID-1 and RAID-5. Based on the different scenarios, you understand the benefits of the vSAN storage policy and how to offload the daily operations in a virtualization environment. If the storage resource requirements on a VxRail 7.x system are ready, you can update the vSAN storage policy at any time without disrupting the service of the VMs.

In the next chapter, you will learn about VxRail cluster expansion, including VxRail scale-out rules, disk group upgrades, and node upgrades.

Questions

The following are a short list of review questions to help reinforce your learning and help you identify areas that require some improvement.

1. Which components are included in a vSAN disk group on each VxRail node?
   1. A flash drive device
   2. A 10 GB Ethernet adapter
   3. VMware vSAN licenses
   4. VMware vSphere licenses
   5. HDD devices
   6. All of the above
2. Which FTT configuration is supported on the VxRail 7.x platform?
   1. No data redundancy
   2. One failure – RAID-1 (mirroring)
   3. Two failures – RAID-1 (mirroring)
   4. One failure – RAID-5 (erasure coding)
   5. Two failures – RAID-6 (erasure coding)
   6. Three failures – RAID-1 (mirroring)
   7. All of the above
3. How many vSAN components will be created automatically if the FTM is set to RAID-1 and FTT is set to 1?
   1. 1
   2. 2
   3. 3
   4. 4
   5. 5
   6. 6
4. How many vSAN components will be created automatically if the FTM is set to RAID-1 and FTT is set to 2?
   1. 1
   2. 2
   3. 3
   4. 4
   5. 5
   6. 6
5. How many vSAN components will be created automatically if the FTM is set to RAID-5 and FTT is set to 1?
   1. 1
   2. 2
   3. 3
   4. 4
   5. 5
   6. 6
6. How many vSAN components will be created automatically if the FTM is set to RAID-6 and FTT is set to 2?
   1. 1
   2. 2
   3. 3
   4. 4
   5. 5
   6. 6
7. What is the number of minimum hosts required in the vSAN cluster if the FTM is set to RAID-1 and FTT is set to 1? This policy can be satisfied by three nodes. A fourth node is required in case of failure.
   1. 3
   2. 4
   3. 5
   4. 6
   5. 7
8. What is the number of minimum hosts required in the vSAN cluster if the FTM is set to RAID-5 and FTT is set to 1? This policy can be satisfied by four nodes. The fifth node is required in case of failure.
   1. 3
   2. 4
   3. 5
   4. 6
   5. 7
9. What is the number of minimum hosts required in the vSAN cluster if the FTM is set to RAID-6 and FTT is set to 2? This policy can be satisfied by six nodes. The seventh node is required in case of failure.
   1. 3
   2. 4
   3. 5
   4. 6
   5. 7
10. Which vSAN license edition is supported if vSAN HCI Mesh is enabled on the VxRail cluster?
    1. VMware vSAN Standard edition
    2. VMware vSAN Advanced edition
    3. VMware vSAN Enterprise edition
    4. VMware vSAN Enterprise Plus edition
    5. All of the above
11. Which types of disk groups can be supported on a VxRail cluster?
    1. The All-Flash disk group only
    2. NL-SAS and SAS disk groups
    3. The Hybrid disk group only
    4. The Hybrid and All-Flash disk groups
    5. The SAS disk group only
    6. All of the above
12. Which features are available on the encryption service in the storage rules when you define the vSAN storage policy?
    1. Data-At-Rest encryption
    2. No encryption
    3. No preference
    4. Data encryption
    5. All of the above