Cluster quorum modes

A cluster quorum mode determines how many nodes and witnesses must fail before the cluster is in a failed state. Nodes are servers that participate in the cluster. Witnesses can be stored on shared storage, on file shares, in Windows Server 2019, and even on a USB drive attached to a network switch; shared storage is the preferred method. This recommendation might eventually change, however, as continuously available file shares are more widely adopted.

Microsoft recommends that you configure a cluster so that an odd number of total votes is spread across member nodes and the witness. This limits the chance of a tie during a quorum vote.

There are four cluster quorum modes:

• Node Majority. This cluster quorum mode is recommended for clusters that have an odd number of nodes. When this quorum type is set, the cluster retains quorum when the number of available nodes exceeds the number of failed nodes. For example, if a cluster has five nodes and three are available, quorum is retained.

• Node And Disk Majority. This cluster quorum mode is recommended when the cluster has an even number of nodes. A disk witness hosted on a shared storage disk, for example iSCSI or Fibre Channel, that is accessible to cluster nodes has a vote when determining quorum, as do the quorum nodes. The cluster retains quorum as long as the majority of voting entities remain online. For example, if you have a four-node cluster and a witness disk, a combination of three of those entities needs to remain online for the cluster to retain quorum. The cluster retains quorum if three nodes are online or if two nodes and the witness disk are online.

• Node And File Share Majority. This configuration is similar to the Node And Disk Majority configuration, but the quorum is stored on a network share rather than on a shared-storage disk. It is suitable for similar configurations to Node And Disk Majority. This method is not as reliable as Node And Disk Majority because file shares generally do not have the redundancy features of shared storage.

• No Majority: Disk Only. This model can be used with clusters that have an odd number of nodes. It is only recommended for testing environments because the disk hosting the witness functions as a single point of failure. When you choose this model, as long as the disk hosting the witness and one node remain available, the cluster retains quorum. If the disk hosting the witness fails, quorum is lost, even if all the other nodes are available.

When you create a cluster, the cluster quorum is automatically configured for you. You might want to alter the quorum mode, however, if you change the number of nodes in your cluster. For example, you might want to alter the quorum mode if you change from a four-node to a five-node cluster. When you change the cluster quorum configuration, the Failover Cluster Manager provides you with a recommended configuration, but you can choose to override that configuration if you want.

You can also perform advanced quorum configuration to specify what nodes can participate in the quorum vote, which you can set on the Select Voting Configuration page of the Configure Cluster Quorum Wizard. When you do this, only the selected nodes’ votes are used to calculate quorum. Also, it’s possible that fewer nodes would need to fail to cause a cluster to fail than would otherwise be the case if all nodes participated in the quorum vote. This can be useful when configuring how multisite clusters calculate quorum when the connection between sites fails.

Cluster storage and cluster shared volumes

Almost all failover cluster scenarios require access to some form of shared storage. Windows Server failover clusters can use SAS, iSCSI, or Fibre Channel for Shared Storage. With the inclusion of the iSCSI Target software in Windows Server, iSCSI is one of the simplest and cheapest of these technologies to implement.

You should configure disks used for failover clustering as follows:

• Volumes should be formatted using NTFS or ReFS.

• Use Master Boot Record (MBR) or GUID Partition Table (GPT).

• Avoid allowing different clusters access to the same storage device. This can be accomplished through LUN masking or zoning.

• Any multipath solution must be based on Microsoft Multipath I/O (MPIO).

Cluster Shared Volumes (CSV) is a technology that was introduced in Windows Server 2008 R2 that allows multiple cluster nodes to have concurrent access to a single Logical Unit Number (LUN). Prior to Windows Server 2008 R2, only the active node had access to shared storage. CSV allows you to have virtual machines on the same LUN run on different cluster nodes. CSV also has the following benefits:

• Support for scale-out file servers

• Support for BitLocker volume encryption

• SMB 3.0 support

• Integration with Storage Spaces

• Online volume scan and repair

You can enable CSV only after you create a failover cluster and you have provided the storage that will be available to the CSV.

Cluster networks

While you can create failover clusters with nodes that have a single network adapter, best practice is to have separate networks and network adapters for the following:

• A connection for cluster nodes to shared storage

• A private network for internal cluster communication

• A public network that clients use to access services hosted on the cluster

In scenarios where high availability is critical, you might have multiple redundant networks connected through several separate switches. If you have a cluster where everything is connected through one piece of network hardware, you can almost guarantee that piece of network hardware is the first thing that fails.

You can either use IPv4 or IPv6 addresses that are dynamically or statically assigned, but you should not use a mix of dynamically and statically assigned IP addresses for nodes that are members of the same cluster. If you use a mixture of dynamically and statically assigned IP addresses, the Validate A Configuration Wizard generates an error.

MPIO

If there are multiple paths to physical storage on a server, you will need to enable multipath I/O (MPIO). Enabling MPIO aggregates multiple paths to a physical disk into a single logical path for data access. Enabling MPIO improves resiliency to failure. You should enable MPIO on all nodes that will participate in the cluster. You can enable MPIO with the following PowerShell command:

Click here to view code image

Install-WindowsFeature -ComputerName Node1 -Name MultiPath-IO

Cluster Aware Updating

Cluster Aware Updating (CAU) is a feature included with Windows Server that allows you to automate the process of applying software updates to failover clusters. Cluster Aware Updating integrates with Windows Update, Windows Server Update Services (WSUS), System Center Configuration Manager, and other software update-management applications.

CAU uses the following process:

1. Obtains the update files from the source location.

2. Puts the first node into maintenance mode.

3. Moves any cluster roles off the node to other nodes in the cluster.

4. Installs software updates.

5. Restarts the cluster node, if necessary.

6. Checks for additional updates. If updates are found, it performs Steps 4 through 6 until all updates are applied.

7. Brings the node out of maintenance mode.

8. Reacquires clustered roles that were moved to other nodes.

9. Puts the next node into maintenance mode and repeats the cycle starting at Step 3 until all nodes have been updated.

The main benefit of CAU is that it updates a process that previously had to be performed manually. You can configure CAU to automatically apply updates to cluster nodes when they are approved through WSUS or System Center Configuration Manager, or you can trigger CAU manually as needed.

You can configure the CAU options so that the updates are rolled back across the cluster if an update fails to install on a node after a specified number of attempts. You can also configure Advanced Options, such as requiring scripts to run before and after the update process has occurred and whether services hosted on the cluster require special shutdown or startup configurations.

Failover and preference settings

Cluster preference settings allow you to configure the preferred owner for a specific cluster role, and you can also configure different preferred owners for different cluster roles. Where possible, the role is hosted on the preferred owner. You can configure a list of preferred owners, so that if the most preferred owner isn’t available, the next preferred owner hosts the role. You configure a role-preferred owner on the role’s Properties dialog box.

You configure whether the clustered role fails back to the preferred owner on the Failover tab of the cluster role’s Properties dialog box. When configuring failback, you need to:

• Determine whether you want to prevent failback

• Determine whether you want to have failback occur automatically as soon as the preferred owner is in a healthy state

• Configure failback to occur within a certain number of hours of the preferred owner returning to a healthy state

Node quarantine settings allow you to configure a node so that it is unable to rejoin the cluster if the node fails a certain number of times within a certain period. Configuring node quarantine blocks workloads from being placed back on a quarantined node until the reason for the repeated failure can be dealt with by a server administrator.

A fault domain is a collection of hardware components that share a common point of failure. Windows Server 2019 allows you to configure fault domains at the site, rack, chassis, and node levels. You can use Storage Spaces and Storage Spaces Direct with fault domains to ensure that copies of data are kept in separate fault domains. For example, if you defined fault domains at the site level, you can configure storage spaces so that redundant data resides in separate sites rather than residing only on separate cluster nodes.

Multisite clusters

Failover clusters can span multiple sites. When configuring a cluster that spans two sites, you should consider the following:

• Ensure that there are an equal number of nodes in each site.

• Allow each node to have a vote.

• Enable dynamic quorum. Dynamic quorum allows quorum to be recalculated when individual nodes leave the cluster one at a time. Dynamic quorum is enabled by default on Windows Server 2019 failover clusters.

• Use a file share witness. You should host the file share witness on a third site that has separate connectivity to the two sites that host the cluster nodes, as shown in Figure 11-1. When configured in this manner, the cluster retains quorum if one of the sites is lost. An alternative to a file share witness is a cloud witness.

If you have only two sites and are unable to place a file share witness in an independent third site, you can manually edit the cluster configuration to reassign votes so that the cluster recalculates quorum.

Manually reassigning votes is also useful to avoid split-brain scenarios. Split-brain scenarios occur when a failure occurs in a multisite cluster and when both sides of the cluster believe they have quorum. Split-brain scenarios cause challenges when connectivity is restored and make it necessary to restart servers on one side of the multisite cluster to resolve the issue. You can manually reassign votes so that one side always retains quorum if intersite connectivity is lost. For example, by setting the Melbourne site with two votes and the Sydney site with one vote, the Melbourne site always retains quorum if intersite connectivity is lost.

Cloud witness

Cloud witness has the cluster witness role hosted in Azure rather than at a third site. Cloud witness is suitable for multisite clusters. To configure a cloud witness, create a storage account in Azure, copy the storage access keys, note the endpoint URL links, and then use these links with the Configure Cluster Quorum Wizard and specify a cloud witness as shown in Figure 11-2.

Virtual machine failover clustering

One of the most common uses for failover clusters is hosting virtual machines. By deploying a workload such as SQL Server or Exchange on a highly available virtual machine, you can achieve high availability without the need for the application to be aware that it is now highly available. The virtual machine functions normally, provides services to clients on the network, and can switch between cluster nodes as necessary in the event that the individual cluster node hosting it requires maintenance or experiences some sort of failure. Building a Hyper-V failover cluster first involves creating a failover cluster and then adding the Hyper-V role to each node of the cluster.

(Get-Cluster).AutoBalancerLevel = <value>

(Get-Cluster).AutoBalancerMode = <value>

Rolling upgrades

Cluster OS Rolling Upgrade allows you to upgrade a cluster hosting Hyper-V or scale out file server workloads without taking those workloads offline. You can perform a Cluster OS Rolling Upgrade from Windows Server 2012 R2 to Windows Server 2016, or you can perform a rolling upgrade from Windows Server 2016 to Windows Server 2019. Other cluster hosted workloads, such as SQL Server, are only offline during failover from an original cluster node to a new or upgraded cluster node.

Upgrading a cluster involves either adding new nodes running Windows Server 2016 to an existing cluster running Windows Server 2012 R2 or upgrading existing nodes running Windows Server 2012 R2 to Windows Server 2016. The cluster can function normally with both Windows Server 2012 R2 and Windows Server 2016 nodes.

The nodes that you can add to a cluster depend on the cluster functional level. A cluster that only has nodes running Windows Server 2012 R2 is running at cluster functional level 8. Cluster functional level 8 supports nodes that run both Windows Server 2012 R2 and Windows Server 2016. You can also introduce cluster nodes that run either Windows Server 2012 or 2016 to the cluster as long as the cluster is still configured to use cluster functional level 8. It’s even possible to roll back to Windows Server 2012 R2 when all nodes are running Windows Server 2016 as long as you haven’t upgraded the cluster functional level from level 8.

A cluster running at functional level 9 supports nodes running Windows Server 2016 and Windows Server 2019. A cluster running at functional level 10 only supports nodes running Windows Server 2019.

You should only upgrade the cluster functional level when you are absolutely certain that you don’t want to roll back the cluster upgrade and that all nodes have been upgraded. You can determine which functional level a cluster is running at with the following Windows PowerShell command:

Click here to view code image

Get-Cluster | Select ClusterFunctionalLevel

You can also use the Get-ClusterNodeSupportedVersion cmdlet on a computer running Windows Server 2019 to determine which nodes in the cluster are running at a specific functional level and cluster upgrade version.

You can upgrade the cluster functional level by using the Update-ClusterFunctionalLevel cmdlet.

Cluster OS Rolling Upgrade has the following conditions:

• You can only upgrade a Hyper-V cluster if the CPUs support Second-Level Addressing Table (SLAT). This is because Windows Server 2016 and Windows Server 2019 Hyper-V is only supported on processors that support SLAT.

• When performing the rolling cluster upgrade, always use the most recent version of Windows Server to add nodes to the cluster.

• All cluster management tasks should be performed from the more recent operating system node than the previous version node.

• Cluster rolling upgrade cannot be used to upgrade VM guest clusters that use a shared virtual hard disk as shared storage. VM guest clusters are clusters in which each node is a virtual machine. A VM guest cluster that uses a shared virtual hard disk for shared storage cannot be upgraded, but a VM guest cluster that uses another form of shared storage, such as iSCSI, can be upgraded by using a rolling cluster upgrade.

• Before upgrading a node, drain the node of workloads and evict the node from the cluster. After you’ve upgraded the node’s operating system, you can join it to the cluster again.

• Prior to commencing Cluster OS Rolling Upgrade, ensure that all elements of the cluster and its workloads are backed up.

• You can’t lower the cluster functional level after it has been raised.

• Cluster OS Rolling Upgrade only works when the original cluster is running Windows Server 2012 R2 or later. Cluster OS Rolling Upgrade does not work with Windows Server 2012, Windows Server 2008 R2, or Windows Server 2008.

Workgroup clusters

Workgroup clusters are a special type of cluster in which cluster nodes are not members of an Active Directory domain. Workgroup clusters are also known as Active Directory Detached Clusters. The following workloads are supported for workgroup clusters:

• SQL Server. When deploying SQL Server on a workgroup cluster, it’s recommended that you use SQL Server Authentication for databases and SQL Server Always On Availability Groups.

• File Server. A supported but not recommended configuration as Kerberos will not be available as an authentication protocol for SMB traffic.

• Hyper-V. A supported but not recommended configuration. Hyper-V live migration is not supported, through it is possible to perform quick migration.

When creating a workgroup cluster, you first need to create a special account on all nodes that will participate in the cluster that has the following properties:

• The special account must have the same username and password on all cluster nodes.

• The special account must be added to the local Administrators group on each cluster node.

• The primary DNS suffix on each cluster node must be configured with the same value.

• When creating the cluster, ensure that the AdministrativeAccessPoint parameter when using the New-Cluster cmdlet is set to DNS. Ensure that the cluster name is present in the appropriate DNS zone, which depends on the primary DNS suffix, when running this command.

• You will need to run the following PowerShell command on each node to configure the LocalAccountTokenFilterPolicy registry setting to 1.

Click here to view code image

new-itemproperty -path HKLM:SOFTWAREMicrosoftWindowsCurrentVersionPoliciesSystem
-Name LocalAccountTokenFilterPolicy -Value 1

To create a workgroup cluster, use the New-Cluster cmdlet with the name parameter listing the cluster name, the node parameters listing the nodes that you wish to join to the cluster where the nodes have been configured according to the prerequisites, and the administrativeaccesspoint parameter configured for DNS. For example, to create a new workgroup cluster named workgrpclst with member nodes node1 and node2, run the following command on one of the nodes:

Click here to view code image

New-Cluster -name workgrpclst -node node1,node2 -AdministrativeAccessPoint DNS

Cluster sets

Cluster sets are a Windows Server 2019 technology that provide a method of grouping multiple failover clusters. When you implement cluster sets, you can combine smaller clusters into larger virtual clusters. These virtual clusters support virtual machine fluidity and a unified storage namespace, meaning that virtual machines can be moved between clusters in a cluster set as easily as they are moved between nodes in a traditional failover cluster. While virtual machines can be live migrated between clusters, Windows Server 2019 cluster sets do not allow virtual machines to fail over between clusters in a cluster set.

Only clusters running at the Windows Server 2019 cluster functional level 10.X or higher can participate in cluster sets. You cannot join clusters running Windows Server 2016 or Windows Server 2012 R2 to a cluster set. All clusters in a cluster set must be members of the same Active Directory forest.

Cluster sets improve the failover cluster lifecycle. Rather than adding and removing nodes from a single cluster when the node hardware is to be retired, you can add a new cluster to the cluster set, migrate workloads across from the original cluster to the new cluster, and then retire that original cluster. Cluster sets are currently supported by Microsoft for up to 64 cluster nodes, which is the same number of nodes supported in an individual failover cluster. That being said, there is no specific limit to the number of nodes that may exist within a cluster set, so going beyond 64 nodes in a cluster set is possible should you want to try it.

Cluster sets consist of a management cluster and member clusters. The management cluster is the cluster set that holds the management role for the cluster set and also hosts the unified storage namespace for the cluster set Scale-Out File Server. The management cluster does not host workloads, such as virtual machines, and its role is to mange the relationship between other clusters in the cluster set and to host the storage namespace. The new role that the management cluster hosts is termed the “Cluster Set Master” (CS-Master). Member clusters hold the “Cluster Set Worker” or CS-Worker role.

Managing failover clustering with PowerShell

You can use cmdlets in the FailoverClusters module, as described in Table 11-3, to manage failover clusters on Windows Server 2016.

Network Load Balancing prerequisites

In terms of setup, NLB is straightforward and only requires you to install the feature. Although you can install the feature on a node, you must ensure that it meets some prerequisites before you can add it to a cluster. Before you add a host to an NLB cluster, you must ensure the following:

• All nodes in the NLB cluster must reside on the same subnet. While you can configure a subnet to span multiple geographic locations, the cluster is unlikely to converge successfully if the latency between nodes exceeds 250 milliseconds.

• All network adapters must either be configured as unicast or multicast. You can’t use Windows NLB to create an NLB cluster that has a mixture of unicast and multicast addresses assigned to network adapters.

• If you choose to use the unicast cluster configuration mode, the network adapter needs to support changing its MAC address.

• The network adapter must be configured with a static IP address.

• Only TCP/IP is supported on the network adapter that is used for NLB. You cannot bind other protocols, such as IPX, Token Ring, or Banyan Vines, on the adapter.

There is no restriction on the number of network adapters in each host. You can have one host with three network adapters in a team and another host with five separate adapters participate in the same NLB cluster. You can run nodes on different editions of Windows Server, although considering that the only difference between the Standard and Datacenter edition is virtual machine licensing, you are unlikely to be running NLB on the Datacenter edition. While it is possible to have working NLB clusters if you are running different versions of Windows Server, such as Windows Server 2012 R2, Windows Server 2016, and Windows Server 2019, you should upgrade all nodes as soon as possible to the same operating system. You should also attempt to ensure that nodes have similar hardware capacity or are running on similarly provisioned virtual machines so that a client doesn’t get a radically different experience when connecting to different nodes.

NLB cluster operation modes

When configuring an NLB cluster, you can choose between one of three separate cluster-operation modes. The mode that you select depends on the configuration of the cluster nodes and the type of network switch to which the cluster nodes are connected. You can configure the cluster-operation mode when creating the cluster. The operation modes are as follows:

• Unicast Mode. If you configure an NLB cluster to use the unicast cluster-operation mode, all nodes in the cluster will use the same unicast MAC address. Outbound traffic from node members uses a modified MAC address that is determined by the cluster host’s priority settings. The drawback to using unicast mode with nodes that have a single adapter is that you can only perform management tasks from computers located on the same TCP/IP subnet as the cluster nodes. This restriction doesn’t apply, however, when the cluster nodes have multiple network adapters that are not NIC team members. When you use unicast with multiple network adapters, one adapter is dedicated to cluster communication, and you can connect to any other adapters to perform management tasks. You can improve cluster operation by placing the adapters that you use for cluster communication and the adapters you use for management tasks on separate VLANs.

• Multicast Mode. This mode is suitable when each node only has one network adapter. When you configure multicast mode, each cluster host keeps its original network adapter’s MAC address, but it is also assigned a multicast MAC address. All nodes in the cluster use the same multicast MAC address. You can only use multicast mode if your organization’s switches and routers support it. Unless you’ve got bargain-basement network equipment, it’s likely that your current network hardware supports multicast mode for NLB.

• IGMP Multicast. Internet Group Management Protocol (IGMP) multicast mode is an advanced form of multicast mode that reduces the chances of the network switch being flooded with traffic. It works by only forwarding traffic through the ports that are connected to hosts that participate in the NLB cluster, but it also requires a switch that supports this functionality.

Managing cluster hosts

You manage NLB clusters through the Network Load Balancing Manager console. The NLB console is available as part of the RSAT tools, meaning that you can manage one or more NLB clusters from a computer that is not a member of the cluster. You do, however, need to manage clusters by using an account that is a member of the local Administrators group on each cluster node. You can use this console to create and manage clusters and cluster nodes.

After the cluster is set up and functioning, most maintenance operations, such as applying software updates, are performed on cluster nodes. When performing maintenance tasks, you should perform the task on one node at a time so the application that the cluster provides to users continues to remain available. Prior to performing maintenance tasks on a node, you should first stop incoming connections and allow existing connections to be completed naturally. When there are no connections to the node, you can then stop the node, apply the updates, and restart the node. You have the following options for managing cluster nodes:

• Drainstop. Blocks new connections to the cluster node but doesn’t terminate existing connections. Use this prior to planned maintenance to gracefully evacuate the cluster of connections.

• Stop. Stops the cluster node. All connections to the cluster node from clients are stopped. Use Stop after you use Drainstop so that you can then perform maintenance tasks, such as applying updates.

• Start. Starts a cluster node that is in a stopped state.

• Suspend. Pauses the cluster node until you issue the Resume command. Using Suspend does not shut down the cluster service, but it does terminate current connections as well as block new connections.

• Resume. Resumes a suspended cluster node.

Port rules

An NLB port rule allows you to configure how the NLB cluster responds to incoming traffic on a specific port and protocol, such as TCP port 80 (HTTP) or UDP port 69 (Trivial FTP). Each host in an NLB cluster must use the same port rules. While it is possible to configure port rules on a per-node basis, it’s safer to configure them at the cluster level to ensure that they are consistent. The default port rule redirects traffic on any TCP or UDP port to each node in the cluster in a balanced way. If you want to create specific rules, delete the default rule.

Filtering and affinity

When you create a port rule, you also choose a filtering mode, which might require you to configure affinity. Filtering allows you to determine whether incoming traffic is handled by one, some, or all nodes in the NLB cluster. If you choose to allow multiple nodes to accept incoming traffic, you need to configure affinity. The affinity setting determines whether one node handles all subsequent requests from the same client, or if subsequent requests from the same client are redistributed across other nodes. Affinity is often important with web applications where a consistent session must exist between the client and one web server after the session is established. You can configure the following filtering modes:

• Multiple Host. This is the default filtering mode. This mode allows traffic to be directed to any node in the NLB cluster. You also need to specify one of the following affinity settings:

  • Single. When you configure this option, the incoming request is distributed to one of the cluster nodes. All subsequent traffic from the originating host is directed to that same node for the duration of the session. This is the default multiple host affinity. Don’t confuse Multiple Host, Single Affinity with Single Host filtering.

  • None. Incoming traffic is distributed to the cluster node under the least load even if there is an existing session. This means that multiple nodes may handle traffic from a single session.

  • Network. This option directs clients to cluster nodes based on the requesting client’s network address.

• Single Host. A single node handles all traffic sent to the cluster on this port rule. In this case, no load balancing occurs. For example, if you have a four-node cluster, but you want node three to be the only one that handles traffic on TCP port 25, you would configure a port rule with the filtering mode set to Single Host.

• Disable The Port Range. When you configure this setting, the NLB cluster drops traffic sent to the cluster that matches the port rule.

Managing NLB with PowerShell

You can use the Windows PowerShell cmdlets listed in Table 11-4 to manage NLB clusters on Windows Server 2019.