Cisco HyperFlex HX-Series System Components

These HX System components comprise the Cisco HX Series server and Cisco HX Data Platform.

• Cisco HX-Series server: You can use any of the following servers to configure Cisco HyperFlex:

  • Converged nodes—All Flash: Cisco HyperFlex HXAF240c M5, HXAF220c M5, HXAF240c M4, and HXAF220c M4.

  • Converged nodes—Hybrid: Cisco HyperFlex HX240c M5, HX220c M5, HX240c M4, and HX220c M4.

  • Compute-only—Cisco B200 M3/M4, B260 M4, B420 M4, B460 M4, B480 M5, C240 M3/M4, C220 M3/M4, C480 M5, C460 M4, B200 M5, C220 M5, and C240 M5.

• Cisco HX Data Platform: The HX Data Platform consists of the following components:

  • Cisco HX Data Platform installer: Download this installer to a server connected to the storage cluster. The HX Data Platform installer configures the service profiles and policies within Cisco UCS Manager, deploys the controller VMs, installs the software, creates the storage cluster, and updates the VMware vCenter plug-in.

  • Storage controller VM: Use the HX Data Platform installer to install the storage controller VM on each converged node in the managed storage cluster.

  • Cisco HX Data Platform plug-in: This integrated VMware vSphere interface monitors and manages the storage in a storage cluster.

Cisco UCS Fabric Interconnects (FIs)

Fabric interconnects provide both network connectivity and management capabilities to any attached Cisco HX-Series server.

FIs that were purchased and deployed as part of the Cisco HyperFlex system are also referred to as the HX FI domain in this chapter. The following fabric interconnects are supported:

• Cisco UCS 6200 Series fabric interconnects

• Cisco UCS 6300 Series fabric interconnects

• Cisco UCS 6400 Series fabric interconnects

Cisco Nexus Switches

Cisco Nexus switches deliver high-density, configurable ports for flexible access deployment and migration.

HyperFlex Software Versions

The HX components—Cisco HX Data Platform Installer, Cisco HX Data Platform, and Cisco UCS firmware—are installed on different servers. Verify that the components on the servers used with and within an HX storage cluster are compatible. For more information, visit https://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-data-platform-software/products-release-notes-list.html.

Host Requirements

A Cisco HyperFlex cluster contains a minimum of three converged HyperFlex nodes. There is an option of adding compute-only nodes to provide additional compute power if there is no need for extra storage. Each server in a HyperFlex cluster is also referred as a HyperFlex node. You need to ensure that each node has the correct settings installed and configured before you deploy the storage cluster. For further information, see https://www.cisco.com/c/en/us/support/hyperconverged-systems/hyperflex-hx-series/products-installation-guides-list.html.

Ensure that the following host requirements are met:

• Use the same VLAN IDs for all the servers (nodes or hosts) in a cluster.

• Use the same administrator login credentials for all the ESXi servers across a storage cluster.

• Keep SSH enabled on all ESXi hosts.

• Configure DNS and NTP on all servers.

• Install and configure VMware vSphere.

• Use a single VIC only for converged nodes or compute-only nodes. Additional VICs or PCIe NICs are not supported.

Disk Requirements

The disk requirements vary between converged nodes and compute-only nodes. To increase the available CPU and memory capacity, you can expand an existing cluster with compute-only nodes as needed. These compute-only nodes provide no increase in storage performance or storage capacity.

Alternatively, adding converged nodes increases the storage performance and storage capacity alongside CPU and memory resources.

Browser Recommendations

Use one of the browsers listed in Table 3-1 to run the listed HyperFlex components. These browsers have been tested and approved. Other browsers might work, but full functionality has not been tested and confirmed.

Port Requirements

If your network is behind a firewall, in addition to the standard port requirements, VMware recommends ports for VMware ESXi and VMware vCenter.

Table 3-2 lists the open port requirements in a network between various components (HX Data Platform installer, ESXi, SCVM, vCenter, and so on) for HyperFlex installation and operations:

• CIP-M is for the cluster management IP address.

• SCVM is the management IP address for the controller VM.

• ESXi is the management IP address for the hypervisor.

Verify that the firewall ports listed in Table 3-2 are open.

HyperFlex External Connections

Table 3-3 covers the details and requirements for HyperFlex external connections to Intersight, Auto-Support, and the postinstallation script.

UCS/Fabric Interconnect Network Provisioning

Prior to setting up a HyperFlex cluster, you need to plan the upstream bandwidth capacity for optimal network traffic management. This ensures that the flow is in steady state, even if there is a component failure or a partial network outage.

By default, the hx-vm-network vSwitch switch is configured as active/active. All other vSwitch switches are configured as active/standby.

Configuration for HyperFlex 3.5

Figure 3-2 provides an overview of the UCS service profile mapping to vmware vswitch and vmnics. This is configured automatically during HyperFlex installation.

The default vSwitch NIC teaming policy and failover policy is set to yes to ensure that all management, vMotion, and storage traffic are locally forwarded to the fabric interconnects to keep the flow in steady state. When vNIC-a fails, ESXi computes the load balancing, and all the virtual ports are repinned to vNIC-b. When vNIC-a comes back online, repinning applies, and virtual ports are rebalanced across vNIC-a and vNIC-b. This reduces the latency and bandwidth utilization upstream of the Cisco UCS fabric interconnects. Figure 3-3 shows traffic flow in a normal state when traffic is limited to within the fabric interconnects.

If one or more server links fail—for instance, if Host 1 loses connectivity to fabric A and Host 2 loses connectivity to fabric B—the traffic must go through the upstream switches. Therefore, the uplink network bandwidth usage increases. Figure 3-4 shows traffic flow when an active link failure occurs, resulting in traffic flows through the upstream switch.

Network Settings

The following are best practices for UCS and ESXi network configuration when deploying HyperFlex:

• Use different subnets and VLANs for each network.

• Directly attach each host to a Cisco UCS fabric interconnect using a 10/25/40 Gbps cable.

• Do not use VLAN 1, which is the default VLAN, as doing so can cause networking issues, especially if disjoint Layer 2 configuration is used.

• For VLANs set as non-native by default, configure the upstream switches to accommodate the non-native VLANs.

Each ESXi host needs the following networks:

• Management traffic network: Handles the hypervisor (ESXi server) management and storage cluster management.

• Data traffic network: Handles the hypervisor and storage data traffic.

• vMotion network: Used for virtual machine vMotion.

• VM network: Handles virtual machine network traffic.

There are four vSwitch switches, each carrying a different network:

• vswitch-hx-inband-mgmt: Used for ESXi management and storage controller management.

• vswitch-hx-storage-data: Used for ESXi storage data and HX Data Platform replication.

• vswitch-hx-vmotion: Used for VM and storage vMotion.

• vswitch-hx-vm-network: Used for VM data traffic.

VLAN and vSwitch Requirements

When installing HyperFlex, it is important to provide at least three VLAN IDs. All VLANs must be configured on the fabric interconnects during the installation. Table 3-4 outlines these requirements.

External Switch Tagging (EST) and vSwitch settings are applied using UCS Manager profiles. The HX Data Platform installer automates and simplifies this process.

Cisco UCS Requirements

When installing HyperFlex, you need to provide information for the UCS fabric interconnect and UCS Manager when prompted, as outlined in Table 3-5 and Table 3-6.

Hypervisor Requirements

When installing HyperFlex, you need to enter the IP address from the range of addresses that are available to the ESXi servers on the storage management network or storage data network through vCenter, as shown in Table 3-7. Provide static IP addresses for all network addresses.

Storage Cluster Requirements

A storage cluster is a component of the Cisco HX Data Platform that reduces storage complexity by providing a single datastore that is easily provisioned in vSphere Web Client. Data is fully distributed across disks in all the servers that are in the storage cluster to leverage controller resources and provide high availability.

A storage cluster is independent of the associated vCenter cluster. You can create a storage cluster by using ESXi hosts that are in the vCenter cluster.

To define a storage cluster, provide the parameters outlined in Table 3-8.

vCenter Configuration Requirements

Configuration of vCenter requires an administrator-level account and password. Ensure that you have an existing vCenter server and also ensure that the following vSphere services are operational:

• Enable Dynamic Resource Scheduler (DRS), if licensed (optional)

• Enable vMotion

• Enable high availability (required to define failover capacity and for expanding the datastore heartbeat)

• Ensure that user VMs are version 9 or later (required to use HX Data Platform, native snapshots, and ReadyClones

Table 3-9 outlines the vCenter information required to complete the HyperFlex installation.

System Services Requirements

Before installing Cisco HX Data Platform, ensure that the following network connections and services are operational. Table 3-10 outlines the system services requirements.

CPU Resource Reservation for Controller VMs

As the storage controller VMs provide critical functionality for the HyperFlex Data Platform, the HX Data Platform installer configures CPU resource reservations for the controller VMs. This reservation guarantees that the controller VMs have the minimum required CPU resources. This is useful in situations where the physical CPU resources of the ESXi hypervisor host are heavily consumed by the guest VMs. Table 3-11 details the CPU resource reservation for storage controller VMs.

Memory Resource Reservation for Controller Virtual Machines

Table 3-12 details the memory resource reservations for the storage controller VMs.

The compute only nodes, for example, the B200 blades, have a lightweight storage controller VM that is configured with only 1 vCPU and 512 MB of memory reservation.

Controller VM Locations

The physical storage locations of the controller VMs differs among the Cisco HX-Series rack servers due to differences in the physical disk locations and connections on those server models. The storage controller VM is operationally no different from any other typical virtual machines in an ESXi environment. The VM must have a virtual disk with the bootable root file system available in a location separate from the SAS HBA that the VM is controlling via VMDirectPath I/O. The configuration details of the models are as follows:

• HX220c M5, HXAF220c M5, HX240c M5, and HXAF240c M5: The server boots the ESXi hypervisor from the internal M.2 form factor SSD. The M.2 SSD is partitioned by the ESXi installer, and the remaining 216 GB of space is used as a VMFS datastore. The controller VM’s root file system is stored on a 2.5 GB virtual disk, /dev/sda, which is placed on this VMFS datastore. The controller VM has full control of all the front- and rear-facing hot-swappable disks via PCI passthrough control of the SAS HBA. The controller VM operating system sees the 240 GB SSD (also commonly called the “housekeeping” disk) as /dev/sdb and places HyperFlex binaries and logs on this disk. The remaining disks seen by the controller VM OS are used by the HX distributed file system for caching and capacity layers. Figure 3-5 and Figure 3-6 detail the storage controller VM placement on ESXi hypervisor hosts for these server types.

• HX220c M4 and HXAF220c M4: The server boots the ESXi hypervisor from the internal mirrored SD cards. The SD card is partitioned by the ESXi installer, and the remaining space is used as a VMFS datastore. The controller VM’s root file system is stored on a 2.5 GB virtual disk, /dev/sda, which is placed on this VMFS datastore. The controller VM has full control of all the front-facing hot-swappable disks via PCI passthrough control of the SAS HBA. The controller VM operating system sees the 120 GB or 240 GB SSD (also commonly called the “housekeeping” disk) as /dev/sdb and places HyperFlex binaries and logs on this disk. The remaining disks seen by the controller VM OS are used by the HX distributed file system for caching and capacity layers. Figure 3-7 details the storage controller VM placement on ESXi hypervisor hosts for these server types.

• HX240c M4 and HXAF240c M4: The server boots the ESXi hypervisor from the internal mirrored SD cards. The HX240c-M4SX and HXAF240c-M4SX servers have a built-in SATA controller provided by the Intel Wellsburg Platform Controller Hub (PCH) chip, and the 120 GB or 240 GB housekeeping disk is connected to it, placed in an internal drive carrier. Since this model does not connect the housekeeping disk to the SAS HBA, the ESXi hypervisor remains in control of this disk, and a VMFS datastore is provisioned there, using the entire disk. On this VMFS datastore, a 2.2 GB virtual disk is created and used by the controller VM as /dev/sda for the root file system, and an 87 GB virtual disk is created and used by the controller VM as /dev/sdb, placing the HyperFlex binaries and logs on this disk. The front-facing hot-swappable disks, seen by the controller VM OS via PCI passthrough control of the SAS HBA, are used by the HX distributed file system for caching and capacity layers. Figure 3-8 details the storage controller VM placement on ESXi hypervisor hosts for these server types.

Auto-Support Requirements

Auto-Support (ASUP) is the alert notification service provided through HX Data Platform. If you enable Auto-Support, notifications are sent from HX Data Platform to designated email addresses or email aliases that you want to receive the notifications. To configure Auto-Support, use the information in Table 3-13.

Single Sign-On Requirements

The single sign-on (SSO) URL is provided by vCenter. If it is not directly reachable from the controller VM, you can configure the location explicitly by using Installer Advanced Settings. The SSO URL can be found in vCenter at vCenter Server > Manage > Advanced Settings, key config.vpxd.sso.sts.uri.

Physical Connectivity Illustrations for Direct Connect Mode Cluster Setup

Only direct connect mode is supported for HX servers with fabric interconnects. For example, the topologies in Figure 3-9, Figure 3-10, and Figure 3-11 are considered supported. These topologies show a sample of direct connect mode physical connectivity for C-Series rack-mount servers with Cisco UCS Domain and Cisco UCS Manager, release 3.1 or later. These images show the cabling configuration for Cisco UCS Manager integration with a C-Series rack-mount server. The paths shown in these figures carry both management traffic and data traffic.

The numbered labels in the preceding figures indicate the following:

1. Cisco UCS 6454 fabric interconnect or Cisco UCS 6200 Series or 6300 FI (fabric A)

2. Cisco UCS 6454 fabric interconnect or Cisco UCS 6200 Series or 6300 FI (fabric B)

3. C-Series rack-mount server

4. Cisco UCS VIC in supported PCIe slot

In Figures 3-9 and 3-10, XGb represents a 40 Gigabit Ethernet connection or a 10 Gigabit Ethernet connection. For 10 Gigabit Ethernet, the following cables are used:

• 4x10 breakout Small Form-Factor Pluggable (SFP) cables

• 4x10 active optical (OAC) cables

• 10G SFP cable that uses the Qualified Security Assessor (QSA) module

Installation Workflow

The installation workflow illustrated in Figure 3-12 summarizes the steps involved in creating a standard cluster using the HX Data Platform installer:

Step 1. Deploy the HX Data Platform installer OVA by using the vSphere Web Client.

Step 2. Enter UCS Manager, vCenter, and hypervisor credentials.

Step 3. Configure server ports and associate HyperFlex servers.

Step 4. Configure UCSM (VLANs, MAC pool, hx-ext-mgmt IP oool for out-of-band CIMC, iSCSi storage, FC storage, and so on).

Step 5. Configure the hypervisor.

Step 6. Configure the IP addresses.

Step 7. Configure and deploy the HyperFlex cluster.

The sections that follow cover the steps of this workflow in detail.

Deploy the HX Data Platform Installer OVA by Using the vSphere Web Client

You need to connect to vCenter to deploy the OVA file and provide the IP address properties. Deploying directly from an ESXi host does not allow you to set the values correctly. Do not deploy the HX Data Platform installer to an ESXi server that is going to be a node in the Cisco HX storage cluster.

The procedure for deploying the HX Data Platform installer OVA using vSphere Web Client is as follows:

Step 1. Locate and download the HX Data Platform installer OVA from cisco.com. Download the HX Data Platform installer to a node that is on the storage management network (for example, Cisco-HX-Data-Platform-Installer-v3.5.2e-31762-esx.ova); this node will be used for the HX Data Platform storage cluster.

Step 2. Deploy the HX Data Platform installer by using the VMware hypervisor to create a HX Data Platform installer virtual machine.

Step 3. Select where the HX Data Platform installer is located. Accept the defaults and select the appropriate network. Then review the details, as shown in Figure 3-14.

Step 4. Enter a static IP address for use by the HX Data Platform installer VM, as shown in Figure 3-15. Click Next.

Step 5. In the screen shown in Figure 3-16, verify that the options listed are correct. Click Finish and wait for the HX Data Platform installer VM to be added to the vSphere infrastructure.

To power on the HX Data Platform installer manually, navigate to the virtual machine list and power on the installer VM.

Step 6. Open the HX Data Platform installer virtual machine console and notice that the initial console display lists the HX Data Platform installer virtual machine IP address:

Click here to view code image

Data Platform Installer.
*******************************************
You can start the installation by visiting
the following URL:
http://192.168.202.20
*******************************************
Cisco-HX-Data-Platform-Installer login:

Step 7. Use the URL listed in the output (for example, http://192.168.202.20) to log in to the HX Data Platform installer.

Step 8. Accept the self-signed certificate.

Step 9. Log in using the username root and the password you provided as part of the OVA deployment.

Configuring and Deploying a Standard HyperFlex Cluster

When you log in to the HX Data Platform installer with root user credentials, you see three cluster installation options under Create Cluster. For this example, select Standard Cluster, as shown in Figure 3-17.

Under Credentials, enter the UCS Manager and vCenter username (for example, admin) and password, as shown in Figure 3-18. Click Continue to configure the server selection.

On the Server Selection page, the UCS Manager credentials you provided are used to discover the UCSM configuration and list the already discovered HyperFlex servers, as shown in Figure 3-19.

There is also an option to configure server ports on UCS Manager from the installer. Select the unassociated servers that will be used for the HyperFlex installation. The example in Figure 3-20 shows three servers selected to deploy a three-node HyperFlex cluster. Click Continue on the Server Selection page to continue the UCSM configuration.

On the UCSM Configuration page, configure VLAN, MAC pool, and Cisco IMC access (out-of-band or in-band CIMC access), as shown in Figure 3-21.

If you want to add external storage, configure iSCSI storage and FC storage details toward the bottom of the UCSM Configuration. In the Advanced section at the bottom of this page, configure the UCS server firmware version, HyperFlex cluster name, and organization name, as shown in Figure 3-22:

• HyperFlex cluster name: The HyperFlex cluster name is applied to a group of HX servers in a given cluster. The HyperFlex cluster name adds a label to service profiles for easier identification.

• Organization name: Specify a unique org name to ensure isolation of the HyperFlex environment from the rest of the UCS domain. Click Continue to apply the UCSM Configuration settings.

On the Hypervisor Configuration page (see Figure 3-23), configure the subnet mark, gateway, and IP address for the hypervisor settings and hypervisor credentials. In the Hypervisor Settings section, select Make IP Addresses and Hostnames Sequential to make the IP addresses sequential. Click Continue to configure the IP addresses.

On the IP Addresses page, shown in Figure 3-24, configure the management and data IP addresses for hypervisor and storage controller. Select Make IP Addresses Sequential to make the IP addresses sequential. For each HX node, enter the hypervisor, storage controller, management, and data IP addresses. For the IP addresses, specify whether the network belongs to the data network or the management network.

Also configure the cluster IP address for management and data. If a node with a cluster IP address becomes unavailable, the cluster IP address is moved to another node in the storage cluster. Click Continue to apply the IP Addresses page settings.

On the Cluster Configuration page, shown in Figure 3-25, for the Cisco HX storage cluster, complete the following fields to begin deploying the HyperFlex cluster:

• In the Replication Factor drop-down menu, specify the number of redundant replicas of your data across the storage cluster. Set the replication factor to either 2 or 3 redundant replicas:

  • For hybrid servers (servers that contain SSD and HDDs), the default value is 3.

  • For all flash servers (servers that contain only SSDs), select either 2 or 3.

A replication factor of 3 is highly recommended for all environments except Hyper- Flex Edge. A replication factor of 2 has a lower level of availability and resiliency. The risk of outage due to component or node failures should be mitigated by having active and regular backups.

• In the Controller VM section, create a new password for the administrative user of the HyperFlex cluster. A default administrator username and password are applied to the controller VMs. The VMs are installed on all converged and compute-only nodes.

• In the vCenter Configuration section, complete the following fields:

  • vCenter Datacenter Name: Enter the vCenter data center name for the Cisco HyperFlex cluster.

  • vCenter Cluster Name: Enter the vCenter cluster name.

In the System Services section, configure the DNS server, NTP server, DNS domain name, and time zone.

In the Connected Services section of the Cluster Configuration page, shown in Figure 3-26, select Enable Connected Services to enable Auto-Support and Intersight management.

In Send service ticket notifications to, enter the email address for sending SR notifications when triggered by Auto-Support.

In the Advanced Configuration section of the Cluster Configuration page, shown in Figure 3-27, configure the following:

• Jumbo Frames: By selecting Enable Jumbo Frames on Data Network, you can set the MTU size for the storage data network on the host vSwitch switches and vNICs, as well as each storage controller VM. The default value is 9000.

• Disk Partitions: If you select Clean up disk partitions, you can remove all existing data and partitions from all nodes added to the storage cluster for manually prepared servers. Select this option to delete existing data and partitions. You must back up any data that should be retained.

• (Optional) vCenter Single-Sign-On Server: This information is required only if the SSO URL is not reachable.

• Virtual Desktop (VDI): Check this option for VDI-only environments.

Click Start to begin deploying the HyperFlex cluster. The Progress page displays the progress of various configuration tasks, as shown in Figure 3-28.

When the installation finishes, you see the Cluster Creation Successful message shown in Figure 3-29.

Browse to the Summary page, shown in Figure 3-30, to review the configuration information.

root@Hyperflex-Installer:~# post_install

Select post_install workflow-

 1. New/Existing Cluster
 2. Expanded Cluster (for non-edge clusters)
 3. Generate Certificate

Selection: 1
Logging in to controller hx-02-cmip.rchs.local
HX CVM admin password:
Getting ESX hosts from HX cluster...
vCenter URL: 192.168.202.35
Enter vCenter username (user@domain): [email protected]
vCenter Password:
Found datacenter RCH-HX-02
Found cluster RCH-HX-02

post_install to be run for the following hosts:
 hx-02-esxi-01.rchs.local
 hx-02-esxi-02.rchs.local
 hx-02-esxi-03.rchs.local

Enter ESX root password:

Enter vSphere license key?  (y/n) n

Enable HA/DRS on cluster? (y/n) y
Successfully completed configuring cluster HA.

Disable SSH warning? (y/n) y

Add vmotion interfaces? (y/n) y
 Netmask for vMotion: 255.255.254.0
 VLAN ID: (0-4096) 208
 vMotion MTU is set to use jumbo frames (9000 bytes). Do you want to
   change to 1500 bytes? (y/n) y
 vMotion IP for hx-02-esxi-01.rchs.local: 192.168.208.17
 Adding vmotion-208 to hx-02-esxi-01.rchs.local
 Adding vmkernel to hx-02-esxi-01.rchs.local
 vMotion IP for hx-02-esxi-02.rchs.local: 192.168.208.18
 Adding vmotion-208 to hx-02-esxi-02.rchs.local
 Adding vmkernel to hx-02-esxi-02.rchs.local
 vMotion IP for hx-02-esxi-03.rchs.local: 192.168.208.19
 Adding vmotion-208 to hx-02-esxi-03.rchs.local
 Adding vmkernel to hx-02-esxi-03.rchs.local

Add VM network VLANs? (y/n) n

Run health check? (y/n) y

Validating cluster health and configuration...

Cluster Summary:
     Version – 3.5(2e)
     Model - HXAF240C-M5SX
     Health - HEALTHY
     ASUP enabled - Trues

root@Hyperflex-Installer:~#

root@Cisco-HX-Data-Platform-Installer:~# post_install
Select post_install workflow-

 1. New/Existing Cluster
 2. Expanded Cluster
 3. Generate Certificate

 Selection: 3
 Certificate generation workflow selected

Logging in to controller 192.168.200.23
HX CVM admin password:
Getting ESX hosts from HX cluster...

Select Certificate Generation Workflow-

 1. With vCenter
 2. Without vCenter

 Selection: 1
vCenter URL: 192.168.202.33
Enter vCenter username (user@domain): [email protected]
vCenter Password:
Starting certificate generation and re-registration.
Trying to retrieve vCenterDatacenter information ....
Trying to retrieve vCenterCluster information ....
Certificate generated successfully.
Cluster re-registration in progress ....
Cluster re-registered successfully.

root@HyperFlex-Installer:~#

HyperFlex Edge Deployment Options

HyperFlex Edge can be deployed using Cisco Intersight from the cloud or by using the on-premises installer, depending on the requirements:

• HyperFlex on-premises OVA installer: Use this option for on-premises Edge deployments for three- and four-node clusters. This type of deployment supports all three network topologies and requires download and installation of the installer along with local network access. Note that use of the on-premises installer is not support for two-node HyperFlex Edge clusters.

• Intersight installer: Use this option to deploy HyperFlex Edge from the cloud. This deployment option supports all Edge cluster sizes and network topologies. See Chapter 7 for more details on this option.

Installation Overview

Table 3-16 summarizes the installation workflow for Edge deployments. As you can see, steps 1–3 are the same for Gigabit Ethernet and 10 Gigabit Ethernet deployments. However, step 4 is applicable to Gigabit Ethernet deployments, and steps 5–6 are for 10 Gigabit Ethernet deployments.

Configuring and Deploying a HyperFlex Edge Cluster (Gigabit Ethernet Only)

After completing the preinstallation checklist, in a web browser, enter the IP address of the installer VM and click Accept or Continue to bypass any SSL certificate errors.

Log in to the Cisco HX Data Platform installer using the username root and the password provided as part of the OVA deployment. Read the end-user license agreement, check I accept the terms and conditions, and click Login.

On the Select a Workflow page, click Create Cluster > Edge Cluster, as shown in Figure 3-31.

On the Credentials page, complete the fields shown in Figure 3-32 and click Continue.

On the IP Addresses page, enter the assigned address for each server, as shown in Figure 3-33, and click Continue.

On the Cluster Configuration page, complete the fields as shown in Figure 3-34 and click Start.

In the Connected Services section of the Cluster Configuration page (see Figure 3-35), select Enable Connected Services to enable Auto-Support and Intersight management. In the Send service ticket notifications to box, enter the email address for sending SR notifications when triggered by Auto-Support.

In the Advanced Configuration section of the Cluster Configuration page, shown in Figure 3-36, configure the following:

• Jumbo frames: By selecting Enable Jumbo Frames on Data Network, you can set the MTU size for the storage data network on the host vSwitch switches and vNICs, as well as each storage controller VM. The default value is 9000.

• Disk Partitions: If you select Clean up disk partitions, you can remove all existing data and partitions from all nodes added to the storage cluster for manually prepared servers. Select this option to delete existing data and partitions. You must back up any data that should be retained.

• (Optional) vCenter Single-Sign-On Server: This information is required only if the SSO URL is not reachable.

• Virtual Desktop (VDI): Check this option for VDI-only environments.

When you click Start, the installation process starts, as shown in Figure 3-37.

When the installation is complete (see Figure 3-38), the installer takes you to the summary page (see Figure 3-39), on which you can Launch HyperFlex Connect to view the cluster information.

Configuring and Deploying a HyperFlex Edge Cluster (10 Gigabit Ethernet Only)

To deploy a 10 Gigabit Ethernet HyperFlex edge cluster, you need to perform an additional step of running the configuration script (hx_edge) along with all the same steps for deploying HyperFlex Edge (Gigabit Ethernet).

Follow these steps:

Figure 3-40 shows the results from a sample script run.

After you run the script and complete the steps, deploy the installer and run the installation tasks as described in the section “Configuring and Deploying a HyperFlex Edge Cluster (Gigabit Ethernet Only),” earlier in this chapter.

HyperFlex Edge Postinstallation Tasks

To run the postinstallation script (which is the same for both HX Edge Gigabit Ethernet and HX Edge 10 Gigabit Ethernet), do the following:

Step 1. In a web browser, navigate to http://<installer VM IP address>/mssh and log in using the username root and your password.

Step 2. Run ~#post_install.py to start the postinstallation tasks in the web-based SSH window.

On execution of the post installation script, the following options appear:

Click here to view code image

Select post_install workflow-

1. New/Existing Cluster
2. Expanded Cluster
3. Generate Certificate

Choose one of the following options as per the requirement:

• 1: Choose this option to run the postinstallation script on a newly created cluster or on an existing cluster. When you select this option, the script runs the postinstallation operations on all the nodes in the cluster.

• 2: Choose this option to run the postinstallation script on expanded nodes or on newly added nodes after executing the expansion workflow. When you select this option, the script fetches the list of expanded nodes and runs the postinstallation operations on the expanded nodes.

• 3: Choose this option to have a unique SSL certificate in the cluster. When you select this option, the current certificate is replaced with the newly created SSL certificate. This option is not required for the cluster expansion.

Step 3. Follow the onscreen prompts to complete the installation.

The post_install script completes the following:

• Licenses the vCenter host.

• Enables high availability/DRS on the cluster per best practices.

• Suppresses SSH/shell warnings in vCenter.

• Configures vMotion according to best practices.

• Adds additional guest VLANs/port groups.

• Performs a HyperFlex Edge configuration check.

On successful completion of the post_install workflow, the summary of the configuration executed based on the chosen option is displayed under Cluster Summary.

Preinstallation Checklist

This section covers prerequisites for successfully deploying a HyperFlex stretch cluster. It describes the guidelines to follow when deploying a stretch cluster.

Network Requirements and Network Topology

Before you deploy a stretch cluster, ensure that you meet the following requirements:

• 10 Gbps dedicated, 5 ms round-trip time (RTT) latency between the two active sites is required.

• 100 Mbps, 200 ms RTT worst-case latency for 16 KB packets between the active sites and witness site is required.

• Existing fabric interconnects are supported, provided that the fabric interconnects support M5 servers.

• User VMs should be capable of vMotion to any site, without impacting external network connectivity to these VMs.

• As with regular HX clusters, two separate IP subnets are required, both of them over stretch L2: one subnet for data traffic and one for management traffic, with the management subnet reachable from vCenter and the witness node.

• The static IP address assigned to the witness VM can be changed only with cluster redeployment. If a DHCP server is used to define the network configuration, the IP address needs to be dedicated to the witness VM.

Figure 3-42 illustrates the stretch cluster network topology.

Deploying the Witness Node

A witness VM must be deployed on an ESXi server that has sufficient hosting capability. The witness VM requires four vCPUs, 8 GB of memory, and 40 GB of disk space. Ensure that the virtual network on this ESXi host is reachable from both of the stretch cluster sites.

Download the HyperFlex witness node onto your desktop or host that runs vSphere Web Client from cisco.com (for example, HyperFlex-Witness-1.0.2.ova). HyperFlex witness node version 1.0.2 is supported in HX Data Platform 3.5(1a) and later. Follow these steps:

Step 1. Log in to vSphere Web Client. Choose the ESXi server where the witness should be deployed. Right-click the ESXi host and select Deploy OVF Template.

Step 2. Browse and select the HyperFlex-Witness.ova file, as shown in Figure 3-43. Click Next.

Step 3. Specify a unique name for the witness node in the Virtual Machine Name field. Select a location for the virtual machine from the drop-down list. Click Next.

Step 4. From the Select a compute resource drop-down list, choose the ESXi host where you want to deploy the witness node. Click Next.

Step 5. In the Review details pane, as shown in Figure 3-44, verify the template details and then click Next.

Step 6. In the Select Storage pane, select the datastore where the virtual machine will reside (and ensure that this datastore has at least 40 GB of available free space). Click Next.

Step 7. In the Select Networks pane, select a destination network port group to which the witness VM must connect. Click Next.

Step 8. On the Customize Template page, complete the fields that are relevant for your configuration (for example, IP address, netmask, default gateway, DNS, NTP). Click Next.

Step 9. On the Ready to complete page (see Figure 3-45), verify all the details entered. Click Finish.

HyperFlex Stretch Cluster Network IP Addressing

IP addresses for a HyperFlex stretch cluster need to be allocated from the appropriate subnets and VLANs.

Ensure that the data and management networks are on different subnets for a successful installation. In addition, ensure that the IP addressing is in the same subnet for the same components (for example, management network IP addresses, data network IP addresses).

Table 3-18 lists the IP address requirements for deploying a HyperFlex stretch cluster. Tables 3-19 and 3-20 provide examples of stretch cluster IP addressing for two sites.

HyperFlex Stretch Cluster Guidelines

Consider the following guidelines when creating a HyperFlex stretch cluster:

• HX Data Platform Enterprise Edition licensing is required to use a HyperFlex stretch cluster.

• vSphere Enterprise Plus licensing is required for full-featured HyperFlex stretch cluster functionalities such as VM load balancing and VM migration.

• Ensure that DRS is enabled.

• Support the stretch cluster scale:

  • There must be a minimum of two nodes on each site. The minimum overall cluster size across both sites is 4.

  • For a small form factor (SFF), the maximum cluster size across both sites is 64. The maximum converged node count is 16 per site. The compute-to- converged node ratio can be 2:1, and the maximum limit of nodes per site must not exceed 32. For example, you can have 11 converged and 21 compute nodes per site.

  • For a large form factor (LFF), the maximum cluster size is 48. The maximum converged node count is 8 per site. The compute-to-converged node ratio can be 2:1, and the maximum limit of nodes per site must not exceed 24. For example, you can have 8 converged and 16 compute nodes per site.

• There must be a redundant fabric interconnect configuration on each site.

• There must be symmetric cluster configuration across both sites. The number of nodes and the model of HX nodes should be the same on both sites.

• VMs are placed correctly within site affinity only if VMware high availability and DRS are enabled before VM creation. Otherwise, the correct placement of VM for proper affinity is not guaranteed.

• HyperFlex Native Replication is supported between stretch clusters and between stretch clusters and standard clusters.

• It is a best practice to have two datastores—one per site—with the appropriate affinity.

HyperFlex Stretch Cluster Limitations

Consider the following limitations when creating a HyperFlex stretch cluster:

• Self-encrypting drives (SEDs) are not supported. However, VM-based third-party software encryption is supported.

• Overlay networking and L3 protocols are not supported. L2 adjacency is required for the data and management networks.

• Auto failback of VMs after site failure is not supported. You must manually migrate the failover VMs to the site.

• A stretch cluster is not supported on Hyper-V platform.

• Online rolling upgrades are supported only for the HX Data Platform. Cisco UCS Manager upgrades and VMware ESXi upgrades must be performed manually on one node at a time or performed offline.

• Upgrading from a standalone cluster to a stretch cluster configuration is not supported.

• Stretch clusters are supported only on M5 nodes. Mixed M4/M5 clusters are not supported.

• NVMe converged nodes are not supported.

• Hardware Acceleration Engine (HX-PCIE-OFFLOAD-1) is not supported.

• A shared witness VM is not supported for stretch cluster deployments.

HyperFlex Stretch Cluster Installation

This section summarizes the steps that are involved in creating a stretch cluster using the HX Data Platform installer. All the configuration for a stretch cluster in this section will be done from a single HX installer. Figure 3-46 gives an overview of the workflow required to deploy a stretch cluster.

Figure 3-47 summarizes the installation workflow steps that are involved in creating a stretch cluster using the HX Data Platform installer.

Follow the workflow outlined in Table 3-21 during installation.

Follow these steps to install a stretch cluster:

Step 1. Log on to the HX installer to start the cluster configuration. If the installer is still showing the previous installation’s status, click on the sprocket in the bar above and select Start Over to begin a fresh install. On the Select a Workflow page, select Create Cluster > Stretch Cluster, as shown in Figure 3-48.

Step 2. Select Configure Site and enter the UCSM credentials and the site name (see Figure 3-49). Then click Continue.

Step 3. On the Server Selection page, as shown in Figure 3-50, select the source servers and click Continue. The Configuration pane on the right displays a detailed list of the credentials used. The Server Selection page also displays a list of unassociated HX servers under the Unassociated tab and a list of discovered servers under the Associated tab.

Step 4. On the UCSM Configuration page, enter the VLAN ID, VLAN names, MAC pool, IP pool for in-band of out-of-band, iSCSI storage, and FC storage (see Figure 3-51). Click Continue.

Step 5. In the Hypervisor Settings section of the UCSM Configuration page, provide all the requested information (see Figure 3-52). Then click Configure Site to begin site configuration. The Progress page displays the progress of various configuration tasks.

Step 6. Confirm that the Site A hypervisor configuration is successful, as illustrated in Figure 3-53.

Step 7. As illustrated in Figure 3-54, click on the sprocket and select Configure Site to begin the Site B configuration.

Step 8. Repeat steps 2 through 6 for Site B.

Step 9. Ensure that the witness VM has been deployed successfully. (Refer to the section “Deploying the Witness Node,” earlier in this chapter.)

Step 10. To begin configuring the stretch cluster, click the sprocket on the installer and select Create Stretch Cluster to begin the stretch cluster configuration.

Step 11. On the credentials screen (see Figure 3-55), provide the source (Site A) and target (Site B) UCSM, as well as credentials, site name, UCSM org name, and vCenter and hypervisor credentials. Click Continue to proceed to the Server Selection screen.

Step 12. On the Server Selection page (see Figure 3-56), configure the server ports and associated HyperFlex servers. Servers from both sites Site A and Site B (named DC1 and DC2) are shown under Associated Servers.

Step 13. On the IP Addresses page (see Figure 3-57), configure the IP addresses to be applied to the storage cluster. For each HX node, enter the hypervisor and storage controller IP addresses. For the IP addresses, specify the IP address belonging to the data network, the management network, the cluster IP address for the management and data networks, the subnet mask, and the IP address of the witness VM installed earlier. Click Continue.

Step 14. On the Cluster Configuration page (see Figure 3-58), configure and deploy the HyperFlex stretch cluster. On this page, you can’t change the Replication Factor setting. For a stretch cluster, it’s 2+2.

Step 15. In the Connected Services section of the Cluster Configuration page (see Figure 3-59), select Enable Connected Services to enable Auto-Support and Intersight management. In Send service ticket notifications to, enter the email address for sending SR notifications when triggered by Auto-Support.

Step 16. In the Advanced Networking section of the Cluster Configuration page (see Figure 3-60), complete the management and data VLAN tag for Site A and Site B and enter the management and data vSwitch names.

Step 17. In the Advanced Configuration section of the Cluster Configuration page (refer to Figure 3-60), configure the following:

• Jumbo frames: By selecting Enable Jumbo Frames on Data Network, you can set the MTU size for the storage data network on the host vSwitch switches and vNICs, as well as each storage controller VM. The default value is 9000.

• Disk Partitions: If you select Clean up disk partitions, you can remove all existing data and partitions from all nodes added to the storage cluster for manually prepared servers. Select this option to delete existing data and partitions. You must back up any data that should be retained.

• (Optional) vCenter Single-Sign-On Server: This information is required only if the SSO URL is not reachable.

• Virtual Desktop (VDI): Check this option for VDI-only environments.

Step 18. Click Start to begin deploying the stretch cluster. The Progress page displays the progress of the various configuration tasks, as shown in Figure 3-61. When the process is complete, the HyperFlex servers are installed and configured, and you see the message Cluster Creation Successful.

HyperFlex Stretch Cluster Postinstallation

After the installation of a stretch cluster using the HX Data Platform installer, run the postinstallation script to finalize the configuration and set up the vMotion network. You can also run this script at a future time if needed.

If you wish to run the post_install.py script later, you can run the script directly from any control VM (CVM) by using the following procedure:

Step 1. Log in to a cluster IP (CIP) through a SSH server.

Step 2. Run the post_install script, which is available at /usr/share/springpath/ storfs-misc/hx-scripts/post_install.py.

Step 3. Follow the prompts and enter the required information.