Creating a Registration Token

A registration token is used to register and consume a product for Smart Licensing. You must create a token to register the product and add the product instance to a specified virtual account. Follow these steps:

Step 1. Log in to the software manager at https://software.cisco.com/.

Step 2. In the License section, click Smart Software Licensing, as shown in Figure 5-2.

Step 3. Under Smart Software Licensing, click Inventory.

Step 4. From the virtual account where you want to register your HX storage cluster, click the General tab and then click New Token (see Figure 5-3). The Create Registration Token dialog box appears.

Step 5. In the Create Registration Token dialog box (see Figure 5-4), do the following:

a. Add a short description for the token.

b. Enter the number of days you want the token to be active and available to use on other products. The maximum is 365 days.

c. Check Allow export-controlled functionality on the products registered with this token.

d. Click Create Token.

As shown in Figure 5-5, the new token shows up under the list of tokens with the expiration date, number of uses, and user who created it.

Step 6. Select the token and copy it to the clipboard (see Figure 5-6).

Benefits of HX Data Platform Native Snapshots

HX Data Platform native snapshots provide the following benefits:

• Reverting registered VMs: If a VM is registered, whether powered on or powered off, native snapshots, just like VM snapshots, can be used to revert to an earlier point in time (that is, time when the snapshot was created).

• High performance: The HX Data Platform native snapshot process is fast because it does not incur I/O overhead.

• VM performance: HX Data Platform native snapshots do not degrade VM performance.

• Crash consistent: HX Data Platform native snapshots are crash consistent by default; this means the correct order of write operations is preserved, to enable an application to restart properly from a crash.

• Application consistent: You can select the quiesce option of the stcli vm snapshot command through the HX Data Platform CLI to enable HX Data Platform native snapshots to be application consistent. The applications in the guest VM run transparently, exactly as they do in the host VM.

  Quiescing a file system involves bringing the on-disk data of a physical or virtual computer into a state suitable for backups. This process might include operations such as flushing dirty buffers from the operating system’s in-memory cache to disk, as well as other higher-level application-specific tasks.

• Scheduled snapshots are tolerant to node failures: Scheduled snapshots are tolerant to administrative operations that require a node shutdown, such as HX maintenance mode and HX online upgrades.

• Unified interface: You can manage native snapshots created through the HX Data Platform plug-in by using the VMware snapshot manager.

• Individual or grouped: You can take native snapshots on a VM level, VM folder level, or resource pool level.

• Granular progress and error reporting: These monitoring tasks can be performed at task level for the resource pool, folder level, and VM level.

• Instantaneous snapshot delete: Deletion of a snapshot and consolidation always occur instantaneously.

• Parallel batch snapshots: HX supports up to 255 VMs in a resource pool or folder for parallel batched snapshots.

• VDI deployment support: HX scheduled snapshots are supported for desktop VMs on VDI deployments using VMware native technology.

• Recoverable VM: The VM is always recoverable when there are snapshot failures.

• Datastore access: Snapshots work on partially mounted/accessible datastores as long as the VM being snapshotted is on an accessible mountpoint.

Native Snapshot Considerations

Some snapshot parameters to consider are as follows:

• Native snapshots: After you create the first native snapshot using the HX Data Platform plug-in, if you create more snapshots in vSphere Web Client, these are considered to be native as well. However, if you create the first snapshot using vSphere Web Client and not the HX Data Platform plug-in, the vSphere Web Client snapshots are considered to be non-native.

• Maximum number of stored snapshots: Currently VMware has a limitation of 31 snapshots per VM. This maximum total includes VMware-created snapshots, the HX Data Platform SENTINEL snapshot, and HX Data Platform native snapshots.

• Scheduled snapshots: Do not schedule overlapping snapshots on VMs and their resource pools.

• Deleted VMs: The life cycle of native snapshots, as with VM snapshots, is tied to the virtual machine. If a VM is deleted, accidentally or intentionally, all associated snapshots are also deleted. Snapshots do not provide a mechanism to recover from a deleted VM. Use a backup solution to protect against VM deletion.

• HX Data Platform storage controller VMs: You cannot schedule snapshots for storage controller VMs.

• Non-HX Data Platform VMs: Snapshots fail for any VM that is not on an HX Data Platform datastore. This applies to snapshots on a VM level, VM folder level, or resource pool level. To make a snapshot, the VM must reside on an HX Data Platform datastore in an HX Data Platform storage cluster.

• Suspended VMs: Creating the first native snapshot, the SENTINEL snapshot, from VMs in suspended state is not supported.

• VM Size: The maximum size of a VM that an HyperFlex snapshot can take depends on the maximum size of the individual Virtual Machine Disk (VMDK), maximum number of attached disks, and overall size of VM.

• VM Name: The VM name must be unique per vCenter for taking a snapshot.

• Ready storage cluster: To allow a native snapshot, the storage cluster must be healthy, including sufficient space, and online. The datastores must be accessible. The VMs must be valid and not in a transient state, such as in process of vMotion.

• vMotion: vMotion is supported on VMs with native snapshots.

• Storage vMotion: Storage vMotion is not supported on VMs with native snapshots. If a VM needs to be moved to a different datastore, delete the snapshots before running Storage vMotion.

• VM datastores: Ensure that all the VM (VMDK) disks are on the same datastore prior to creating native snapshots. This applies to snapshots created with HX Snapshot Now and snapshots created with HX Scheduled Snapshots.

• Thick disks: If the source disk is thick, then the snapshot of the VM’s disk will also be thick. Increase the datastore size to accommodate the snapshot.

• Virtual disk types: VMware supports a variety of virtual disk backing types. The most common is the FlatVer2 format. Native snapshots are supported for this format. There are other virtual disk formats, such as Raw Device Mapping (RDM), SeSparse, and VmfsSparse (Redlog format). VMs containing virtual disks of these formats are not supported for native snapshots.

Native Snapshot Best Practices

Always use the HX Data Platform Snapshot feature to create your first snapshot of a VM. This ensures that all subsequent snapshots are in native format. Here are some additional recommended best practices:

• Do not use the VMware Snapshot feature to create your first snapshot.

  VMware snapshots use redo log technology that results in degraded performance of the original VM. This performance degrades further with each additional snapshot. Native format snapshots do not impact VM performance after the initial native snapshot is created. If you have any redo log snapshots, on the ESXi hosts where the redo log snapshots reside, edit the /etc/vmware/config file and set snapshot.asyncConsolidate="TRUE."

• Add all the VMDKs to the VM prior to creating the first snapshot.

  When VMDKs are added to the VM, additional SENTINEL snapshots are taken. Each additional SENTINEL consumes space for additional snapshots. For example, if you have an existing VM and you add two new VMDKs, at the next scheduled snapshot, one new SENTINEL is created. Check the snapshot schedule retention number to be sure you have sufficient snapshot slots available: one for the new SENTINEL and one for the snapshot.

• When creating large numbers of snapshots, consider the following:

  • Schedule the snapshots at a time when you expect data traffic might be low.

  • Use multiple resource pools or VM folders to group VMs rather than using a single resource pool or VM folder. Then stagger the snapshot schedule by group. For example, for resourcePool1 schedule snapshots at :00, for resourcePool2 schedule snapshots at :15, and for resourcePool3 schedule snapshots at :30.

• If you have vCenter running on a VM in the storage cluster, do not take a native snapshot of the vCenter VM.

Understanding SENTINEL Snapshots

When you create the first snapshot of a VM, through either Snapshot Now or Scheduled Snapshot, the HX Data Platform plug-in creates a base snapshot called a SENTINEL snapshot. The SENTINEL snapshot ensures that follow-on snapshots are all native snapshots.

SENTINEL snapshots prevent reverted VMs from having VMware redo log-based virtual disks. Redo log-based virtual disks occur when an original snapshot is deleted and the VM is reverted to the second-oldest snapshot.

SENTINEL snapshots are in addition to the revertible native snapshot. The SENTINEL snapshot consumes 1 snapshot of the total 31 available per the VMware limitation.

Keep in mind two important considerations when using SENTINEL snapshots:

• Do not delete the SENTINEL snapshot.

• Do not revert your VM to the SENTINEL snapshot.

Native Snapshot Timezones

Three objects display and affect the timestamps and schedule of snapshots:

• vSphere and vCenter use UTC time.

• vSphere Web Client uses the browser time zone.

• HyperFlex Data Platform components such as the HX Data Platform plug-in, storage cluster, and storage controller VM use the same configurable time zone; the default is UTC.

The storage controller VM time is used to set the schedule. The vSphere UTC time is used to create the snapshots. The logs and timestamps vary depending on the method used to view them.

Creating Snapshots

Redo log snapshots are snapshots that are created through the VMware Snapshot feature and not through the HX Data Platform Snapshot feature. If you have any redo log snapshots for VMs in an HX storage cluster, edit the ESXi host configuration where the redo log snapshots reside. If this step is not completed, VMs might be stunned during snapshot consolidation. Follow these steps to edit the ESXi host configuration:

Step 1. Log in to the ESXi host command line.

Step 2. Locate and open the file /etc/vmware/config for editing.

Step 3. Set the snapshot.asyncConsolidate parameter to TRUE (that is, snapshot.asyncConsolidate="TRUE").

Creating Snapshots Workflow

Step 1. From the vSphere Web Client navigator, select the VM level, VM folder level, or resource pool level. For example, select vCenter Inventory Lists > Virtual Machines to display the list of VMs in vCenter.

Step 2. Select a VM and either right-click the VM and click Actions or click the Actions menu in the VM information portlet.

Note

Ensure that there are no non-HX Data Platform datastores on the storage cluster resource pool, or the snapshot will fail.

Step 3. From the Actions menu, select Cisco HX Data Platform > Snapshot Now, as shown in Figure 5-10, to open the Take VM Native Snapshot for Test dialog box.

Step 4. In the Take VM Native Snapshot for Test dialog box (see Figure 5-11), enter a name for the snapshot and type a description of the snapshot. Click OK to accept your configuration.

Scheduling Snapshots

To schedule snapshots, follow these steps:

Step 1. From the vSphere Web Client navigator, select the VM or resource pool list. For example, select vCenter Inventory Lists > Virtual Machines to display the list of VMs in vCenter.

Step 2. Select a VM or resource pool and either right-click the VM or resource pool and click Actions or click the Actions menu in the VM information portlet.

Step 3. From the Actions menu, select Cisco HX Data Platform > Schedule Snapshot (see Figure 5-12) to open the Schedule Snapshot dialog box.

Step 4. Complete the Schedule Snapshot dialog box, shown in Figure 5-13, as follows:

a. To select the snapshot frequency, click the boxes for hourly, daily, and/or weekly frequency and set the starting days, times, and duration.

b. Set the number of snapshots to retain. When the maximum number is reached, older snapshots are removed as newer snapshots are created.

c. Unselect existing scheduled items, as needed. If a previous schedule existed, unselecting items deletes those items from the future schedule.

d. Click OK to accept the schedule and close the dialog.

Reverting to a Snapshot

Reverting to a snapshot means returning a VM to a state stored in a snapshot. Reverting to a snapshot is performed on one VM at a time. It is not performed at the resource pool or VM folder level. Reverting to snapshots is performed through the vCenter Snapshot Manager and not through the HX Data Platform plug-in. Follow these steps to revert to a snapshot:

Step 1. From the vSphere Web Client navigator, select the VM level, VM folder level, or resource pool level. For example, select vCenter Inventory Lists > Virtual Machines to display the list of VMs in vCenter.

Step 2. Select a storage cluster VM and either right-click the VM and click Actions or click the Actions menu in the VM information portlet.

Step 3. From the Actions menu, select Snapshots > Manage Snapshots (see Figure 5-14) to open the vSphere Snapshot Manager.

Step 4. In the Snapshot Manager, select a snapshot to revert to from the hierarchy of snapshots for the selected VM and then select All Actions > Revert to, as shown in Figure 5-15.

Step 5. Click Yes (to confirm the reversion (see Figure 5-16).

The reverted VM is included in the list of VMs and powered off. In selected cases, a VM reverted from a VM snapshot is already powered on. See Table 5-1 for more details.

Step 6. If the reverted VM is powered off, select the VM and power it on.

Deleting Snapshots

You delete snapshots through the vSphere interface and not through the HX Data Platform plug-in. Follow these steps:

Step 1. From the vSphere Web Client navigator, select VMs and Templates > vcenter_server > Snapshots > datacenter > VM.

Step 2. Right-click the VM and select Snapshots > Manage Snapshots.

Step 3. Right-click the snapshot you want to delete, click Delete.

Step 4. In the Confirm Delete dialog box that appears, click YES, as shown in Figure 5-17.

Benefits of HX Data Platform ReadyClones

The HX Data Platform ReadyClones feature provides the following benefits:

• Create multiple clones of a VM at a time: Simply right-click a VM and create multiple clones of the VM by using the ReadyClones feature.

• Rapid cloning: HX Data Platform ReadyClones is extremely fast and more efficient than legacy cloning operations because it supports VMware vSphere Storage APIs—Array Integration (VAAI) data offloads. VAAI, also called hardware acceleration or hardware offload APIs, is a set of APIs to enable communication between VMware vSphere ESXi hosts and storage devices. Use HX Data Platform ReadyClones to clone VMs in seconds instead of minutes.

• Batch customization of guest VMs: Use the HX Data Platform Customization Specification to instantly configure parameters such as IP address, hostname, and VM name for multiple guest VMs cloned from a host VM.

• Automation of several steps to a one-click process: The HX Data Platform ReadyClones feature automates the task of creating guest VMs.

• VDI deployment support: ReadyClones is supported for desktop VMs on VDI deployments using VMware native technology.

• Datastore access: ReadyClones works on partially mounted/accessible datastores as long as the VM being cloned is on an accessible mountpoint.

Supported Base VMs

HX Data Platform supports:

• Base VMs stored on an HX Data Platform datastore

• Base VMs with HX Data Platform Snapshots

• A maximum of 2048 ReadyClones from 1 base VM

• A maximum of 256 ReadyClones created in 1 batch at a time

HX Data Platform does not support:

• Powered-on base VMs with Windows 2008 Server and Windows 2012 Server guest

• Powered-on base VMs with more than 30 snapshots

• Powered-on base VMs with redo log snapshots

ReadyClones Requirements

The requirements for ReadyClones are as follows:

• VMs must be within the HX Data Platform storage cluster. Non-HX Data Platform VMs are not supported.

• VMs must reside on an HX Data Platform datastore, VM folder, or resource pool.

• ReadyClones fail for any VM that is not on an HX Data Platform datastore. This applies to ReadyClones on a VM level, VM folder level, or resource pool level.

• VMs can have only native snapshots. ReadyClones cannot be created from VMs with snapshots that have redo logs (that is, non-native snapshots).

• SSH must be enabled in ESXi on all the nodes in the storage cluster.

• You can use only the single vNIC customization template for ReadyClones.

ReadyClones Best Practices

When working with ReadyClones, keep the following best practices in mind:

• Use the customization specification as a profile or a template.

• Ensure that properties that apply to the entire batch are in the customization specification.

• Obtain user-defined parameters from the HX Data Platform ReadyClones batch cloning workflow.

• Use patterns to derive per-clone identity settings such as the VM guest name.

• Ensure that the network administrator assigns static IP addresses for guest names and verify these addresses before cloning.

• You can create a batch of 1 through 256 at a given time.

• Do not create multiple batches of clones simultaneously on the same VM (when it is powered on or powered off) because doing so causes failures or displays incorrect information on the master task updates in the HX Data Platform plug-in.

Creating ReadyClones Using HX Connect

Use the HX Data Platform ReadyClones feature to populate a cluster by creating multiple clones of a VM, each with a different static IP address. Follow these steps:

Step 1. Log in to HX Connect as an administrator.

Step 2. From the Virtual Machines page, select a virtual machine and then click ReadyClones, as shown in Figure 5-18.

Step 3. Complete the ReadyClones dialog box, shown in Figure 5-19, as outlined in Table 5-2.

Step 4. Click Clone. HX Data Platform creates the appropriate number of ReadyClones with the naming and location specified.

Creating ReadyClones Using the HX Data Platform Plug-in

If you use the VMware cloning operation, you can create only a single clone from a VM. This operation is manual and slower than batch processing multiple clones from a VM. For example, to create 20 clones of a VM, you must manually perform the clone operation over and over again. Follow these steps to create ReadyClones using the HX Data Platform plug-in:

Step 1. From the vSphere Web Client navigator, select Global Inventory Lists > Virtual Machines to open a list of VMs in vCenter.

Step 2. Select a VM and either right-click the VM and click Actions or click the Actions menu in the VM information portlet.

Step 3. From the Actions menu, select Cisco HX Data Platform > ReadyClones, as shown in Figure 5-20.

The ReadyClones dialog box appears, as shown in Figure 5-21.

Step 4. Enter any changes you want to make and click OK to apply these configuration changes.

Adding Datastores

Datastores are logical containers, similar to file systems, that hide specifics of physical storage and provide a uniform model for storing VM files. You can also use datastores to store ISO images and VM templates. To add a datastore, follow these steps:

Step 1. Choose an interface using either of these methods:

• From the vSphere Web Client navigator, select vCenter Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Manage > Datastores.

• From HX Connect, select Datastores.

Step 2. Click on create datastore.

Step 3. Enter a name for the datastore. vSphere Web Client enforces a 42-character limit for the datastore name, and each datastore name needs to be unique.

Step 4. Specify the datastore size and choose GB or TB from the drop-down list.

Step 5. Specify the data block size. From HX Connect, choose 8K or 4K; the default is 8K. In the HX Data Platform plug-in, the default is assumed. For VDI workloads, the default is 4k.

Step 6. Click OK to accept your changes or Cancel to cancel all changes.

Step 7. To verify the addition of the datastore, click the Refresh icon and ensure that the new datastore is listed. From the HX Data Platform plug-in, Click the Manage > Datastores > Hosts to see the mount status of the new datastore. If you check the datastore through the vSphere Client application, by selecting host > Configuration> Datastores, the drive type is listed as Unknown; this is expected vSphere behavior.

Creating Datastores Using the HX Data Platform Plug-in

The workflow in Figure 5-22 shows how to create datastores using the HX Data Platform plug-in on VMWare vCenter.

Creating Datastores Using HX Connect

Figure 5-23 shows how to create a datastore using HX Connect.

Adding a Converged Node

You can add a converged node to a HyperFlex cluster after cluster creation. The storage on a converged node is automatically added to the cluster’s storage capacity.

Follow these steps to add a converged node to an existing standard cluster:

Step 1. Launch the Cisco HX Data Platform installer. (Use the same version of installer as the version of the HX cluster.)

Step 2. On the Workflow page, select Expand Cluster > Standard Cluster, as shown in Figure 5-24.

Step 3. On the Credentials page that appears, complete all the fields, as shown in Figure 5-25, and click Continue.

Step 4. On the Cluster Expand Configuration page that appears (see Figure 5-26), select the HX cluster that you want to expand and click Continue.

Step 5. On the Server Selection page, review the list of unassociated HX servers under the Unassociated tab (see Figure 5-27) and the list of discovered servers under the Associated tab. Select the servers under the Unassociated tab to include in the HyperFlex cluster. Click Continue.

If HX servers do not appear in this list, check Cisco UCS Manager and ensure that they have been discovered.

Step 6. On the UCSM Configuration page that appears, complete the fields for each network and configure the HyperFlex cluster name, as shown in Figure 5-28, and click Continue.

Step 7. On the Hypervisor Configuration page that appears (see Figure 5-29), complete all the fields and click Continue.

Step 8. On the IP Addresses page that appears (see Figure 5-30), add more compute or converged servers, as desired, by clicking Add Compute Server or Add Converged Server. Be sure to select Make IP Addresses Sequential to make the IP addresses sequential. For the IP addresses, specify whether the network should belong to the data network or the management network. For each HX node, complete the appropriate fields for hypervisor management and data IP addresses. When you’re finished with the settings on this page, click Start. A Progress page displays the progress of various configuration tasks.

Step 9. When cluster expansion is complete, start managing your storage cluster by clicking Launch HyperFlex Connect.

Adding a Compute Node

You can add a compute-only node to a HyperFlex cluster after cluster creation to provide extra compute resources. The Cisco UCS server does not need to have any caching or persistent drives as they do not contribute any storage capacity to the cluster. Use similar steps for adding a compute node as used for expanding a HyperFlex cluster with a converged node.

Minor changes in the workflow as compared to converged node include the following:

• Select Compute Only Server under Server Selection.

• In the IP Addresses section, compute-only nodes do not need storage controller management and storage controller data.

Figure 5-31 shows the configuration option when adding a compute server.

Configuring Sites for Expanding a Cluster

Before you can expand a cluster, you need to re-create the sites in the installer as they are deployed using this procedure. Follow these steps:

Step 1. Log in to the Cisco HX Data Platform installer.

Step 2. On the Select a Workflow page, select Expand Cluster > Stretch Cluster, as shown in Figure 5-32, and click Continue.

Step 3. On the Cluster page that appears, enter the cluster management hostname, as shown in Figure 5-33, and click Continue.

Step 4. On the Credentials page that appears, as shown in Figure 5-34, select Configure Site and then enter UCS manager and hypervisor credentials. Click Continue.

Step 5. On the Server Selection page that appears, configure the server ports and associate the new HX expansion nodes with the site, as shown in Figure 5-35, and click Continue.

Step 6. On the Node Configuration page that appears, configure the subnet mask, gateway, and hypervisor settings as shown in Figure 5-36, and click Start to begin site configuration for the expanded cluster. A progress page displays the progress of various configuration tasks.

Step 7. Repeat steps 1 through 7 for the second site.

Expanding a Cluster

To expand a cluster, follow these steps:

Step 1. On the Cluster page, as shown in Figure 5-37, enter the cluster management hostname and click Continue.

Step 2. On the Credentials page that appears, as shown in Figure 5-38, select Expand Stretch Cluster, enter the credentials information, and click Continue.

Step 3. Configure the server ports and associate HyperFlex servers.

Step 4. On the IP Addresses page, as shown in Figure 5-39, configure the hypervisor and IP addresses, select the site, and click Start to start the cluster expansion process.

Removing Converged Nodes

Depending on the node maintenance task, removing a node can occur while the storage cluster is online or offline. Ensure that you have completed the preparation steps before removing a node.

The steps to take in removing a node depend on the cluster size. Table 5-3 provides an overview of the steps for removing clusters of different sizes.

Removing a Node from an Online Storage Cluster

Depending on the node maintenance task, removing a node can occur while the storage cluster is online or offline. Removing a node from a storage cluster while the cluster remains online has slightly different requirements from removing a node while a cluster is offline. Follow these steps to remove a node from an online storage cluster:

Note

It is highly recommended that you work with TAC when removing a converged node in a storage cluster. Do not remove the controller VM or other HX Data Platform components.

Step 1. To prepare to remove a node, do the following:

• Ensure that the cluster is healthy by entering the stcli cluster info command.

• Ensure that SSH is enabled in ESX on all the nodes in the storage cluster.

• Ensure that DRS is enabled or manually move the VMs from the node.

• Put the node being removed into HX maintenance mode.

• Log in to the controller VM of a node that is not being removed.

Step 2. Rebalance the storage cluster to ensure that all datastores associated with the node will be removed. The rebalance command is used to realign the distribution of stored data across changes in available storage and to restore storage cluster health. If you add or remove a node in the storage cluster, you can manually initiate a storage cluster rebalance by using the stcli rebalance command.

Note

Rebalancing might take some time, depending on the disk capacity used on the failed node or disk.

Log in to a controller VM in the storage cluster. From the controller VM command line, run the stcli rebalance start --force command and then wait and confirm that rebalance has completed.

Step 3. Remove the desired node by using the stcli node remove command, which has the following syntax:

Click here to view code image

stcli node remove [-h] {--id-1 ID1 | --ip-1 NAME1}
[{--id-2 ID2 | --ip-2 NAME2}] [-f]

For example:

Click here to view code image

# stcli node remove --name-1 esx.SVHOST144A.complab

The response should be something like the following:

Click here to view code image

Successfully removed node: EntityRef(type=3, id='', name=
'esx.SVHOST144A.complab')

Figure 5-40 shows this command in use.

Note

The stlci node remove command unmounts all datastores, removes node from the cluster ensemble, resets the EAM for this node, stops all services (stores, cluster management IP), and removes all firewall rules. This command does not remove the node from vCenter; the node remains in vCenter. It also does not remove the installed HX Data Platform elements, such as the controller VM.

When the stcli node remove command completes successfully, the system rebalances the storage cluster until the storage cluster state is healthy. Do not perform any failure tests during this time. The storage cluster remains healthy.

As the node is no longer in the storage cluster, you do not need to exit HX maintenance mode.

Note

If you want to reuse a removed node in another storage cluster, contact Cisco TAC. Additional steps are required to prepare the node for another storage cluster.

Step 4. To confirm that the node has been removed from the storage cluster, run this command:

# stcli cluster info

Check the ActiveNodes entry in the response to verify that the cluster has one less node.

Step 5. Confirm that all the node-associated datastores are removed.

Note

If any node-associated datastores are listed, then manually unmount and delete those datastores.

Step 6. Remove the host from the vCenter Hosts and Cluster view as follows:

a. Log in to vSphere Web Client navigator and navigate to Host in the vSphere Inventory.

b. Right-click the host and select Enter Maintenance Mode. Click Yes.

c. Right-click the host and select All vCenter Actions > Remove from Inventory. Click Yes.

Step 7. Decommission the host from UCS Manager as follows:

a. Log in to UCS Manager.

b. In the navigation pane, Select Equipment > Chassis > Chassis Number > Servers.

c. Choose the HX server you want to decommission. In the work pane, click the General tab.

d. In the Actions area, click Server Maintenance.

e. In the Maintenance dialog box, click Decommission and then click OK.

# stcli node remove --ip-1 10.10.2.4 --ip-2 10.10.2.6

# stcli node remove --name-1 esx.SVHOST144A.complab --name-2
esx.SVHOST144B.complab.lab

Successfully removed node: EntityRef(type=3, id='',
name='10.10.2.4' name='10.10.2.6')

# stcli cluster info

Removing a Compute Node

Compute-only nodes do not contribute to storage in a HyperFlex cluster. Therefore, when removing compute nodes, there is no limitation on the number of nodes you can remove. The steps to remove a compute node are as follows:

Step 1. Migrate all the VMs from a compute node that needs to be removed.

Step 2. Unmount the datastore from the compute node.

Step 3. Check whether the cluster is in the healthy state by running the following command:

Click here to view code image

# stcli cluster info --summary

Step 4. Put the ESXi host in HX maintenance mode.

Step 5. Remove the compute node by using the stcli node remove command from CMIP:

# stcli node remove --ip-1

Where --ip is the IP address of the node to be removed. (Use the Cisco HX Connect IP address as it is the cluster IP address.)

Step 6. Remove any DVS from the ESXi host in vCenter, if there is a DVS.

Step 7. Remove the ESXi host from vCenter.

Step 8. Check whether the cluster is in the healthy state by running the following command:

Click here to view code image

# stcli cluster info --summary

Step 9. Clear stale entries in the compute node by logging out of Cisco HX Connect and then logging into Cisco HX Connect.

Step 10. Disable and reenable the High Availability (HA) and Distributed Resource Scheduler (DRS) services to reconfigure the services after node removal.

Replacing NVMe SSDs

The procedures for replacing an SSD vary depending on the type of SSD. This section describes the steps for replacing NVMe cache SSDs.

The requirements for using NVMe in HX are as follows:

• NVMe SSDs are supported in HX240 and HX220 All Flash servers.

• Replacing NVMe SSDs with an HGST SN200 disk requires HX Data Platform version 2.5.1a or later.

• NVMe SSDs are only allowed in slot 1 of the server. Other server slots do not detect NVMe SSDs.

• NVMe SSDs are only used for cache. Using them for persistent storage is not supported.

• Using an NVMe SSD as the housekeeping drive is not supported.

• Using NVMe SSDs for hybrid servers is not supported.

The steps for replacing NVMe SSDs are as follows:

Step 1. Confirm that the failed disk is an NVMe cache SSD by performing a physical check. These drives are in Drive Bay 1 of the HX servers. NVMe cache SSDs and housekeeping SSDs do not respond to beacon requests. If the failed SSD is not an NVMe SSD, see the earlier section “Replacing SSDs.”

Step 2. Put the ESXi host into HX maintenance mode by logging in to HX Connect and selecting System Information > Nodes > node > Enter HX Maintenance Mode.

Step 3. Follow the instructions for removing and replacing a failed SSD in the host, per the server hardware guide.

After the cache or persistent drive is replaced, HX Data Platform identifies the SDD and updates the storage cluster. When disks are added to a node, the disks are immediately available for HX consumption.

Step 4. Reboot the ESXi host. This enables ESXi to discover the NVMe SSD.

Step 5. Exit the ESXi host from HX maintenance mode.

Step 6. To enable the Cisco UCS Manager to include new disks in the UCS Manager > Equipment > Server > Inventory > Storage tab, re-acknowledge the server node. This applies to cache and persistent disks.

Step 7. If you replaced an SSD and see the message “Disk successfully scheduled for repair,” it means that the disk is present but is still not functioning properly. Check that the disk has been added correctly per the server hardware guide procedures.

Replacing Housekeeping SSDs

This procedure applies to HXAF220c M4, HX220c M4, HXAF220c M5, HX220c M5, HXAF240c M5, and HX240c M5 servers only. To replace the housekeeping SSD on an HXAF240c M4 or HX240c M4 server, contact Cisco TAC.

To replace a housekeeping SSD, perform the following steps:

Step 1. Identify the failed housekeeping SSD by physically checking the SSD drives, as housekeeping drives are not listed through a beacon check.

Step 2. Remove the SSD and replace it with a new SSD of the same kind and size. Follow the steps in the server hardware guide, which should describe the physical steps required to replace the SSD.

Step 3. Using SSH, log in to the storage controller VM of the affected node and run the following command:

Click here to view code image

# /usr/share/springpath/storfs-appliance/config-bootdev.sh
-r -y

This command consumes the new disk, adding it into the storage cluster.

A sample response might resemble the following:

Click here to view code image

Creating partition of size 65536 MB for /var/stv ...
Creating ext4 filesystem on /dev/sdg1 ...
Creating partition of size 24576 MB for /var/zookeeper ...
Creating ext4 filesystem on /dev/sdg2 ...
Model: ATA INTEL SSDSC2BB12 (scsi)
Disk /dev/sdg: 120034MB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt ....
discovered. Rebooting in 60 seconds

Step 4. Wait for the storage controller VM to automatically reboot.

Step 5. When the storage controller VM completes its reboot, verify that partitions are created on the newly added SSD by running the df -ah command.

A sample response might resemble the following:

Click here to view code image

...........
/dev/sdb1 63G 324M 60G 1%
/var/stv /dev/sdb2 24G 173M 23G 1% /var/zookeeper

Step 6. Identify the HX Data Platform installer package version installed on the existing storage cluster by running the stcli cluster version command. The same version must be installed on all the storage cluster nodes. Run this command on the controller VM of any node in the storage cluster but not the node with the new SSD.

Step 7. Copy the HX Data Platform installer packages into the storage controller VM in the /tmp folder:

Click here to view code image

# scp <hxdp_installer_vm_ip>:/opt/springpath/packages/storfs-
packages-<hxdp_installer>.tgz /tmp

# cd /tmp

# tar zxvf storfs-packages-<hxdp_installer>.tgz

Step 8. Run the HX Data Platform installer deployment script:

# ./inst-packages.sh

Step 9. After the package installation, the HX Data Platform starts automatically. Check the status:

# status storfs
storfs running

The node with the new SSD rejoins the existing cluster, and the cluster returns to a healthy state.

Using UCS Manager

Using Cisco UCS Manager to verify the current UCS server firmware, follow these steps:

Step 1. Log in to UCS Manager.

Step 2. Select the Server tab.

Step 3. Select the host firmware package policy by navigating to Policies > Root > Sub-Organizations > <hx-cluster> > Host Firmware Packages > HyperFlex.

Step 4. Under properties, note the current rack package version, which should be listed as X.Y(Z)C. For example, Figure 5-46 shows version 4.0(2d)C.

Using HX Connect

Using Cisco Hyperflex Connect to verify the current UCS server firmware, follow these steps:

Step 1. Log in to HX Connect.

Step 2. In the navigation pane, select Upgrade.

Step 3. Select the UCS Firmware checkbox and click Discover.

Step 4. Note the current C-bundle version displayed. Figure 5-47 shows an example where the current server firmware is 4.0(2d).

Viewing the HyperFlex Cluster Health

From HyperFlex Connect, select System Information > Nodes page. Verify if the HyperFlex cluster is healthy and online.

From the vSphere Web Client navigator, select vCenter Global Inventory Lists > Cisco HyperFlex Systems > Cisco HX Data Platform > cluster > Summary. View the cluster widget to verify if the HyperFlex cluster is healthy and online.

Verify whether all HX cluster nodes are connected to the vCenter and whether they are online.

Checking the Cluster Storage Capacity

It is recommended that you check the cluster storage capacity before starting the upgrade of an existing installation of Cisco HX Data Platform. If the storage cluster capacity is above 70%, it is highly recommended to either reduce the amount of storage capacity used or increase the storage capacity by adding new nodes or disks. This confirmation of cluster storage capacity is important because if a node goes down in such a situation, the cluster will not be able to rebalance and will stay unhealthy (online).

Verifying That DRS Is Enabled

Follow these steps to ensure that DRS is enabled on the node:

Step 1. From the vSphere Web Client navigator, select vCenter Inventory Lists > Clusters > cluster > Configure tab. Verify that DRS is enabled.

Step 2. Click the vSphere DRS tab. Ensure that Migration Automation Level is set to Fully Automated, as shown in Figure 5-48.

Verifying and Configuring the Net.TeamPolicyUpDelay Default Value

To avoid loss of storage access during fabric interconnect reboots for firmware updates, perform this check prior to UCSM infrastructure upgrade.

Upgrades to 3.5(2) require that the default value of the ESXi host Net.TeamPolicyUpDelay be set to 30000. Complete the following steps to verify and, if needed, modify the default value of the ESXi host Net.TeamPolicyUpDelay to 30000 (see Figure 5-49):

Step 1. From the vSphere Web Client navigator, select ESXi Host > Configure > System > Advanced System Settings.

Step 2. In Advanced System Settings, scroll down to Net.TeamPolicyUpDelay.

Step 3. If needed, change the value to 30000. The default value is 100.

Viewing ESX Agent Manager

From the vSphere Web Client navigator, select Administration > vCenter Server Extensions > vSphere ESX Agent Manager > Configure > ESX Agencies.

Verify that the ESX Agent Manager (EAM) status is normal, as shown in Figure 5-50.

Verifying the Health of a HyperFlex Cluster in Cisco UCS Manager

Cisco UCS Manager health checks help with verifying the status of the UCS infrastructure and servers before you perform an upgrade. Use the following steps for verification:

Step 1. Verify whether the high availability status of the fabric interconnects shows that both fabric interconnects are up and running. Log in to UCSM and browse to Equipment > Fabric Interconnect > FIA > Expand High Availability Details and verify that the Ready and State fields show Yes and Up, respectively, as shown in Figure 5-51.

Step 2. Verify that the data path is up and running by entering the following commands:

a. To enter NX-OS mode for the fabric interconnect, enter the following command:

Click here to view code image

UCS-A /fabric-interconnect # connect nxos {a | b }

b. To determine the number of active Ethernet interfaces, enter the following command:

Click here to view code image

UCS-A(nxos)# show int br | grep -v down | wc –l

Verify that the number returned matches the number of Ethernet interfaces that were up prior to the upgrade.

c. To determine the total number of MAC addresses, enter the following command:

Click here to view code image

UCS-A(nxos)# show platform fwm info hw-stm | grep '1.' |
wc –l

Verify that this number matches the number of MAC addresses prior to the upgrade.

See the Cisco UCS Manager Firmware Management Guide (https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/GUI-User-Guides/Firmware-Mgmt/3-2/b_UCSM_GUI_Firmware_Management_Guide_3_2.html) for more information.

Step 3. Verify that the HyperFlex servers have no faults that might impact an upgrade. For example, Figure 5-52 shows an example of a HyperFlex server on Cisco UCSM that has status OK and doesn’t have any critical, major, or minor alerts.

Verifying vMotion Interfaces

Make sure the vMotion VMkernel interfaces necessary for vMotion to work are configured on each ESXi host. You must predefine vMotion networking by creating a vSwitch switch and defining the vNICs and VLANs in UCS Manager.

Before you perform maintenance operations on a Cisco HyperFlex cluster, you need to verify that all nodes in the HX cluster are configured for vMotion.

Verifying Upstream Network Connectivity

You need to ensure that the hx-storage-data and vMotion upstream switches are configured for jumbo frames. Skipping this step could lead to input/output interruption during Cisco UCS infrastructure upgrade. Follow this process:

Step 1. Put a node in Cisco HX maintenance mode.

Step 2. SSH to the ESXi host in step 1.

Step 3. Verify that ping is working by pinging the corresponding vmk1 IP interface of another host.

If using jumbo frames, use the following command:

Click here to view code image

# vmkping -I vmk1 -d -s 8972 <data IP of address of another
host>

If not using jumbo frames, use the following command:

Click here to view code image

# vmkping -I vmk1 -d -s 1472 <data IP of address of another
host>

Step 4. Swap the active interfaces in vswitch-hx-storage-data to force traffic upstream:

Click here to view code image

# esxcli network vswitch standard policy failover set -a
vmnic1 -s vmnic5 -v vswitch-hx-storage-data

Step 5. Again verify that the ping is working by pinging the corresponding vmk1 IP interface of another host.

If using jumbo frames, use the following command:

Click here to view code image

# vmkping -I vmk1 -d -s 8972 <data IP of address of another
host>

If the ping fails, try again with this command:

Click here to view code image

# vmkping -I vmk1 -d -s 1472 <data IP of address of another
host>

If not using jumbo frames, use the following command:

Click here to view code image

# vmkping -I vmk1 -d -s 1472 <data IP of address of another
host>

Step 6. Return the interface back to the defaults even if the ping fails:

Click here to view code image

# esxcli network vswitch standard policy failover set -a
vmnic5 -s vmnic1 -v vswitch-hx-storage-data

Step 7. Exit the node from Cisco HX maintenance mode.

Configuring the Cluster Access Policy in Lenient Mode

The cluster access policy works with the data replication factor to set levels of data protection and data loss prevention. There are two cluster access policy options:

• Strict mode: Applies policies to protect against data loss. If nodes or disks in the storage cluster fail, the cluster’s ability to function is affected. If more than one node fails or one node and disk(s) on a different node fail, this is called a simultaneous failure. The strict setting helps protect the data in the event of simultaneous failures.

• Lenient mode: Applies policies to support longer storage cluster availability. This is the default.

The mode is not configurable during installation but can be changed after installation and initial storage cluster configuration.

To modify the cluster access policy to lenient, use the following procedure:

Step 1. SSH to any one of the controller VMs and log in as root.

Step 2. Check whether lenient mode is already configured:

Click here to view code image

#stcli cluster get-cluster-access-policy

Step 3. If the mode is set to strict, change it to lenient:

Click here to view code image

#stcli cluster set-cluster-access-policy --name lenient

If it is already set to lenient, no further action is required.

Step 4. Confirm the change:

Click here to view code image

# stcli cluster info | grep -i policy

Verifying That No Major Alarms Are Reported for the HyperFlex Cluster in HyperFlex Connect

Log in to HX Connect and browse to Alarms and, verify that there are major alarms, as shown in Figure 5-53.

Storage Controller VM and ESXi Node Checks

Hypercheck performs the following checks on each storage controller VM:

• Cluster services check: Verifies the status of the storfs, stMgr, and stNodeMgr services.

• Enospc state check: Checks whether the cluster space usage is above the warning threshold.

• Zookeeper check: Checks whether Zookeeper is running.

• Exhibitor check: Verifies the status of the Exhibitor service, which manages Zookeeper.

• HDD health check: Reports if there is any blacklisted disk in a cluster.

• DNS check: Checks whether DNS is configured and reachable.

• vCenter reachability check: Checks whether vCenter is reachable on the required ports.

• Timestamp check: Checks whether all the controller VMs have exactly the same time.

• NTP sync check: Checks whether NTP is reachable from the storage controller VMs.

• Check package and versions: Checks for packages and versions on controller VMs.

• Check iptables count: Compares Iptables counts on all controller VMs.

• Cluster upgrade check: Checks whether there are any previous stale upgrade entries.

• Extra pnodes check: Looks for any extra/duplicate pnode entries in the cluster.

• Disk usage (/var/stv) check: Checks whether the utilization of /var/stv is more than 80%.

• Disk usage (/var/zookeeper) check: Checks the utilization of /var/zookeeper.

• Out of memory check: Checks through the log files if the cluster had any out-of-memory events.

• Supported vSphere check: Lists supported vSphere Versions with current HX Data Platform version.

• Network checks: Checks the connectivity between the management and storage networks.

Hypercheck performs the following checks on each ESXi node:

• HX user account check: Verifies whether hxuser is on all the hosts and has admin rights.

• vMotion enabled check: Checks whether the vMotion network is configured.

• vMotion reachability check: Checks the connectivity between vMotion networks.

• Check for ESXI failback timer: Checks for the ESXi failback timer on ESXi hosts.

• Network check: Sends a ping to vmk0, eth0, and eth1 (between the management and storage networks).

• Dumpfile in springpathDS: Checks whether SpringpathDS is configured for a coredump.

• VMware tools location check: Checks for VMWare tools location if it was modified from default.

Installing and Running Hypercheck

To install and run Hypercheck, perform the following steps:

Step 1. Download HyperFlex-Hypercheck.zip from https://github.com/CiscoDevNet/HyperFlex-Hypercheck, as shown in Figure 5-54.

Always download the latest version of the tool before you use it. This tool is enhanced frequently, and using older version might result in missing important checks.

Step 2. Upload the tool to the Storage Controller VM (SCVM) with the Cluster Management IP (CMIP). Use your preferred method (scp/sftp/ftp/tftp) to copy the HyperFlex-Hypercheck.zip to the /tmp directory.

Step 3. Extract the contents of HyperFlex-Hypercheck.zip:

a. Type cd /tmp to change to the /tmp directory:

Click here to view code image

root@SpringpathController7PVQWP6WV1:~# cd /tmp/

b. Type unzip HyperFlex-Hypercheck.zip to extract the files:

Click here to view code image

root@SpringpathController7PVQWP6WV1:/tmp# unzip
HyperFlex-Hypercheck.zip

Archive:  HyperFlex-Hypercheck.zip
b61c59f7962b72902692ce70548ba3d760efdf06
   creating: HyperFlex-Hypercheck/
  inflating: HyperFlex-Hypercheck/HXTool.py
  inflating: HyperFlex-Hypercheck/LICENSE.txt
  inflating: HyperFlex-Hypercheck/ReadMe.txt
  inflating: HyperFlex-Hypercheck/TestInfo.txt
  inflating: HyperFlex-Hypercheck/prettytable.py
  inflating: HyperFlex-Hypercheck/progressbar.py
root@SpringpathController7PVQWP6WV1:/tmp#

Step 4. Execute the HXTool Python script:

a. Type cd HyperFlex-Hypercheck to go to the HyperFlex-Hypercheck directory:

Click here to view code image

root@SpringpathControllerABCDE01234:/tmp# cd HyperFlex-
Hypercheck

b. Type python HXTool.py to execute the script and provide the root passwords for the controller VM and ESX:

Click here to view code image

root@SpringpathControllerABCDE01234:/tmp/HyperFlex-
Hypercheck# python HXTool.p
              .   HX Health Check 3.8
Please enter below info of HX-Cluster:

Enter the HX-Cluster Root Password:

Enter the ESX Root Password:

The tool then starts the automated checks on HyperFlex and takes around three to five minutes to complete the checks on a four-node cluster (though the duration varies depending on the cluster size).

Example 5-1 shows sample HXTool output from a three-node cluster. It shows the progress bar during the process and gives a binary PASS/FAIL for each of the checks performed. If one or more checks fail, perform the corresponding corrective action or contact Cisco TAC to fix the issue before proceeding with the upgrade process.

Example 5-1 Sample HXTool Output from a Three-Node Cluster

Click here to view code image

root@SpringpathControllerABCDE01234:/tmp/HyperFlex-Hypercheck# python HXTool.py

                 HX Health Check 3.8

Please enter below info of HX-Cluster:
Enter the HX-Cluster Root Password:
Enter the ESX Root Password:

SSH connection established to HX Node: 10.X.X.79
SSH connection established to HX Node: 10.X.X.80
SSH connection established to HX Node: 10.X.X.81

<Output Truncated ...>

HX Controller: 10.X.X.79
Cluster services check          [########################] PASS
ZooKeeper & Exhibitor check     [########################] PASS
HDD health check                [########################] PASS
Pre-Upgrade Check               [########################] PASS
Network check                   [########################] PASS

<Similar Output Truncated ...>

HX Controller: 10.X.X.79
Test Summary:
+----------------------------------+-----------+----------+
| Name                             | Result    | Comments |
+----------------------------------+-----------+----------+
| Cluster services check           | PASS      |          |
+----------------------------------+-----------+----------+
| Enospc state check               | PASS      |          |
+----------------------------------+-----------+----------+
| Zookeeper check                  | PASS      |          |
+----------------------------------+-----------+----------+
| Exhibitor check                  | PASS      |          |
+----------------------------------+-----------+----------+
| ZooKeeper Disk Usage             | PASS      |          |
+----------------------------------+-----------+----------+
| HDD health check                 | PASS      |          |
+----------------------------------+-----------+----------+
| DNS check                        | PASS      |          |
+----------------------------------+-----------+----------+
| vCenter reachability check       | PASS      |          |
+----------------------------------+-----------+----------+
| Timestamp check                  | PASS      |          |
+----------------------------------+-----------+----------+
| NTP sync check                   | PASS      |          |
+----------------------------------+-----------+----------+
| Check package & versions         | PASS      |          |
+----------------------------------+-----------+----------+
| Check Iptables count             | PASS      |          |
+----------------------------------+-----------+----------+
| Cluster upgrade status           | PASS      |          |
+----------------------------------+-----------+----------+
| Extra pnodes check               | PASS      |          |
+----------------------------------+-----------+----------+
| Memory usage check               | PASS      |          |
+----------------------------------+-----------+----------+
| Incidence of OOM in the log file | PASS      |          |
+----------------------------------+-----------+----------+
| Supported vSphere versions       | 6.0.0-U3  |          |
|                                  | 6.5.0-U1  |          |
|                                  | 6.5.0-U2  |          |
|                                  | 6.5.0-U3  |          |
|                                  | 6.7.0-UGA |          |
|                                  | 6.7.0-U1  |          |
|                                  | 6.7.0-U2  |          |
|                                  | 6.7.0-U3  |          |
+----------------------------------+-----------+----------+
| Check permissions for /tmp       | PASS      |          |
+----------------------------------+-----------+----------+

<Similar Output Truncated ...>

ESX Host: 10.X.X.76
+--------------------------------------+--------+----------+
| Name                                 | Result | Comments |
+--------------------------------------+--------+----------+
| HX User Account check                | PASS   |          |
+--------------------------------------+--------+----------+
| vMotion enabled check                | PASS   |          |
+--------------------------------------+--------+----------+
| vMotion reachability check           | PASS   |          |
+--------------------------------------+--------+----------+
| Check for ESXI Failback timer        | PASS   |          |
+--------------------------------------+--------+----------+
| Check ping to vmk0, eth0, eth1       | PASS   |          |
+--------------------------------------+--------+----------+
| No extra controller vm folders check | PASS   |          |
+--------------------------------------+--------+----------+
| VMware Tools location check          | PASS   |          |
+--------------------------------------+--------+----------+

<Similar Output Truncated ...>

Main Report File: HX_Tool_Main_Report_15-03-2020_05-24-12_.txt
Report tar file: HX_Report_15_03_2020_17_15_17.tar
Report file copied to path: /var/log/springpath

Release Notes:
https://www.cisco.com/c/en/us/support/hyperconverged-systems/HyperFlex-hx-data-
  platform-software/products-release-notes-list.html

Upgrade Guides:
https://www.cisco.com/c/en/us/support/hyperconverged-systems/HyperFlex-hx-data-
  platform-software/products-installation-guides-list.html

Note:
1) Please check the status of Compute nodes manually, script only verifies the
   config on the converged nodes.
2) Hypercheck doesnot perform FAILOVER TEST, so please ensure that the upstream is
   configured for network connectivity for JUMBO or NORMAL MTU size as needed.

Be sure to resolve all the issues identified during the preceding verification process and then proceed to the next step of performing the HyperFlex upgrade procedure.

Recommended Upgrade Method

For both the combined upgrade and the split upgrade, Cisco recommends upgrading the HyperFlex components in the following order for optimizing the upgrade time:

1. Upgrade the Cisco UCS infrastructure.

2. Upgrade Cisco HX Data Platform.

3. Upgrade Cisco customized VMware ESXi.

4. Upgrade the Cisco UCS firmware.

Online Upgrade Process

When using the online upgrade process workflow, consider the following:

• First upgrade the Cisco UCS infrastructure to the latest version and then use the automated upgrade workflow for a combined upgrade of Cisco UCS firmware and Cisco HX Data Platform. An online upgrade uses host firmware packages to upgrade all server endpoints.

• During an online upgrade, as one node is being upgraded (placed into maintenance mode), the number of tolerated node failures is reduced based on the Data Replication Factor and Access Policy settings.

• If upgrading both HX Data Platform and the UCS firmware, a combined upgrade can be selected through HX Connect, depending on the length of the maintenance window.

Upgrading a HyperFlex Cluster Using the HX Connect UI

To upgrade a HyperFlex cluster using the HX Connect UI, follow these steps:

Step 1. Use Auto Bootstrap and Upgrading Management Packages to upgrade Cisco HX Data Platform. Log in to HX Connect, navigate to https://<storage- cluster-management-ip>, enter the administrative username and password, and click Login.

Step 2. In the navigation pane, select Upgrade. The Select Upgrade Type page appears, as shown in Figure 5-61.

Step 3. Upload the new Cisco HX Data Platform upgrade package, enter your vCenter credentials, as illustrated in Figure 5-62 and Figure 5-63, and then click Upgrade.

Step 4. In the Initiating Pre-Upgrade message box that appears (see Figure 5-64), click Confirm. The Pre-Upgrade Process auto bootstrapping and updating management plug-in begins to run on all nodes, as shown in Figure 5-65.

Step 5. When the Pre-Upgrade process is complete, and the HX Connect UI prompts you with the onscreen message shown in Figure 5-66, click Upgrade again to complete the HX Data Platform part of the upgrade.

Note

The cluster upgrade is not complete until the full upgrade is initiated. The upgrade is only partially complete after the pre-upgrade auto bootstrapping steps (steps 1 through 5).

Step 6. Choose the type of upgrade from the Select Upgrade Type page:

• HX Data Platform and UCS Server Firmware

• HX Data Platform and ESXi

• HX Data Platform, UCS Server Firmware, and ESXi

Step 7. Depending on the type of upgrade you want to perform, complete the following fields on the Select Upgrade Type page(see Figure 5-67):

UCS Server Firmware:

• In the UCS Manager FQDN/IP field, enter the Cisco UCS Manager FQDN or IP address.

• In the User Name field, enter the Cisco UCS Manager <admin> username.

• In the Admin Password field, enter the Cisco UCS Manager <admin> password.

• Click the Discover button to view the current UCS firmware package version (which is listed under M5 Current Version).

Data Platform:

• Upload the Cisco HyperFlex Data Platform upgrade bundle for upgrading existing clusters (the one used in Step 3).

• Notice that the current HyperFlex Data Platform version is displayed.

• Notice that the HyperFlex Data Platform version of the uploaded bundle is displayed.

ESXi:

• Upload the latest Cisco HyperFlex custom image offline bundle for upgrading existing ESXi hosts (for example, HX-ESXi-6.7U3-14320388-Cisco-Custom-6.7.3.1-upgrade-bundle.zip).

• Notice that the current ESXi version is displayed.

• Notice that the ESXi version of the uploaded bundle is displayed.

When you’re done with the settings in this screen, click Upgrade to begin the cluster upgrade process. The validation screen on the Progress page shows the progress of the checks performed (see Figure 5-68).

When the validations are complete, the upgrade process starts, and the Progress page displays the upgrade progress for all the nodes (see Figure 5-69).

When the upgrade completes, the Progress page shows that the HX Data Platform upgrade has completed and asks you to refresh the browser session to see the upgrade changes (see Figure 5-70).

Offline Upgrade Process

Before you proceed with either a combined upgrade or a split upgrade, consider the following guidelines:

• The package name must match the file that you uploaded to the controller VM.

• You need to enter passwords when prompted to do so.

• Nodes are upgraded with the new version of the Cisco HX Data Platform software and rebooted one at a time.

• Offline cluster upgrades with nested vCenter are not supported.

The steps for an offline upgrade are as follows:

Step 1. Download Cisco UCS infrastructure A, blade bundle B, and rack bundle C on UCSM.

Step 2. Ensure that the hx-storage-data and vMotionupstream switches are configured for full network failover capability.

Step 3. Upgrade the Cisco UCS infrastructure as required (manually).

Step 4. Perform a graceful shutdown of an HX cluster (SCVMs must remain powered on):

a. SSH to any controller VM in the cluster.

b. Check the cluster health by using the command stcli cluster info | grep health.

c. If the cluster is healthy, shut down the cluster with the command stcli cluster shutdown.

d. Wait a few minutes for the shutdown to complete and the prompt to return.

Step 5. Disable the snapshot schedule by using the command stcli snapshot-schedule --disable.

Step 6. Do a manual bootstrap upgrade by copying storfs-package to /tmp and confirming the plug-in version on vCenter. Run the cluster bootstrap script from the node that has the Cluster Management IP. Bootstrap is a process to update the management packages and plugin on vCenter, as in the example as shown here:

Click here to view code image

# tar –zxvf storfs-packages-4.0.1a-33028.tgz
#./cluster-bootstrap.sh

Step 7. Do an offline upgrade using CLI. The following is an example of a combined upgrade for M5 servers:

Click here to view code image

# stcli cluster upgrade --components ucs-fw, hxdp, hypervisor
--location /tmp/storfs-packages-3.5.1a-19712.tgz --ucsm-host
eng-fi16.eng.storvisor.com --ucsm-user admin --ucs5fw-version
'3.1(2g)'

Step 8. When the upgrade is complete, start the cluster and power on the VMs by following these steps:

a. Log in to any controller VM through SSH by using the command stcli cluster start, as in this example:

Click here to view code image

root@SpringpathControllerZRVF040451# stcli cluster
upgrade-status
Cluster upgrade succeeded

root@SpringpathControllerZRVF040451# stcli cluster start
waiting for Cluster to start on nodes:
This will start the cluster and mount the HX datastores.
Wait for cluster to come online.
Cluster is online

b. Wait for the cluster to become healthy before starting the VMs. Then run the command stcli cluster info| grep health, as in this example:

Click here to view code image

root@SpringpathControllerZRVF040451# stcli cluster info |
grep health
healthState: healthy
state: healthy
 storage cluster is healthy

c. When the cluster is healthy, launch vSphere Web Client or Thick Client, navigate to Hosts and Cluster > Datacenter, right-click Cluster, and select Power > Power On to start the VMs.

Step 9. Enable the snapshot schedule by using the command stcli snapshot-schedule --enable.