You want to know which network storage types ESXi supports and what each type offers.

Review the comparison tables in this recipe.

Table 2-1 lays out the types of storage ESXi supports. Table 2-2 lists the features of different storage types that can be utilized in ESXi. It’s important to understand the technologies and their capabilities and limitations when setting up your ESXi environment.

You will achieve higher success and performance within your environment when using VMware with a SAN, regardless of which type of SAN you use.

Let’s take a look at each technology and the benefits of each.

You need best practice guidelines for placing your virtual machines on specific types of storage.

In this recipe, we have put together some industry standard best practices for placing virtual machines on certain types of storage.

Before we begin, it’s important to mention that a lot of factors are involved when placing your virtual machines on storage. As the industry expands and new technologies come out, there is a standardization on storage tiering, which allows the SAN or storage array to automatically move data between different types of disks. For example, you might have a mix of 15,000 RPM and 7,200 RPM drives in a SAN. The goal of tiering is to place each type of data on a disk with a speed appropriate to the need and use for the data. The software on the SAN will automatically move work loads between the different types of disks, generally on the block level. It’s important to note that each SAN vendor does things slightly differently.

Here we will take a look at the different tiers and what might be placed inside those tiers from a virtual machine perspective.

As you can see, there is no real best answer for virtual machine placement. It’s a best effort based on your knowledge of the workloads in your environment. Because virtualization allows you to easily place virtual machines on different datastores, you can pick and choose until you find a suitable storage location. However, VMware has additional tools like vCenter Operations Manager that can help ensure specific metrics can be met inside your virtualized environment.

You wish to understand how ESXi names its devices when working with storage devices in ESXi.

This recipe breaks down the naming scheme so you can understand how it works.

Figure 2-1 shows a typical list of volumes and their device names. In this figure we’ve listed multiple iSCSI volumes arranged them by the identification fields, followed by their device names, capacity, free space, and the type of filesystem on the volume.

Figure 2-1. Datastore device names

The format of a storage device name in ESXi consists of three or four numbers separated by colons. As an example, in Figure 2-1, the device name for the first volume is vmhba38:C0:T0:L0. The numbers have the following meanings:

HBA:Adapter:Channel:Target:LUN

Our first volume’s datastore HBA has a device ID of 38. The second number, 0, is the storage channel number, the third number indicates that the target of the LUN is 0, and the LUN number is 0.

The third value—the target number—is incremented for each volume added to the HBA. It should be noted that if the same target is shared across multiple ESXI servers, this value can be different.

The first and third numbers may change for the following reasons (if they are changed, they will still reference the same physical device to which they were originally connected):

You want to create a separate iSCSI network to isolate storage traffic for servers when communicating with the storage device.

Using vCenter, create a network and VMkernel port on which the iSCSI device can communicate. Because we are using software iSCSI, we will create one vSwitch assigned to vmnic1.

Before ESXi can communicate with an iSCSI device, a VMkernel network port must be created within the network component of the vCenter server.

The VMkernel port can be configured on an existing network, but we strongly advise you to put your iSCSI traffic on its own network and port group, isolated from all other traffic. This ensures maximum performance for your virtual machines. Follow these steps:

Figure 2-2. Adding an iSCSI VMkernel, selecting vmnic1

You want to use iSCSI connections to store area networks on an ESXi without an iSCSI host bus adapter.

Configure the software iSCSI initiator using vCenter.

Because SCSI is an efficient, low-cost interface and many systems use the popular iSCSI protocol to reach network storage over a TCP/IP network, VMware ESXi allows iSCSI to connect ESXI servers to SANs. It is strongly recommended that you create a dedicated network for this traffic, as described in Recipe 2.3.

ESXi supports two different types of iSCSIs out of the box: hardware iSCSI and software iSCSI. Both are very powerful, but they’re set up differently and require different components to work. Each uses a different kind of software translation, called an initiator, to send traffic from the ESXI server to the network.

Hardware iSCSI uses third-party HBAs to transmit iSCSI traffic over the network. Typically, if you can afford the iSCSI HBA cards, you will benefit from faster data transfers. These cards also offer configuration options for fine tuning and can allow you to boot your ESXI server off the iSCSI SAN. Booting an ESXI server from the iSCSI SAN can be helpful in a situation where you have limited local disk space or are utilizing blade servers.

Software iSCSI uses built-in code in ESXi, specifically the VMkernel, to run the iSCSI protocol over standard Ethernet cards. This eliminates the cost of HBAs, but it puts a significant load on your ESXI server’s physical CPUs, which will affect system performance under high I/O loads. However, a lot of enterprise-grade systems will have TCP/IP offload engine-enabled Ethernet ports that can handle this offload and act like HBAs.

This section explains the basic configuration for a software iSCSI. By default, the iSCSI initiator is disabled, so you must enable it and indicate which SAN volumes you are communicating with:

Figure 2-5. Software iSCSI confirmation

Figure 2-7. Software iSCSI status

Figure 2-8. Adding an iSCSI target

Figure 2-10. Showing the iSCSI target

Recipes 2.5 and 2.13

You want to make iSCSI connections to storage area networks on ESXi with an iSCSI host bus adapter.

Use vCenter to configure the iSCSI HBA cards.

Hardware-based iSCSI HBAs, such as a QLogic HBA, provides a dedicated and specially designed processor to send and receive iSCSI traffic. Before you purchase any iSCSI HBA cards, you should check the VMware HCL. It requires a hardware iSCSI initiator, which you can set up using the instructions in this section:

Figure 2-11. Displaying the QLogic QLE4062c iSCSI HBA

Figure 2-12. Configuring the IP settings for an iSCSI HBA

New LUNs will appear in the Details window of the selected HBA card (Figure 2-13). The server can have a maximum of 64 LUNs, numbered SCSI Target 0, SCSI Target 1, and so on. In our example, the server has four targets, as identified in the Details window under vmhba4.

Figure 2-13. Showing the LUN after rescan

On the SCSI Target 0 LUN (Figure 2-13), notice that the path and the canonical path differ. This is because we are looking at the vmhba4 path view on the second port of the HBA, and the canonical path is set to route all traffic through the first port of the HBA, which is vmhba3.

Recipe 2.4

You want a Windows virtual machine to communicate directly with a SAN over your iSCSI connection.

Using the Microsoft iSCSI Initiator, you can configure your virtual machine to talk to your iSCSI SAN directly.

Using Microsoft’s iSCSI Initiator, you can directly connect a volume that resides on a SAN directly to a virtual machine that is running Windows. This recipe assumes you have set up a separate network for the ESXI server and a virtual machine to use for iSCSI traffic, and that you have assigned a dedicated Ethernet port on the virtual server for ESXi traffic. This section explains how to download and install the initiator.

Allowing your virtual machine to directly connect via iSCSI to your SAN will allow you to use features from the SAN directly, for example, snapshots done on the SAN or other types of backups that require access to a specific LUN. Additionally, some software vendors might require the storage live outside a VMFS volume and this method can provide RDM-like capabilities.

If you are running Windows Vista or Windows 2008, the iSCSI Initiator is already included and no download is necessary. However, if you’re using Windows XP, 2000, or 2003, you’ll need to download the initiator from Microsoft’s website. Microsoft provides both 32-bit and 64-bit versions of the application.

Users who are required to download the application can install it by double-clicking the executable file and following the on-screen instructions. You will be presented with a new window giving you a set of options that include the following user-selectable options. However, we suggest leaving them checked by default.

Continue the installation of the initiator by accepting the license agreement. When the installation has completed, you will have a new icon on your desktop called Microsoft iSCSI Initiator. You’ll use this application to manage your iSCSI connections in Windows.

Figure 2-14. Adding a new target

Figure 2-15. Available iSCSI targets in Windows

Figure 2-16. iSCSI target options

Your firewall is blocking your ESXI server from communicating with storage over its iSCSI connection.

Use vCenter to open the necessary firewall port.

In order for the iSCSI software initiator to communicate with its targets, port 3260 needs to be opened for outbound traffic on the ESXi Server’s firewall. In ESXi 5.0, this port will be opened for you automatically when software iSCSI is enabled. However, for troubleshooting, or if there becomes a need to disable and reenable this service, follow the directions that follow.

Using vCenter, this task is easy. For each ESXI server that is part of your cluster or will be using the iSCSI Software Initiator, follow these steps:

Figure 2-17. Enabling software iSCSI client firewall rules

Recipe 2.4

You want to route iSCSI traffic from an ESXI server over multiple paths for speed or redundancy.

Use vCenter to view and change multipath settings.

One of the nice things about iSCSI is that it is IP based, so it already has built-in support for multipathing using IP routing if you are using dynamic routing protocols on your network. This configuration of iSCSI represents a simpler alternative to Fibre Channel.

The steps for configuring multipathing are as follows:

Figure 2-18. Showing hardware iSCSI paths

Figure 2-19. Managing paths

You want to give your ESXI servers access to additional storage on a Fibre Channel SAN.

Use vCenter to configure the new storage and present the LUN to the ESXI servers.

Before you can configure the Fibre Channel disks on the ESXI server, you must first create the LUN on the disk array (SAN) and set up the specific Fibre Channel zoning permissions. Normally, this is done in a utility that is provided by the SAN manufacturer. Fibre Channel zoning will take place on your Fibre Channel fabric on your switching environment. That is outside the scope of this recipe, so please refer to the documentation from your SAN and Fibre Channel switch manufacturers.

VMware has some best practice guidelines in place when using Fibre Channel storage. We’ll outline the VMware best practices in the following before we look at adding new VMFS datastores via the Fibre Channel fabric.

When the SAN-side configuration is completed, you can use vCenter to add the new disk to your ESXI servers:

You want direct access, without going through the virtual filesystem, from a virtual machine to a disk on your storage network.

Use vCenter to configure raw device mapping (RDM) for the virtual machine.

RDM allows virtual machines to have direct access to a LUN on a physical storage system without the use of a VMFS datastore.

VMware generally suggests that you store your virtual machine files on a VMFS partition. However, certain situations may require the use of RDM, such as MSCS clustering that spans over physical hosts, or the use of SAN technologies inside your virtual machine. RDM is supported only over Fibre Channel and iSCSI at this time.

RDM has two different modes:

To add an RDM disk to a virtual machine that has already been created:

Figure 2-20. Selecting the LUN for the RDM

Once you have made your selection, click Next to continue and click Finish to add the RDM to the virtual machine.

You want to connect an NFS file share to an ESXi Server.

Create a VMkernel port to allow ESXi to communicate with NFS.

Although a lot of larger ESXi environments use Fibre Channel or iSCSI, ESXi also supports the use of NFS. As with software-based iSCSIs, you will need to create a VMkernel on the ESXI server for NFS traffic to pass over. It is recommended that you configure the VMkernel to use a dedicated network, but you can configure it on an existing network if necessary. Here are the steps:

Figure 2-21. Adding the VMkernel port

Before ESXi can communicate with the NFS datastore, you have to configure it to use the storage, as described in the next section.

You wish to add an NFS datastore to your ESXi Server.

Use vCenter to configure the ESXI server so it recognizes the NFS device.

NFS is becoming a popular storage tier because it is cost-effective and provides reliable speed. VMware supports HA, DRS, vMotion, and virtual machine snapshots on NFS datastores.

Before you follow this recipe, set up a VMkernel port to communicate with the NFS datastore, as described in Recipe 2.11. Then configure the ESXI server as follows:

Figure 2-24. Selecting NFS to create a datastore

Figure 2-25. Entering NFS device properties

You want to add a new VMFS volume to an ESXi Server.

Using vCenter, you can easily create and attach new VMFS volumes.

Adding an additional VMFS volume to your ESXI servers is pretty straightforward for all aspects of storage. For example, once configured on the SAN side, Fibre Channel and iSCSI disks will be visible to your ESXI server, and local disks will be detected automatically.

When adding a new disk to your ESXI servers in a clustered environment, where multiple ESXI servers will be accessing the SAN LUN/datastore, you only need to add it to one ESXI server. After the datastore is created, you can rescan or refresh the existing servers and the datastore will appear.

ESXi 5 allows you to have 256 VMFS datastores per host, with a maximum size of 64TB. The minimum size for a VMFS datastore is 1.3GB; however, VMware recommends a minimum size of 2GB.

Follow these steps to create the VMFS volume:

Recipe 2.15

You need to Rescan your storage adapters.

Use the rescan feature in vCenter Server.

There may be times when it is necessary to rescan your ESXi Server’s storage devices. These situations include:

To rescan a server’s storage adapters:

You must create a new VMFS volume but you do not have access to vCenter Server, or you want to create an automation script.

Using the vmkfstools command on your ESXI server allows you to create new volumes.

The vmkfstools command creates a new VMFS volume. It also assigns it a unique UUID, which is a hexadecimal value that incorporates the SCSI ID and label name into the volume’s metadata. Run this command on the ESXi host’s service console:

vmkfstools --createfs vmfs5 --blocksize 1m disk_ID:P

The --createfs option tells the command which type of VMFS volume to create. In our case, we created a VMFS-5 volume. The -S option allows you to specify a label for your volume, which should be a simple name.

Recipe 2.13

You need to find information about a VMFS volume, but this information is contained in files on that volume rather than in the vCenter.

Each virtual machine has a directory with files that define and control the virtual machine. You can view the contents of this directory and read the files themselves, because they’re text based.

Many files make up a virtual machine. Understanding the purpose of each one helps you keep your virtual machine environment running at top performance:

These files are located within the virtual machine’s directory on the storage volume, which is typically /vmfs/volumes/DATASTORE/VIRTUALMACHINE. For example, our server, which is named W2KStandardBase, is on the ELISCSI01 database store, and its path is /vmfs/volumes/ELISCSI01/W2K3StandardBase.

One of the datastores on your ESXI server is running out of space.

Using the vCenter server, you can add an extent to your existing datastore to add more space.

An extent is a physical hard drive partition on a physical storage device, such as a Fibre Channel, iSCSI SAN, or local disk. These partitions can be dynamically added to an existing VMFS-based datastore, allowing you to grow as needed. Datastores can span multiple extents and will be presented to ESXi as a single volume.

Adding new extents can be done while the existing VMFS datastore is online. This makes adding new space really easy. Each datastore can span up to 32 extents.

It should be noted that the first VMFS datastore (disk) in the extent holds the metadata for the entire datastore, including all new extents. If the first extent is corrupted or damaged, you are at risk of losing all the data on the entire extent set.

Add an extent as follows:

Figure 2-31. Showing the storage volume that will get an extent

Once the datastore has been expanded, you can click on the datastore to view the new extent in the lower Details window. You will also need to refresh the datastore on each host so the new capacity will be reported correctly.

Recipes 2.4, 2.5, and 2.15

You wish to view the metadata for a specific VMFS volume.

Use the vmkfstools command.

The metadata in a VMFS volume is made up of six parts:

The vmkfstools command lets you view the metadata in a specific VMFS volume:

vmkfstools -P -h pathname

The -P option allows you to read the metadata, and the -h option tells the vmkfstools command to display amounts in megabytes, kilobytes, or gigabytes, instead of the default bytes. The pathname is the pathname of your VMFS filesystem. For example:

VMFS-5.54 file system spanning 1 partitions.
File system label (if any): esx02_datastore_lun1
Mode: public
Capacity 849.8 GB, 848.8 GB available, file block size 1 MB
UUID: 4f546274-32257e8a-132c-5404a64c91b1
Partitions spanned (on "lvm"):
	t10.FreeBSD_iSCSI_Disk______001d60f814da001_________________:1
Is Native Snapshot Capable: NO

Your ESXI server is missing a diagnostic partition.

Use the vCenter to create a diagnostic partition.

ESXI servers need to have a diagnostic or dump partition in order to run. These partitions store core dumps for debugging and are used by the VMware technical support team. Diagnostic partitions can be created on a local disk, on a shared LUN on a Fibre Channel device, or on a device accessed by a hardware-based iSCSI initiator connection. Diagnostic partitions are not supported on software-based iSCSI initiators.

The diagnostic partition must be at least 100MB in size. If you use a shared storage device, each ESXI server must have its own separate diagnostic partition.

If you choose the Recommended Partitioning scheme when installing your ESXI server, the installer automatically creates the diagnostic partition for you. The following steps create a diagnostic partition if your ESXi installation lacks it:

You wish to remove an old datastore from your ESXi Server.

Use the vCenter to remove a volume from ESXi.

The steps to remove a volume follow:

Using the vCenter, find if a specific VMFS datastore is on a local disk or connected to a SAN or NFS device.

Through vCenter, you can determine the physical location of a VMFS datastore.

To find out where your VMFS datastore is located:

When you try to add new storage in the vCenter, you receive timeout errors.

Adjust the timeout value in the vCenter.

timeouts can occur for various reasons when adding new storage via the vCenter. It may be simply that the timeout values are just too short, but be aware that lengthening these values is not a fix-all solution; there may be a larger underlying problem in networking or I/O that you should investigate.

To lengthen the timeout in the vCenter client, navigate to Edit→Client Settings→Remote Command timeout→Use a custom value. The value is shown in seconds. Adjust it to a higher number (perhaps two times what is already set).

A Windows guest operating system will sometimes timeout when a SAN is rebooted or goes through a failure and recovery.

You can adjust the disk timeout value in the Windows registry.

Default timeouts on Windows servers may be too short for a SAN recovery. During the time your SAN is down and Windows is trying to write data, it is possible for data to be lost or corrupted if your timeout values are not high enough. You can change the timeouts on both Windows Server 2000 and 2003 by editing the system registry:

Once these changes have been made, Windows will wait 60 seconds before generating disk errors. If 60 seconds isn’t long enough, you can adjust the value to suit your specific needs.

You need to rename your datastores.

You can use vCenter to rename your datastores. Follow these simple steps.