Chapter 21. Performing Basic Troubleshooting of ESXi and vCenter Server Operations
This chapter covers the following subjects:
Verifying Network Configuration
Troubleshooting Common Storage Issues
Troubleshooting Common VM Issues
Verifying Correct Network Resources
Troubleshooting Virtual Switch and Port Group Configuration Issues
Troubleshooting Physical Network Adapter Configuration Issues
Identifying and Detecting Common Knowledge Base Article Solutions
This chapter covers basic troubleshooting on your virtual networks and storage. In addition, you learn about troubleshooting common issues with VMs, port groups, and physical adapters. Finally, it covers using the VMware Knowledge base to learn from the experts about how to fix your issues. This understanding will help you troubleshoot your own systems and is essential to successfully navigate the troubleshooting questions on the exam.
“Do I Know This Already?” Quiz
The “Do I Know This Already?” quiz allows you to assess whether you should read this entire chapter or simply jump to the “Exam Preparation Tasks” section for review. If you are in doubt, read the entire chapter. Table 21-1 outlines the major headings in this chapter and the corresponding “Do I Know This Already?” quiz questions. You can find the answers in Appendix A, “Answers to the ‘Do I Know This Already?’ Quizzes and Chapter Review Questions.”
1. You are creating a management connection for your vSphere environment. Which of the following is a valid option?
a. Create a VMkernel port and a separate VM port group that will each be used for management traffic.
b. Create two VMkernel port groups that use the same vmnic.
c. Create two VMkernel port groups that each use a separate vmnic.
d. Create two VM port groups that are both for management and use two separate vmnics.
2. Which of the following are not true? (Choose two.)
a. You can use VMkernel ports to enable vMotion.
b. You can use VM port groups to enable vMotion.
c. You can assign attributes to VM port groups.
d. You cannot assign attributes to individual ports on a vDS.
3. You have clicked on Home/Storage. Which of the following might be visible in your Navigator pane? (Choose two.)
a. Data centers
b. Hosts
c. Host Clusters
d. Datastores
4. You want to view all hosts connected to a specific datastore. Which of the following is a valid option?
a. Home, Hosts and Clusters, click on Host, Manage, Storage Devices
b. Home, Storage, click on Datastore, Related Objects, Datastores
c. Home, Storage, click on Datastore, Related Objects, Hosts
d. Home, Hosts and Clusters, click on Host, Manage, Storage
5. Which of the following might improve the network performance of a VM? (Choose two.)
a. Place VMs that communicate to each other frequently onto the same host on the same switch in the same subnet so they can communicate without using the external network at all.
b. Change the speed and duplex settings on the vnic.
c. Change the speed and duplex settings on the vmnic to which its port group is connected.
d. Restart the guest operating system on the VM.
6. Which of the following are types of datastores? (Choose two.)
a. VMFS
b. RDM
c. NFS
d. iSCSI
7. Which of the following can you configure on a VM port group? (Choose two.)
a. Security
b. Default gateway
c. NIC teaming
d. IP subnet
8. Which of the following are true regarding vSwitch, port group, and port settings? (Choose two.)
a. Settings for a vSS will override settings for port groups on it.
b. Settings for port groups on a vSS will override settings for the vSwitch.
c. Settings for a vDS will override settings for the port groups on it.
d. Settings for a port on a vDS will override settings on the switch.
9. You want to view the point on the host at which the traffic moves out of the vSphere and into the physical network. Which terms might be used for this point of reference? (Choose two.)
a. vmnic
b. vnic
c. Virtual port
d. Physical NIC
10. Which of the following are valid search options on the VMware Knowledge Base? (Choose two.)
a. Article ID
b. Product
c. Date
d. Time
Foundation Topics
Verifying Network Configuration
Your vSphere network should connect your VMs to each other and allow your VMs to connect to physical resources outside your vSphere. In addition, your network should provide a management port (or multiple management ports) that allows you to control your hosts and VMs. Finally, your network might very well be involved with your storage if you are using IP storage options such as Internet Small Computer System Interface (iSCSI), storage-area networking (SAN), or Network File System (NFS) datastores.
Because your vSphere network is such an integral part of your virtual data center, you should understand the network components and their correct configuration so that you can troubleshoot them when necessary. In this section, you learn how to verify and troubleshoot network configurations, including your VMs, port groups, and physical network adapters. In addition, you learn how to identify the root cause of a network issue based on troubleshooting information.
At the very least, your network configuration should include a VMkernel port for management; otherwise, you won’t be able to control the host remotely. In fact, one is provided for you with the default installation of an ESXi host. If you are using vSSs, you will need at least one VMkernel management port on each host. If you are using a vDS, you will need at least one VMkernel management port on the vDS. Of course, it is possible to configure more than one management port, and that is certainly recommended on a vDS. Another option is to configure one VMkernel port but then configure it to use more than one physical NIC (vmnic). In addition, you might have additional VMkernel ports for a myriad of reasons, including an additional heartbeat network for high availability (HA), an additional port for IP storage (iSCSI or NFS), fault tolerance (FT) logging for vSphere fault tolerance, Virtual SAN, and vMotion.
Other than the VMkernel ports, the rest of the ports on a switch will be used for uplinks to the physical world or, for VM port groups, most will likely be used for VM port groups. The correct use of VM port groups enables you to get more options out of a single switch (vSS or vDS) by assigning different attributes to different port groups. As you know, with vDSs, you can even assign different attributes at the individual port level. VM port groups give you options on which to connect a VM.
Verifying your network configuration consists of viewing your network with an understanding of how all of these virtual components are linked together. Only by understanding how it should be connected will you be able to troubleshoot any configuration issue. Figure 21-1 shows one of the views you can use through your vSphere Web Client to manage the networking of your host.
Troubleshooting Common Storage Issues
As you know, it’s possible for a VM to be given visibility to its actual physical storage locations, as with a physical compatibility raw device mapping (RDM). However, it should not be the norm in your virtual data center. In most cases, you will use either a Virtual Machine File System (VMFS) datastore or an NFS datastore, either of which hides the specifics of the actual physical storage from the VM. Also, you may begin to use a Virtual SAN.
Regardless of what type of storage you use, you need to configure it properly to get your desired result. In this section, you learn about verifying storage configuration. Later, the chapter covers troubleshooting many aspects of storage, including storage contention issues, overcommitment issues, and iSCSI software initiator issues. In addition, storage reports and storage maps that you can use for troubleshooting are discussed. Finally, you learn how to identify the root cause of a storage issue based on troubleshooting information.
Verifying Storage Configuration
Your vCenter includes two views that will assist you in verifying your storage configuration: the Manage, Storage link in Hosts and Clusters view and the Storage view. Each of these tools lists information about your storage, and there is some overlap with regard to what these tools list. If you are focusing on what a host can see, you might use the Manage, Storage link, as shown in Figure 21-2.
Click Refresh to make sure that you are seeing the latest information. You can use the Manage, Storage link to quickly identify the storage adapters and storage devices that are accessible to that host. In addition, you can view the status, type, capacity, free space, and so on, for each one. You can even customize what you show by right-clicking at the top of a column and selecting only what you want to see, as shown in Figure 21-3.
The Storage view allows you to see some of the same information as the Manage, Storage link, but also much more detail about datastores. You can determine which hosts are connected to each datastore, but that is not the primary focus. Instead, the primary focus is detailed information about the datastores to which the hosts are connected.
You should click the Refresh link to make sure that you are seeing the latest information. Figure 21-4 shows the Storage view with a datastore selected in the Navigator (left pane) and the Summary tab selected in the details pane. As you can see, you can also show many more tabs.
Troubleshooting Common VM Issues
I’ve seen and written about many models of troubleshooting that look great on paper but might be overkill for the real world. Also, VMware doesn’t subscribe to a certain five-step or seven-step model of troubleshooting with regard to the exam. Therefore, you should be able to “think through” a troubleshooting question based on what you know about virtual networking.
In general, a VM’s network performance is dependent on two things: its application workload and your network configuration. Dropped network packets indicate a bottleneck in the network. Slow network performance could be a sign of load-balancing issues or the lack of load balancing altogether.
You’ll know if you have high latency and slow network performance; there is no hiding that! How will you know if you have dropped packets? You can use the Advanced performance charts to examine dropped transmit (droppedTx) and dropped receive (droppedRx) packets. (These should be zero, or very close to it, if you don’t have a bottleneck on this resource.)
If these utilities indicate that there is an issue, you can verify or adjust each of the following to address the issue:
Verify that each of the VMs has VMware Tools installed.
Verify that vmxnet3 vNIC drivers are being used wherever possible.
If possible, place VMs that communicate to each other frequently onto the same host on the same switch in the same subnet so they can communicate without using the external network at all.
Verify that the speed and duplex settings on your physical NICs are what you expected.
Use separate physical NICs to handle different types of traffic, such as VM, iSCSI, VMotion, and so on.
If you are using 1-Gbps NICs, consider upgrading to 10-Gbps NICs or using Link Aggregation Groups (LAGs).
Use vNIC drivers that are TSO-capable
This is not an exhaustive list, but it’s a good start toward better virtual network performance. You should apply each of these potential solutions one at a time and retest. In this way, you can determine the root cause of your network issue, even as you are fixing it.
Also, as you know, it’s possible for a VM to be given visibility to its actual physical storage locations, as with a physical compatibility raw device mapping (RDM). However, it should not be the norm in your virtual data center. In most cases, you will use either a Virtual Machine File System (VMFS) datastore or an NFS datastore, either of which hides the specifics of the actual physical storage from the VM. Also, you may begin to use a Virtual SAN.
Regardless of what type of storage you use, you need to configure it properly to get your desired result. This includes verifying storage configuration and determining configurations in regard to connectivity and multipathing, iSCSI connections, and so on. Each of these is briefly discussed, along with how you can obtain information about them that is necessary for troubleshooting.
Note
The “Storage Reviews” tab functionality has been removed from vSphere 6.0 releases vs. earlier 5.x release(s).
Verifying Storage Configuration
Your vCenter includes two views that assist you in verifying your storage configuration: the Manage, Storage link in Hosts and Clusters view and the Storage view. Each of these tools lists information about your storage, and there is some overlap with regard to what these tools list. If you are focusing on what a host can see, you might use the Manage, Storage link, as shown in Figure 21-5.
Click Refresh to make sure that you are seeing the latest information. You can use the Manage, Storage link to quickly identify the storage adapters and storage devices that are accessible to that host. In addition, you can view the status, type, capacity, free space, and so on, for each one. You can even customize what you show by right-clicking at the top of a column and selecting only what you want to see, as shown in Figure 21-6.
The Storage view allows you to see some of the same information as the Manage, Storage link, but also much more detail about datastores. You can determine which hosts are connected to each datastore, but that is not the primary focus. Instead, the primary focus is detailed information about the datastores to which the hosts are connected.
You should click the Refresh link to make sure that you are seeing the latest information. Figure 21-7 shows the Storage view with a datastore cluster selected in the Navigator (left pane) and the Monitor tab selected in the details pane. As you can see, you can also show many more tabs. For example, the Related Objects tab in Figure 21-8 shows the hosts that have visibility to this datastore.
Troubleshooting Storage Reports and Storage Maps
As you have already noticed, you can use a great number of reports and tools for troubleshooting vSphere. In most cases, you are going to be better off learning how to use the vSphere Web Client. Many of the latest features are available only through the Web Client, such as Cross-Host vMotion. Also, the Windows-based vSphere Client is on its way out.
However, there are a few exceptions. For example, at press time, you cannot view maps of any kind through the vSphere Web Client. Because of this, I will present this section on the Windows-based vSphere Client.
You can use the Storage Views tab on the vSphere Client in Reports view to gather a tremendous amount of data about your storage. You can get this same data from the vSphere Web Client, but vSphere Client offers just another location to see a lot of data. In addition, on your Windows-based vSphere Client, you can use the Maps view to see a graphical representation of the relationships between the objects in your vSphere. In fact, you can view storage reports and maps for every object in your data centers except for the networking objects, which have their own reports and maps. This section briefly discusses the use of these storage reports and maps.
Storage Reports
Using your Storage Views tab, you can display storage reports to view storage information for any object except networking. For example, you can view datastores and LUNs used by a VM, the adapters that are used to access the LUN, and even the status of the paths to the LUNs. To access storage reports from the Storage Views tab, follow the steps outlined in Activity 21-1.
Activity 21-1 Viewing Storage Reports
1. Log on to your vCenter with your vSphere Client.
2. In the console pane, select the object on which you want to view connected storage (in this case, VM-02), and then open the Storage Views tabs and click the Reports button, as shown in Figure 21-9.
3. Select View and then Filtering to display the Show All [Category of Items] or click the amazingly small drop-down arrow, as shown in Figure 21-10.
4. Move the cursor over the column heading to the description of each attribute, as shown in Figure 21-11.
Storage Maps
As you can see, Storage Reports can give you a lot of information about your data stores, but all the information is in the form of text. The problem is that we (people) don’t think in text; we think in pictures. We can generally understand a situation better if someone will take the time to draw us a picture.
In essence, that’s just what VMware has done with the Maps view of the Storage Views tab. You can use the view to display a graphical representation of every object that relates to your storage. For example, you can tell whether a specific VM has access to a host that has access to a storage processor that has access to a LUN, and whether or not there is a datastore on the LUN. To use your Maps view on your Storage Views tab, follow the steps outlined in Activity 21-2.
Activity 21-2 Viewing Storage Maps
1. Log on to your vCenter with your vSphere Client.
2. In the console pane, select the object on which you want to view connected storage objects (in this case, VM-03), and then open the Storage Views tab and click the Maps button, as shown in Figure 21-12.
3. You can choose the objects that you would like to display on your map.
4. You can also hover your mouse pointer over an object for a few seconds to see the callout that gives a detailed description of that object.
Verifying Correct Network Resources
As I mentioned earlier, port groups give you options on which to connect a VM. In my opinion, you can see this more clearly from the VM’s standpoint. In Figure 21-13, I right-clicked a VM and then selected Edit Settings. As you can see, I have a list of port groups from which to choose for the virtual network interface card (vNIC) on this VM called Network adapter 1. These port groups are all VM port groups on this switch or on the vDS to which this host is connected. Also, note the Device Status check boxes at the top right of the screen. These should be selected on an active connection. When the VM is connected to the appropriate port group, it can be configured with the correct network resources. If it is not on the correct port group, many issues could result, including having the wrong security, traffic shaping, NIC teaming options, or even having a total lack of connectivity.
Troubleshooting Virtual Switch and Port Group Configuration Issues
Just connecting the VM to a port group does not guarantee that you get the desired configuration. What if the port group itself is not configured properly? You should understand that any configuration options on a vSS will be overridden by conflicting options on a port group of the same switch. In addition, any options on a port group of a vDS will be overridden by conflicting options on a specific port. These options were covered earlier in Chapter 6, “Configuring vSS and vDS Features,” so this section does not go into great detail about security, traffic shaping, NIC teaming, and so on, but Figure 21-14 shows the general area in which you can find them on a vDS. The main point here is to verify that you have set the properties appropriately for the VMs that are connected to the port group.
Troubleshooting Physical Network Adapter Configuration Issues
It can’t all be virtual! At some point, you have to connect your vSphere to the physical world. The point at which the data moves out of the host and into the physical world can be referred to as a physical network adapter, a vmnic, or an uplink. Because the configuration of this point of reference is for a piece of physical equipment, the available settings are what you might expect for any other physical adapter, namely speed, duplex, wake on LAN, and so on, as shown in Figure 21-15.
Identifying and Detecting Common Knowledge Base Article Solutions
Everyone makes mistakes and has problems from time to time. The difference between the wise people and those who are not so wise is that the wise ones learn from the mistakes and problems of others, not just their own. This is the purpose of the VMware Knowledge Base. It is a quick resource that you can use to learn more about virtualization issues that you face on a daily basis. It works on the principle that whatever you are experiencing has most likely been experienced by someone before you and will be experienced again by someone after you.
You can access the VMware Knowledge Base at kb.vmware.com, as shown in Figure 21-16. There you can read the most popular documents, most recent documents, alerts, or type in the information for your own customized search. You can narrow it down by product or search by an Article ID to which you have been referred. There is even a Knowledge Base article (Article ID 878) about how to use the Knowledge Base! Using the Knowledge Base is free, so frequent use will help you stay up to date with the latest information from VMware in the most economical way possible.
Summary
This chapter covers the following main topics:
Basic troubleshooting techniques for your network, storage, and VMs
How you can verify that you have the correct network resources on the VMs so that you get the most performance from them
Troubleshooting virtual connections as well as the physical connections of your hosts
How you can learn from others by consulting the VMware Knowledge Base
Exam Preparation Tasks
Review All the Key Topics
Review the most important topics from the chapter, noted with the Key Topic icon in the outer margin of the page. Table 21-2 lists these key topics and the page numbers where each is found. Know how to perform basic troubleshooting on ESXi hosts, vSphere networks, vSphere storage, and HA/DRS clusters.
Review Questions
The answers to these review questions are in Appendix A.
1. You are creating a management port for your vSphere environment. Which of the following is a valid option?
a. Create a VMkernel port and a separate VM port group that will each be used for management traffic.
b. Create two VMkernel port groups that use the same vmnic.
c. Create one VMkernel port group that uses two different vmnics.
d. Create two VM port groups that are both for management and use two separate vmnics.
2. Which of the following are true? (Choose two.)
a. You can use VMkernel ports to enable vMotion.
b. You can use VM port groups to enable vMotion.
c. You can assign attributes to VM port groups.
d. You cannot assign attributes to individual ports on a vDS.
3. You want to quickly view all the storage options that are available to all the hosts in your vSphere with least administrative effort. Which of the following should you do?
a. Click Home, Storage
b. Click Home, Host and Clusters, Manage, Storage
c. Click Search, Storage Options
d. Click Storage Options, Hosts
4. You want to view all the datastores to which a specific host is connected. Which of the following is a valid option?
a. Home, Hosts and Clusters, click on Host, Manage, Storage Devices
b. Home, Hosts and Clusters, click on Host, Related Objects, Datastores
c. Home, Storage, click on Datastore, Related Objects, Hosts
d. Home, Hosts and Clusters, click on Host, Manage, Storage
5. Which of the following might improve the network performance of a VM? (Choose two.)
a. Install VMware Tools.
b. Use vmxnet3 vNIC drivers.
c. Increase the throughput of the vNIC on the Network tab.
d. Restart the guest operating system on the VM.
6. Which of the following are true regarding RDMs?
a. You should use RDMs whenever possible, instead of datastores.
b. A physical compatibility RDM gives the VM visibility to the actual storage location.
c. VMFS and NFS datastores are generally preferable to RDMs.
d. A physical compatibility RDM is just a pointer to a datastore.
7. Which of the following can you configure on a VM port group? (Choose two.)
a. Security
b. Traffic shaping
c. Storage location
d. IP subnet
8. Which of the following are true regarding vSwitch, port group, and port settings? (Choose two.)
a. Settings for a vSS will override settings for port groups on it.
b. Settings for port groups on a vSS will override settings for the switch.
c. Settings for a vDS will override settings for the port groups on it.
d. Settings for a port on a vDS will override settings on the switch.
9. You want to view the point on the host at which the traffic moves out of the vSphere and into the physical network. Which terms might be used for this point of reference? (Choose two.)
a. vmnic
b. vnic
c. Virtual port
d. Uplink
10. Which of the following are on the Home page of the VMware Knowledge Base? (Choose two.)
a. Most Popular Documents
b. Most Recent Documents
c. Most Debated Documents
d. Most Interesting Documents