Hypervisors are basically the backbone of network virtualization. Virtual machines are able to leverage NSX features primarily because the ESXi host is a network-virtualized host. One of the most critical pillars of an NSX installation is ESXi host preparation. If we don't have the right modules running in the ESXi host, the whole purpose of leveraging NSX features will be defeated. Symptoms would be that we might not be able to install feature X, or we can configure feature X, but the functionality is impacted. Watch out for the following VIBs in the ESXi host:
- esx-vxlan
- esx-vsip
- esx-dvfilter-switch-security (starting from NSX 6.2.0, esx-dvfilter-switch-security is part of esx-vxlan vibs)
This is the command to check if VIB is installed in ESXi host:
esxcli software vib list | grep vibname
Since these are VIBs, we can manually uninstall and install the same during break fix scenarios. But the real question is, who is pushing these VIBs? That's where I have seen the majority of issues. Behind the scenes, vCenter Server ESX Agent Managers (EAM) are responsible for installing these VIBs. So, first and foremost, the EAM service should be up and running. The following steps are useful for collecting EAM based upon the operating system and vCenter Server flavor.
Following are the EAM log locations for respective vCenter Server and operating system versions:
- VMware vSphere 5.1.x/5.5.x (EAM is a part of the common Tomcat server):
- Windows 2003:
C:Documents and SettingsAll UsersApplication DataVMwareVMware VirtualCenterLogseam.log - Windows 2008: Same as Windows 2003, the VC log directory is located at
C:ProgramDataVMwareVMware VirtualCenterLogs - vCenter Server Virtual Appliance (VCVA):
/storage/log/vmware/vpx/eam.log
- Windows 2003:
- VMware vSphere 6.x (EAM is a standalone service and has embedded tcserver):
- Windows 2003:
C:Documents and SettingsAll UsersApplication DataVMwareCISlogseameam.log - Windows 2008: Same as Windows 2003, the VC log directory is located at
C:ProgramDataVMwareVMware VirtualCenterLogs - CloudVM:
/storage/log/vmware/eam/eam.log
- Windows 2003:
I have seen a lot of issues, especially when a vCenter Server installation is a Windows-based installation, with EAM trying to use port 80 for downloading VIBs. At times, we might have other applications or services running in VC, which might be leveraging port 80 and will cause VIB download failures, so we have to change the default EAM ports. However, starting from VMware vSphere 6.0, VIB downloads over port 443 (instead of port 80) are supported. This port is opened and closed dynamically. The intermediate devices (firewalls) between the ESXi hosts and vCenter Server must allow traffic using this port. With that, we will move on to our next topic: control plane and data plane log collection.