Before we define or discuss what SIOC control is, there are a few concepts that form the basics and also explain the need for a mechanism like SIOC.

ESXi runs a local host scheduler to balance the I/O between the virtual machines. This means that if there are virtual machines churning a considerable amount of I/O (more than normal), then it is important to make sure that the other virtual machines residing on the same datastore remain unaffected, in a manner that they should be allowed to issue I/O to the device. This is achieved by controlling the volume of I/O each of the participating virtual machines can issue, with the help of per-vmdk shares. This works pretty much like the CPU or Memory shares. The default virtual disk share value is 1000, high being 2000 and low being - 500. The disk with a relatively higher share value will get to issue a larger volume of I/O operations to the device.

Now, all of this will work just fine as long as the datastore is seen by a single ESXi host. Unfortunately, that is not a common case. datastores are often shared among multiple ESXi hosts. When datastores are shared, you bring in more than one local host scheduler into the process of balancing the I/O among the virtual machines. Moreover, these lost host schedulers cannot talk to each other. Their visibility is limited to the ESXi hosts they are running on. This easily contributes to a serious problem called the noisy neighbor situation.

In the following example, since VM-C is the only virtual machine on ESX-02, it gets to consume the entire queue depth, which could starve virtual machines on the other two hosts. If VM-C indeed does a lot of IO consuming of the LUN's queue depth, then it will be referred to as a noisy neighbor:

The job of SIOC is to enable some form of communication between these local host schedulers so that I/O can be balanced between virtual machines running on separate hosts. It does so by maintaining a shared file in the datastore that all hosts can read/write/update. When SIOC is enabled on a datastore, it starts monitoring the device latency on the LUN backing the datastore. If the latency crosses the threshold, it throttles the LUN's queue depth on each of the ESXi hosts in an attempt to distribute a fair share of access to the LUN for all the Virtual Machines issuing the I/O.

Let's consider the previous scenario, wherein there are six virtual machines running on three different ESXi hosts, accessing a single-shared LUN. Among the six VMs, four of them have a normal share value of 1000 and the remaining two have high (2000) disk share value sets on them. These virtual machines have only a single VMDK attached to them. VM-C on host ESX-02 is issuing a large number of I/O operations. Since that is the only VM accessing the shared LUN from that host, it gets the entire queue's bandwidth. This can induce latency on the I/O operations performed by the other VMs: ESX-01 and ESX-03. If the SIOC detects the latency value to be greater than the dynamic threshold, then it will start throttling the queue depth.

The following table will help you understand how fair share percentage is calculated by SIOC:

The throttled DQLEN for a VM is calculated as follows:

DQLEN for the VM = (VM's Percent of Shares) of (Queue Depth)

Example: 12.5 % of 64 → (12.5 * 64)/100 = 8

The throttled DQLEN per host is calculated as follows:

DQLEN of the Host = Sum of the DQLEN of the VMs on it

Example: VM-A (8) + VM-B(16) = 24

The following diagram shows the effect of SIOC throttling the queue depth:

The local scheduler on each of the ESXi hosts uses the iostats file to keep its companion hosts aware of the device I/O statistics observed on the LUN. The file is placed in a directory (naa.xxxxxxxxx) on the same datastore: