The following screenshot shows the NUMA configuration options available for the processor configuration of a VM. These options are hidden away for good reason. In nearly all scenarios, you should not mess with these settings. Hyper-V will do the best job of setting the right NUMA topology based on the physical Hyper-V host. However, there are a few scenarios where you may need to change these values.

These settings are related to the number of processors (cores), memory, and the number of nodes on a single CPU socket:

Let's consider the following two scenarios.

Scenario 1: You have two hosts with different hardware NUMA topologies and where VMs may be live-migrated between servers. In this case, the NUMA configuration should be changed to match the smallest NUMA topology among all the hosts in your environment to which the VM may be migrated. Additionally, in most clusters, the hosts all have the same NUMA topology, this also means that keeping your cluster nodes specifications the same helps a lot here.

Let's suppose we have two Hyper-V hosts with the following specs:

If a VM was created on Host 1, that is the NUMA topology that would be configured for that VM. If the same VM was then moved to Host 2, the VM would have incorrect NUMA configuration and will not have optimal resource assignments, because what it believes is a single NUMA node would actually span multiple NUMA boundaries. It would therefore make sense to manually set the NUMA topology of the VM to match that of Host 2.

In the preceding screenshot, note that there is the Use Hardware Topology button. If you change the settings and realize you don't know what the values were originally, you can click this button and the values will be reset back to the Hyper-V recommended values for that particular host. For most scenarios, this is best done using the Use Hardware Topology button. Microsoft made it easy for you to revert to the host's physical NUMA topology. Also, on Hyper-V clusters with lots of live migration, you need to make sure to keep the NUMA layout on the nodes the same, for consistent performance.

Scenario 2: Let's suppose you have an application that requires a minimum of 2 CPUs in order to run, which is a very rare case, that application does a check for CPU count, which actually only counts as "sockets"; you have been asked to move that application to a VM.

If this is the case and the app requires more than one socket, then it's possible to expose two or more CPUs (vNUMA) to a VM if the following conditions are true:

In this scenario, the VM gets configured with two virtual NUMA nodes. It also gets two sockets, as shown in the following screenshot:

How does that compute? Well, it's the virtualization of the underlying physical CPUs. The vCPUs are just scheduled time slices of compute time on the physical cores. Please remember that vCPU and vNUMA are logical concepts, with time slices of interrupts on a processor, and the Hypervisor always does its best to use the best possible NUMA layout. In nearly all scenarios, you do not need to have two sockets in the guest, because software should not worry about the number of physical CPUs. That's about logical cores, and one socket with more logical cores can do parallelism (multithreading) perfectly.

In general, virtual NUMA works well and delivers the best possible results for all use cases, provided you choose the best option for your needs.

As discussed earlier in this section, the best performance for a multi-NUMA nodes system comes from processes running on processor cores using local memory within the NUMA node rather than having to span NUMA nodes, which means the memory required is connected to another processor; this is known as remote memory, and has a higher latency than local memory.

There are two types of NUMA spanning configurations: configuration at a host level and configuration at a VM level.

Host Settings: By default, starting with Windows Server 2012 onwards, Hyper-V enables NUMA spanning, which enables a VM to be allocated memory across the boundary of the NUMA node of the processor cores being used. This not only allows more VMs to run, it also might lead to an overall performance hit, because using memory outside of the same NUMA node of the processor increases latency.

The Hyper-V-Worker event log reflects this and you'll find a warning event ID 3054:

If you disable NUMA spanning under Hyper-V Settings, NUMA Spanning, deselect Allow virtual machines to span physical NUMA nodes, as shown in the following screenshot. Then, a VM can't be allocated memory from remote NUMA nodes and would be limited to using memory on its local NUMA node only:

Disabling or enabling NUMA spanning requires the Hyper-V Virtual Machine Management service (vmms.exe) to be restarted:

NUMA spanning allows for a less optimized layout, if resources are not available within the NUMA nodes, at the cost of performance, but it enhances density and flexibility. When you disable NUMA spanning, you demand consistent best performance at the cost of resources, as it will not span NUMA nodes.

Mr. Ben Armstrong (Principal Program Manager lead on the Hyper-V team), provides us with a more detailed and accurate statement about NUMA spanning:

By disabling NUMA spanning at the host level, you disable it for all VMs on the host and you ensure that the VMs' virtual NUMA nodes are served from one NUMA node, giving the best performance.

If a VM is configured with Dynamic Memory, there is a potential that this VM may not be able to start if the required amount of memory for the VM is not available on a single NUMA node. It also means you may not be able to live-migrate VMs to other nodes if the target node cannot satisfy the NUMA requirements.

The following screenshot reflects this, thus, the VM will not start:

However, if a VM is configured with static memory, the VM will fail to start if any single virtual NUMA node cannot be placed entirely within any single physical NUMA node.

The Get-VMHostNumaNode PowerShell cmdlet gets the NUMA topology of a Hyper-V host and returns an object for each of the host's NUMA nodes. If the results return more than one NUMA node, as shown in the following screenshot, your host is NUMA-based. However, if the results return one node, your host is not NUMA-based:

As shown in the following screenshot, the virtual NUMA topology can be configured for each individual VM as well. The maximum amount of memory and the maximum number of virtual processors in each virtual NUMA node can be configured, in addition to the maximum number of virtual NUMA nodes presented in each virtualized socket. By default, these values are set to align with the host's physical NUMA topology, and it's strongly recommended not to change these settings:

As described in Scenario 1 above, if a VM will be imported or migrated between multiple hosts with different physical NUMA topologies, and NUMA spanning has been disabled, it's required to configure the virtual NUMA setting for each VM to the lowest default value that is available across the potential hosts. This ensures the virtual NUMA topology for the VM will fit within the physical NUMA topology on each host that it could be migrated to.

However, if NUMA spanning has been enabled, it's advisable to set the virtual NUMA setting for each VM as well. This ensures if the VM is moved to a host with different physical NUMA topologies, the VM resource assignments will not be impacted.

When NUMA spanning is disabled on the host, you can use (Get-VMHost).NumaStatus or Get-VMHostNumaNodeStatus cmdlets; those cmdlets will show you the memory distribution for all VMs across all NUMA nodes.

The following screenshot reflects this. The VM named WS-DC has obtained 50% (100 GB) of its memory from NUMA NodeId 1 and 50% (100 GB) from NUMA NodeId 0:

If performance is needed at all times, meaning consistent best possible performance, then you can disable NUMA Spanning, but you must make sure that all VMs with their vCPU/vNUMA topology can get all the memory within a single NUMA node (per vNUMA node).

Remember that if a VM uses Dynamic Memory, the Virtual NUMA is disabled for that VM.

I encourage you to read more on Virtual NUMA and NUMA Spanning on TechNet at the following link: Hyper-V Virtual NUMA Overview (https://technet.microsoft.com/en-us/library/dn282282.aspx).

For bulk configuration and automation, you can also use a cmdlet—Set-VMMemory—that is designed only to manage the Dynamic Memory settings. All the configurations, such as maximum bytes, minimum bytes, and startup bytes, can be specified using this command line. In the following example, all the VMs starting with SP had the Dynamic Memory option enabled, with the other values configured as well.

A new, helpful PowerShell feature is that you can use abbreviations such as MB for megabytes, GB for gigabytes, and TB for terabytes:

Set-VMMemory -VMName SP* -DynamicMemoryEnabled $true -MaximumBytes 6GB -MinimumBytes 4GB -StartupBytes 5GB

Also, you can use the Set-VM cmdlet to change the same settings, as shown in the following example:

Set-VM -Name SP* -DynamicMemory -MemoryMinimumBytes 4GB -MemoryMaximumBytes 6GB -MemoryStartupBytes 5GB

Runtime Memory Resize is configurable via PowerShell as well: you can use the Set-VM cmdlet to change the static memory settings for a running VM, as shown in the following example:

Set-VM -Name SP* -MemoryStartupBytes 4GB 

You can also start creating scripts to query memory assignment and memory demand, and use that output to redistribute memory available on the host among the VMs.

Here is an example of this scenario: the following screenshot shows we have a VM that has a memory demand of 1,423 MB, which is higher than the assigned memory of 1,024 MB. If this is case, we will add the memory demand on top of the assigned memory and increase the static memory on the fly:

You can use the following script to change the static memory settings for a running VM when the workload arises:

$VMs = Get-VM * | Where-Object {$_.DynamicMemoryEnabled -eq $false}

Foreach ($VM in $VMs) {
# Memory Demand Before
$VMMemory = Get-VM -Name $VM.Name | `
        Select Name, State,@{Label="CPU Usage %";Expression={$_.CPUUsage}}, `
        @{Label="Assigned Memory MB";Expression={$_.MemoryAssigned/1048576}}, `
        @{Label="Memory Demand MB";Expression={$_.MemoryDemand/1048576}}, MemoryStatus

Write-Output "Current Memory Demand" $VMMemory

If ($VMMemory.'Memory Demand MB' -gt $VMMemory.'Assigned Memory MB') {
   [int64]$RAM = 1MB*($VMMemory.'Assigned Memory MB'+$VMMemory.'Memory Demand MB'+1)
   Set-VM -Name $VMName -MemoryStartupBytes $RAM

   # Memory Demand After
    $VMMemory = Get-VM -Name $VM.Name | `
        Select Name, State,@{Label="CPU Usage %";Expression={$_.CPUUsage}}, `
        @{Label="Assigned Memory MB";Expression={$_.MemoryAssigned/1048576}}, `
        @{Label="Memory Demand MB";Expression={$_.MemoryDemand/1048576}}, MemoryStatus

    Write-Output "Updated Memory Demand" $VMMemory
    }

} 

As you can see in the following screenshot, the static memory has been increased according to the memory demand: