Of course, if your VM has fewer resources that it needs, it might affect the performance, but there is a problem with oversized VMs as well.

The VMware ESXi CPU scheduler allocates physical CPU time slots to vCPUs in VMs. If your VMs are configured with multiple vCPUs, the CPU scheduler must wait for physical CPUs to become available. Unused vCPUs will continue to consume system resources even when the system isn't using them. If other single-vCPU VMs use the system, your multi-vCPU VM will have to wait for CPU time.