Moving VMs Quickly

You now have a VM that is cleanly sized and NUMA-aligned on a host that is perfectly configured on the UEFI side. Oracle or HANA are installed and running without any problems. Sooner or later, though, the server firmware will certainly need to be updated or the ESXi server upgraded to a new version or patch level, and that triggers a restart. In this case, one of the best and most popular features of VMware virtualization comes into play: vMotion (i.e., live migration from one ESXi host to another).

The good news is that vMotion has repeatedly been optimized over the various vSphere versions, with the last major improvements being introduced in vSphere 7.0U2. vMotion can move most VMs without restriction, regardless of size.

If you want to migrate a VM, you will usually also want it to happen as quickly as possible, especially if the ESXi host has state-of-the-art 25, 40, or even 100 gigabit (Gb) network adapters, the VM is extremely large, and, say, 5.4TB of RAM needs to change locations. Now for the bad news: Even if the server does have a 100Gb adapter, you will still need to do a bit of vMotion tuning to be able to use the bandwidth. I'll skip the basics of how vMotion works here and get right to the tuning side. For the sake of simplicity, assume you have a 100Gb adapter and a single vMotion VMkernel port, which is the reality – or close to it – in many environments.

Setting out to move a VM triggers a vMotion process and various helper processes. The latter includes the Completion helper, the Crypto helper, and the Stream helper. However, focus on the Stream helper here, because this process is the one responsible for transferring the RAM content from the source ESXi server to the target; these processes also exist on the target server.

Now for the actual problem: A single vMotion stream (and therefore Stream helper) is limited to an average bandwidth of 15Gb, which means that despite the 100Gb network interface controller (NIC), you would only achieve a maximum of 15Gb vMotion bandwidth, and the vMotion process would be relatively slow. However, if you create multiple streams, the bandwidth increases and vMotion is accelerated:

• one stream ~ 15Gb
• two streams ~ 30Gb
• three streams ~ 45Gb
• six streams ~ 90Gb
• seven streams ~ 105Gb

In other words, if you want to make full use of your 100Gb interface, you have to configure seven streams, which is not usually automatic, and you will want to be on the safe side with your monster VM. To do this, you need to set the Migrate.VMotionStreamHelpers parameters in the Advanced Systems Settings on all your ESXi servers to the appropriate value: 7 in this case. The default value is 0 , which means that streams are created automatically, and you will not usually see more than four Stream helpers launched. Therefore, you set the values manually.

However, you also have a hardware queue, or multiple hardware queues, between the VMkernel port and the physical NIC. To achieve maximum vMotion performance, you also need to adjust the number of hardware queues to the number of software queues (the same as the number of Stream helpers). Again this is 7 in this example. You can do this by editing the Net.TcpipRXDispatchQueue parameter in the ESXi host's advanced system settings. The default is 2 . Because this change is hardware related, the server then needs to reboot.

Conclusions

You have now completed the most important steps to size your monster VMs properly, distribute them to NUMA nodes, and create the conditions for worry-free vMotion. If you follow these rules, 85 to 90 percent of all monster VMs, whether HANA, Oracle, or Microsoft SQL, should be available, offer good performance, and support moving. The remaining 10 to 15 percent will probably present pitfalls, such as high-latency sensitivity in the memory area and requirements for the lowest possible network latency because of system replication or real-time applications.