Windows Network Direct: Your better bet is with Windows Server 2019

I have been always struggling to get RDMA working inside a Windows virtual machine. I had tried Mellanox ConnectX-3, Mellanox ConnectX-5 and Chelsio T62100-LP-CR network adapters, with Windows Server 2012 or 2016, and even with Direct Device Assignment in Windows Server 2016, I could not get RDMA working flawlessly in any virtual machine.

Recently, I retried RDMA in a Windows VM (2019) on a Windows Server 2019 host, with a Chelsio T62100-LP-CR - and finally have RDMA (iWarp) working correctly (even without a switch - you can connect port 1 to port 2 to form a 100GbE link). It enabled SMB Direct between the VM and the host (and between VMs as well), and performance was acceptable (needs tuning).

If you are after any RDMA application inside a Windows VM, or simple just want to use SMB Direct in a VM, Windows Server 2019 or later is your better bet in this case.

Do note that you need the following:

• SR-IOV support from BIOS - this sometimes means enabling the ASPM option in BIOS.
• A network card that supports RDMA - I like iWarp because it is simpler (virtually no configuration needed). If you like RoCE then you may need DCB configured properly, or even need a 40GbE/100GbE switch.
• Windows Server 2019 or higher - both host and VM. You may use Windows 10 (latest) - I didn't try that out but theoretically it should work.
• Workstation / Server grade hardware - I have seen many times people complaining about not being able to enable SR-IOV due to missing implementation like IOMMU or ACS etc. with consumer grade hardware. Your CPU supports all these features doesn't mean your motherboard / BIOS has support of all features.