[German]A German blog reader has pointed out recently a problem with Windows 10 and Windows Server 2019 regarding poor network performance (low throughput) in connection with Hyper-V guests (but also on bare metal Windows 10 clients without virtualization). The reason is a change by Microsoft in the TCP stack, which causes problems. I'm preparing this in an article, because there is a simple solution in form of PowerShell scripts.
Advertising
Blog reader Flying Sorcerer wrote me the days by email to address the issue (thanks for that).
Since I haven't found it anywhere else, here's a hint.
A colleague asked me at the beginning of the week about problems with server 2019 and hyper guests in the network area.After a somewhat longer search I came across two threads.
Microsoft has made some crap in Server 2019 and Windows 10 since 1709…
The blog reader then sent me various links with places to be found on the web, which deal with the low network throughput of Hyper-V guests on the mentioned operating system platforms.
This problem exists with Hyper-V guests
If a guest operating system under Microsoft's Hyper-V virtualization solution is set up on a host running Windows 10 (version 1709 or higher) or Windows Server 2019, the network throughput for Hyper-V guests is extremely modest. On German site administrator.de someone has already taken up this issue in spring 2019 within this post. Here is an excerpt translated in English:
I have a problem here with extremely slow network connections from multiple VMs.
Hardware are two almost identical new machines (MB Supermicro X11SPM-F with 2x 1 GB Intel X722 onBoard NIC and sep. IPMI, Intel Xeon Silver 4110, 96 GB RAM, local SSD storage, as far as nice) on each of which Windows Server 2019 hrs with activated Hyper V role as host is installed.
Drivers are all original from the manufacturer what Supermicro provides (Intel MB chipset, Intel Pro 23.2 and others). The machines have a sep. IPMI NIC, the first X722 NIC from the server I use for the Hyper V management interface, the second as HV switch for the external connection of all VM. .
- When I install Windows Server 2019 hrs. in a VM everything is fine, everything runs as it should.
- If I install Windows Server 2016 hrs. or Server 2008R2 hrs. in a VM on the host, these VMs have a very slow connection to other physical hosts.
Between the VMs the transfer rate is as desired (iperf. almost 10 GBit/s), from the VM to the host as well (since this is done externally via a switch only about 1 GBit/s […].
To the physical computers on the outside – in the real network – I only get 5-10 Mbps on one machine and 20-120 Mbps on the other, with a strongly fluctuating tendency – depending on the machine.
The affected person has tested some things in the laboratory, and his conclusion is: As soon as a Server 2019 runs as Hyper V on the host, and in a VM the guests are not Windows Server 2019 (he tested it with Windows Server 2016 and Windows Server 2008 R2), the network connection of this VM via the onBoard NIC Intel X722 to the outside is extremely slow. But the CPU performance does not increase either, there are no error packets at the hardware switch, but there is no network throughput.
Temporary solution: separate PCIe NIC
The affected person then solved the problem temporarily by installing another separate physical PCIe NIC (a simple Intel i210) on the host. He tied this NIC to a Hyper-V switch and allowed all VMs with their guests to run over this switch network-wise. In this scenario the network throughput is as desired.
A second case
The blog reader pointed out, that in mid-January 2020 another affected person documented this problem on Spiceworks. There was a lot of testing and documentation published. The following figure shows a protocol of the network throughput.
Advertising
(Source: Spicework)
Then the person concerned came up with the glorious idea of disabling Virtual Receive Side Scaling (vRSS) on the server and on the client. Then he deactivated Receive Segment Coalescing (RSC) on the Windows Server 202012. And suddenly he got the expected values for network throughput between Hyper-V guests and the host. .
In further steps he then came up with how to optimize network throughput in Windows Server 2019 (and Windows 10) as a Hyper-V host. The problem is due to a significant change in the TCP stack by Microsoft (keywords: CongestionProvider, CUBIC, see also this Spiceworks post with PowerShell output).
Solution via PowerShell Script
After many tests the affected person wrote a powershell script called "THE HOLY NGIS VM-WS2019 NETWORK BACK OPTIMIZATION SCRIPT V1.0" and published it on Spiceworks (a slimmed down version can be found at German site administrator.de).
The script has to be executed on Windows Server 2019 acting as host via Run as Administrator. It resets various network settings so that Microsoft's mechanisms no longer work. The network throughput then reaches the desired speed.
The creator notes that this should not be used in running productive environments. Because the PowerShell script restarts the NIC. Therefore a restart of the server is necessary to make all changes effective. However, this is something you would rather not do in a productive Windows Server environment. Also note this post from a Microsoft representative with additional information.
It also affects Windows 10 machines without Hyper-V
In an addendum, the blog reader notes that this network throughput problem also affects simple Windows 10 clients from version 1709 onwards due to the 'optimized TCP stack. He writes about this:
We had here the case that we copied from client to client with 80 KB (!!!) data, after execution of the PowerShell script from Administrator.de (the Windows 10 version) with 80 MB…
The script in the english forum is for Server 2019 (Server, Hyper-V and guests), the one in the german forum is for Windows 10.
A PowerShell script for Windows 10 hosts which solves the problem was published by user MysticFoxDE at administrator.de. My thanks to the blog reader for the hint. Maybe he helps the one or other server administrator or annoyed Windows 10 power users.
Similar articles
Patchday: Windows Server 2008 R2 boot in Recovery mode
Windows Server 2012 goes into update loop
Pending Update Issue with Windows Server 2012
Windows Server 2016: Security-Bug in Profile Security Settings
Advertising
