Windows Server 2019/2022: Updates KB5031361/KB5031364 cause Hyper-V issues

[German]I got reports that update KB5031361 for Windows Server 2019 (and Windows 10 Enterprise 2019 LTSC) is causing issues with VMs running on Hyper-V. The same is reported for update KB5031364 for Windows Server 2022. The virtual machines cannot be started or are not found after installing the update. Uninstalling the affected updates fixes these problems again.

Windows 10 Update KB5031361

Cumulative Update KB5031361 has been available since October 10, 2023 for Windows Server 2019 as well as Windows 10 2019 Enterprise LTSC and Windows 10 IoT Enterprise LTSC (see Patchday: Windows 10 Updates (Oktober 10, 2023)). This updates fixes security issues in the operating system, but also introduces some new features such as the optional installation of azure Arc. Microsoft notes that the update should be installed to protect against CVE-2023-44487. Details can be found in the support article for the update.

Issues with Hyper-V VMs

On October 11, 2023, various German blog readers reported serious problems with Hyper-V systems. In this German comment, Sascha A reports that the update KB5031361 led to a failure of the Hyper-V cluster (2x Server 2019 Datacenter) at a customer. A special scenario is used there.

• The cluster nodes are running "Cluster-enabled updating", which installs the Windows updates on both Hyper-V nodes one after the other according to schedule.
• Once one Hyper-V node has finished updating, the VMs are moved to the second Hyper-V node via live migration and then a reboot is performed.
• After that, the second node starts using the same procedure.

For the October 2023 patchday, there were problems with the systems because after the first Hyper-V node installed the update and restarted, it could not start the DC VM for the cluster network. There was the error message: "Invalid parameter". After Sascha A. uninstalled the update KB5031361 on the first Hyper-V node and restarted the server, the VM could be started again immediately.

Other German users like Mark confirmed that the VMs could not be started under Hyper-V after installing the update KB5031361 . After installing the update and then restarting Hyper-V, the following error message appears:

Fehler beim Starten von "dc01" (ID des virtuellen Computers xxx).
"dc01" Synthetic SCSI Controller (Instanz-ID yyy): Fehler "Unzulässige Funktion." (0x80070001) beim Einschalten (ID des virtuellen Computers: xxx).
"dc01": Die Anlage "E:\HyperV\dc01\Virtual Hard Disks\dc01.vhdx" konnte aufgrund des folgenden Fehlers nicht geöffnet werden: "Unzulässige Funktion." (0x80070001) (ID des virtuellen Computers: xxx).

or in English:

Error starting "dc01" (virtual computer ID xxx).
"dc01" Synthetic SCSI controller (instance ID yyy): Error "Illegal function." (0x80070001) on power up (virtual computer ID: xxx).
"dc01": Failed to open attachment "E:\HyperV\dc01\Virtual Hard Disks\dc01.vhdx" due to the following error: "Illegal function." (0x80070001) (Virtual computer ID: xxx).

After uninstalling the update, the problem was fixed. In the comments to the German blog post Patchday: Windows 10-Updates (10. Oktober 2023), more affected German administrators have reported similar issues.

In the Microsoft Answers forum, there is this post on October 15, 2023, which also confirms the problems. In the Spiceworks Community, there is an entry Hyper-v vm won't start – VHDX incorrect function that describes the same problem.

Windows Server 2022 with Update KB5031364 also affected

In the Microsoft Q&A section there is the post Virtual machines failed to start after installing Oct 2023 Update (KB5031364), which also addresses the same Hyper-V problem with guest operating systems not starting in Windows Server 2022 after installing update KB5031364. There are other users there who confirm the problem. A support ticket has been opened by an administrator at Microsoft.

Possible workarounds

Uninstalling the update should fix the problem in all scenarios. On Microsoft Q&A) you can find the post VM's won't start since Windows Updates October 2023 in which someone describes the procedure (renaming the VHDX files and adjusting the configuration file with regard to the renaming) with which he could solve the problem for himself.

Within my German blog, users also gave hints. It seems that in the folder with the VHDX files, additional (helper) files (.cdb = Changed Block Tracking; mrt = Microsoft Resilence Tracking; crt = Resilient Change Tracking) are causing the problem. Deleting these files (except the .vhdx files) should fix the error (without the necessity of renaming the VHDX files). However, I'm not sure how broadly this approach works.

Addendum: I received a German comment, after publishing the German article, pointing to the Veeam forum thread Windows Server 2019 Hyper-V VM I/O Performance Problem. There a Microsoft guy has posted on October 9, 2023, that the long awaited 'RCT fix', for the RCT side, should be released in the October 2023 Windows update for WS2019 and WS2022. Maybe it the culprit, because the guy wrote, that this ist for the issue with general VM IO slow down with .rct files in use.