Hyper-V VM Shutdown issue in Windows Server 2019

[German]There is an issue with Hyper-V on Windows Server 2019 (and possibly Windows Server 2016). The Hyper-V platform cannot shut down virtual machines properly.


Advertising

German blog reader Oli has informed me about the issue by mail a few days ago (thanks for that). Since it might be of interest to other administrators, here are some details. Oli wrote:

Windows Hyper-V Server 2019 – Shutdown issues with VMs

Since I just have this issue myself, here is a hint about a but in Hyper-V Server 2019 (and possibly Hyper-V Server 2016 as well?):

VMs, if you set them to “Shutdown” instead of “Save” (aka Snapshot, which is not a good idea e.g. for virtualized domain controllers), will not be shut down properly. After booting you will be greeted with the “Windows wasn’t shut down properly” message.

The joke is that this issue with Windows Hyper-V Server 2012R2/8.1 AND Windows Server Hyper-V 2016 also already occurred in the past, or occurs with Windows Hyper-V Server 2016 now again.

I didn’t have had this issue because I’m coming from Windows Hyper-V Server 2012 and the problem never occurred in 6 years. Maybe this is interesting for one or the other admin.

Oli sent the link to this Technet forum thread, where the problem has also been discussed.

Hyper-V 2019 – Guest VM’s shutdown unexpectedly

Since upgrading my Hyper-V servers to Windows Server 2019, including KB4471332 from last night, my guest VM’s do not power down on restart of the 2019 host.

I can confirm integration components are as up to date as possible, and each guest is configured to shutdown when the host restarts.

Each guest gets the “Why did I shutdown unexpectedly message”.

The even viewer yields the following all within 30 seconds of restarting the host: Shut down physical computer. Stopping/saving all virtual machines…

‘ADFS’ failed to perform the ‘Shutting Down’ operation. The virtual machine is currently performing the following operation: ‘Shutting Down’. (Virtual machine ID 0CFDF648-EE9D-4141-9EBD-9A6D911C3442)
‘ADFS’ failed to shut down. (Virtual machine ID 0CFDF648-EE9D-4141-9EBD-9A6D911C3442)

The Update Orchestrator Service service terminated with the following error: This operation returned because the timeout period expired.

Failed to restore configuration for port 59229EE2-C880-4BF2-9849-89A21EC17772 (Friendly Name: ) on switch 300C0E17-E123-4182-BE62-AAD1860BE841 (Friendly Name: ), status = Object Name not found..

Everything is configured as it was on the 2016 Hyper-V host.

The message ADFS failed to shut down because it is shutting down seems terribly odd to me.  Of course its shutting down, just need the host to wait. (Yes the registry is set to 120 as the default time for the host….)

Fairly simple to recreate….just spin up a 2019 server, install Hyper-V role, throw in a VM and test.

There the issue has been discussed since December 2018, with users confirming the issue. It’s unpleasant that this bug has been reported earlier within the support article KB289680 from Microsoft. In my view it’s stupid what the Microsoft employees posting within the Technet tread. A user wrote ‘posting a bug here in the forum is like discussing with a wall’. In any case, the hope of those affected that a fix would come in January 2019 was disappointed. Whether an update in February will solve the issue will have to be seen. Maybe it will help you if someone is affected.


Advertising


This entry was posted in issue, Virtualization, Windows and tagged , , . Bookmark the permalink.

12 Responses to Hyper-V VM Shutdown issue in Windows Server 2019

  1. Stephan says:

    Hi Born

    Sadly the bug is still here. I’ve updated the host and VMs today with the newest patches and the problem is still there. Shutdown the Host and the VMs get killed instead of properly shutted down.
    Just as information that this bug still seems to exist (Server 2019).

    Best regards,
    Stephan

    • Dave S says:

      I made a workaround today.
      Used a GPO shutdown PS script to shutdown the VMs. Then remove vmms from the PreShutdownOrder in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\

      • Bart van de Beek says:

        Instead of removing vmms from the PreShutdownOrder registry-key, you should simply move the entry past the gpsvc-entry, like so:
        DeviceInstall
        UsoSvc
        gpsvc
        vmms
        trustedinstaller

        If you remove it, vmms will keep running. Then TrustedInstaller will be shutdown. Normally not an issue, but can potentionally be an issue if TrustedInstaller is later (still before the actual shutdown is complete) re-started to do it’s work pre-installing files from updates. If vmms is then still running it might prevent the update from succeeding resulting in the next boot where windows will revert the updates and reboot again.

        Also, on slow hosts, make sure to add ample time in your shutdown-script to allow all VMs to shutdown gracefully. Issuing Get-VM|Stop-VM will only wait max. ~5 minutes total and VMs will be shutdown in parallel. You can use Get-VM|%{Stop-VM $_} to serialize that and stretch that to ~5 minutes per VM. If a VM still needs longer add some Sleep:
        Get-VM|%{Stop-VM $_;Sleep -s 300}.
        Or you might take a look at DWORD ShutdownTimeout
        HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization
        which defines the max wait-time per VM in seconds before turning it off ungracefully.

  2. mata says:

    Hello all,
    This issue will be fix with 2019.9.C update

  3. GeoBog says:

    Similar issue we experience with approx. 4000 servers on Windows Server 2016 but only when we reboot the hosts and there are Windows patches that have been installed. The main problem comes from the VMMS service which re-starts itself before host shutdown. The way we experience the issue is:
    – We trigger/schedule reboots of the host servers following patch deployment
    – VMMS performs a graceful shutdown of all Virtual Machines
    – VMMS service is stopped
    – While Windows installs patches, the VMMS service re-starts in the background
    – The Virtual Machines are started automatically
    – The host server is finished processing the patches and is finally shutting down/rebooting

    The problem is that when the host is finally rebooting, it will not stop VMMS again and this leads to the Virtual Machines being practically power-cut instead of gracefully shut down. We have not found the reason yet why VMMS service re-starts itself before the host shuts down but the solution presented here with the PreshutdownOrder key seems to be a good workaround.

    • Stuart says:

      Sadly not working for me. Added the reg change described by Dave on the Hyper-V Host. Still 2019 VMs have dirty shutdown. Is that all you did to fix the issue?

  4. Advertising

  5. Kamil says:

    Is it fixed already?

  6. Lol Phirae says:

    It’s fixed starting with KB4520062 (not available on WSUS) and KB4523205 (2019-11 cumulative update, also available on WSUS). I.e., build 17763.832 or later. Note: Do not waste your time looking up the issue in their useless KB notes, MS could not be bothered to document it there.

  7. Stuart says:

    Not fixed for me, the KB above dont help. Windows 1o Pro (2004 and prev build), with server 2019 servers hosts. Reboot or shutdown hosts from either inside the OS or via virtual machine manager. VMs report dirty shutdown. Will try the method above, but please be clear the above KBs do not help.

    My 2019 vms are running latest patches.

  8. Stuart says:

    Ok this is weird, I have gone through event logs, and despite the ‘unexpected shutdown’ dialog box, I dont see any errors in application or system logs related to unexpected shutdowns.

    Perhaps a red herring, false positive?

    I see a 1074 event log:

    The process C:\Windows\system32\SystemSettingsAdminFlows.exe (CAISSUING01) has initiated the restart of computer CAISSUING01 on behalf of user CCS\LabAdmin for the following reason: Other (Unplanned)
    Reason Code: 0x5000000
    Shutdown Type: restart
    Comment:

    No other logs. I see all the services shutdown in system log, and start back up, so why is this ‘unexpected shutodown’ message appearing!

  9. Hi!

    I have just found this at technet. It is not exactly the reason why we have such weird problems but it is worth checking for further reasoning:

    https://social.technet.microsoft.com/Forums/en-US/79ceb8b5-a52e-4a84-8dcd-80209f9b7e1a/why-does-hypervvmswitch-produce-an-error-log-of-event-id-15?forum=win10itprovirt&prof=required

    Here is an excerpt of the what I would like to bring to our attention:

    Hi,

    Event ID 15 is logged if you have orphaned “Port ID’s” under your \SwitchList inside the Registry. These orphaned entries gets mostly created if you import a VM from another Hyper-V Server.

    Unfortunately, VMMS.exe doesn’t make an evaluation during the import operation, instead it creates the exact same “Port ID” which was used on the Source Hyper-V Server where you exported the VM.

    As a result you will have lots of orphaned Port ID’s once you import the VM on another Hyper-V Server. A detailled Root Cause Analysis of the Event ID 15 and how you can detect and delete oprhaned Port ID’s can be found under the following article.

    http://www.kuskaya.info/2020/04/19/how-to-troubleshoot-event-id-15-from-source-hyper-v-vmswitch-failed-to-restore-configuration-for-port-xxx-on-switch-xxx-object-name-not-found/

    Regards,
    Cengiz Kuskaya

Leave a Reply

Your email address will not be published. Required fields are marked *