[German]There is an issue with Hyper-V on Windows Server 2019 (and possibly Windows Server 2016). The Hyper-V platform cannot shut down virtual machines properly.
Advertising
German blog reader Oli has informed me about the issue by mail a few days ago (thanks for that). Since it might be of interest to other administrators, here are some details. Oli wrote:
Windows Hyper-V Server 2019 – Shutdown issues with VMs
Since I just have this issue myself, here is a hint about a but in Hyper-V Server 2019 (and possibly Hyper-V Server 2016 as well?):
VMs, if you set them to "Shutdown" instead of "Save" (aka Snapshot, which is not a good idea e.g. for virtualized domain controllers), will not be shut down properly. After booting you will be greeted with the "Windows wasn't shut down properly" message.
The joke is that this issue with Windows Hyper-V Server 2012R2/8.1 AND Windows Server Hyper-V 2016 also already occurred in the past, or occurs with Windows Hyper-V Server 2016 now again.
I didn't have had this issue because I'm coming from Windows Hyper-V Server 2012 and the problem never occurred in 6 years. Maybe this is interesting for one or the other admin.
Oli sent the link to this Technet forum thread, where the problem has also been discussed.
Hyper-V 2019 – Guest VM's shutdown unexpectedly
Since upgrading my Hyper-V servers to Windows Server 2019, including KB4471332 from last night, my guest VM's do not power down on restart of the 2019 host.
I can confirm integration components are as up to date as possible, and each guest is configured to shutdown when the host restarts.
Each guest gets the "Why did I shutdown unexpectedly message".
The even viewer yields the following all within 30 seconds of restarting the host: Shut down physical computer. Stopping/saving all virtual machines…
'ADFS' failed to perform the 'Shutting Down' operation. The virtual machine is currently performing the following operation: 'Shutting Down'. (Virtual machine ID 0CFDF648-EE9D-4141-9EBD-9A6D911C3442)
'ADFS' failed to shut down. (Virtual machine ID 0CFDF648-EE9D-4141-9EBD-9A6D911C3442)The Update Orchestrator Service service terminated with the following error: This operation returned because the timeout period expired.
Failed to restore configuration for port 59229EE2-C880-4BF2-9849-89A21EC17772 (Friendly Name: ) on switch 300C0E17-E123-4182-BE62-AAD1860BE841 (Friendly Name: ), status = Object Name not found..
Everything is configured as it was on the 2016 Hyper-V host.
The message ADFS failed to shut down because it is shutting down seems terribly odd to me. Of course its shutting down, just need the host to wait. (Yes the registry is set to 120 as the default time for the host….)
Fairly simple to recreate….just spin up a 2019 server, install Hyper-V role, throw in a VM and test.
There the issue has been discussed since December 2018, with users confirming the issue. It's unpleasant that this bug has been reported earlier within the support article KB289680 from Microsoft. In my view it's stupid what the Microsoft employees posting within the Technet tread. A user wrote 'posting a bug here in the forum is like discussing with a wall'. In any case, the hope of those affected that a fix would come in January 2019 was disappointed. Whether an update in February will solve the issue will have to be seen. Maybe it will help you if someone is affected.
Hi Born
Sadly the bug is still here. I've updated the host and VMs today with the newest patches and the problem is still there. Shutdown the Host and the VMs get killed instead of properly shutted down.
Just as information that this bug still seems to exist (Server 2019).
Best regards,
Stephan
I made a workaround today.
Used a GPO shutdown PS script to shutdown the VMs. Then remove vmms from the PreShutdownOrder in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\
Instead of removing vmms from the PreShutdownOrder registry-key, you should simply move the entry past the gpsvc-entry, like so:
DeviceInstall
UsoSvc
gpsvc
vmms
trustedinstaller
If you remove it, vmms will keep running. Then TrustedInstaller will be shutdown. Normally not an issue, but can potentionally be an issue if TrustedInstaller is later (still before the actual shutdown is complete) re-started to do it's work pre-installing files from updates. If vmms is then still running it might prevent the update from succeeding resulting in the next boot where windows will revert the updates and reboot again.
Also, on slow hosts, make sure to add ample time in your shutdown-script to allow all VMs to shutdown gracefully. Issuing Get-VM|Stop-VM will only wait max. ~5 minutes total and VMs will be shutdown in parallel. You can use Get-VM|%{Stop-VM $_} to serialize that and stretch that to ~5 minutes per VM. If a VM still needs longer add some Sleep:
Get-VM|%{Stop-VM $_;Sleep -s 300}.
Or you might take a look at DWORD ShutdownTimeout
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Virtualization
which defines the max wait-time per VM in seconds before turning it off ungracefully.
Aug, 2022 – Windows Server 2019 up do date with last CU, 7 VMs. The problem is still exists. When I shutdown my host property, all VMs are dirty shutdown with unexpected shutdown in the next boot. It's really weird: MS does not know what to do? I've read many techs, but it's useless. The problem is when I have AC fails, VMs could really be crashed.
Hello all,
This issue will be fix with 2019.9.C update
Similar issue we experience with approx. 4000 servers on Windows Server 2016 but only when we reboot the hosts and there are Windows patches that have been installed. The main problem comes from the VMMS service which re-starts itself before host shutdown. The way we experience the issue is:
– We trigger/schedule reboots of the host servers following patch deployment
– VMMS performs a graceful shutdown of all Virtual Machines
– VMMS service is stopped
– While Windows installs patches, the VMMS service re-starts in the background
– The Virtual Machines are started automatically
– The host server is finished processing the patches and is finally shutting down/rebooting
The problem is that when the host is finally rebooting, it will not stop VMMS again and this leads to the Virtual Machines being practically power-cut instead of gracefully shut down. We have not found the reason yet why VMMS service re-starts itself before the host shuts down but the solution presented here with the PreshutdownOrder key seems to be a good workaround.
Sadly not working for me. Added the reg change described by Dave on the Hyper-V Host. Still 2019 VMs have dirty shutdown. Is that all you did to fix the issue?
Is it fixed already?
Not sure, some German blog reader told me, it's fixed – but I don't use Hyper-V
It's fixed starting with KB4520062 (not available on WSUS) and KB4523205 (2019-11 cumulative update, also available on WSUS). I.e., build 17763.832 or later. Note: Do not waste your time looking up the issue in their useless KB notes, MS could not be bothered to document it there.
Not fixed for me, the KB above dont help. Windows 1o Pro (2004 and prev build), with server 2019 servers hosts. Reboot or shutdown hosts from either inside the OS or via virtual machine manager. VMs report dirty shutdown. Will try the method above, but please be clear the above KBs do not help.
My 2019 vms are running latest patches.
Ok this is weird, I have gone through event logs, and despite the 'unexpected shutdown' dialog box, I dont see any errors in application or system logs related to unexpected shutdowns.
Perhaps a red herring, false positive?
I see a 1074 event log:
The process C:\Windows\system32\SystemSettingsAdminFlows.exe (CAISSUING01) has initiated the restart of computer CAISSUING01 on behalf of user CCS\LabAdmin for the following reason: Other (Unplanned)
Reason Code: 0x5000000
Shutdown Type: restart
Comment:
No other logs. I see all the services shutdown in system log, and start back up, so why is this 'unexpected shutodown' message appearing!
Hi!
I have just found this at technet. It is not exactly the reason why we have such weird problems but it is worth checking for further reasoning:
https://social.technet.microsoft.com/Forums/en-US/79ceb8b5-a52e-4a84-8dcd-80209f9b7e1a/why-does-hypervvmswitch-produce-an-error-log-of-event-id-15?forum=win10itprovirt&prof=required
Here is an excerpt of the what I would like to bring to our attention:
Hi,
Event ID 15 is logged if you have orphaned "Port ID's" under your \SwitchList inside the Registry. These orphaned entries gets mostly created if you import a VM from another Hyper-V Server.
Unfortunately, VMMS.exe doesn't make an evaluation during the import operation, instead it creates the exact same "Port ID" which was used on the Source Hyper-V Server where you exported the VM.
As a result you will have lots of orphaned Port ID's once you import the VM on another Hyper-V Server. A detailled Root Cause Analysis of the Event ID 15 and how you can detect and delete oprhaned Port ID's can be found under the following article.
http://www.kuskaya.info/2020/04/19/how-to-troubleshoot-event-id-15-from-source-hyper-v-vmswitch-failed-to-restore-configuration-for-port-xxx-on-switch-xxx-object-name-not-found/
Regards,
Cengiz Kuskaya
On my side, Windows Virtual Machines have no problem to shut down together with the HOST running Win 2k19 + HyperV .. but when VM is Linux .. than I have Dirty Shutdown in steed of Clean Shutdown like on Virtual Box or WMWare who was working fine via API daemon.
Anyone knows how to setup HyperV as HOST or Linux Ubuntu (or Centos or Debian) as Virtual Machine … in order to have a Clean Shutdown while Host Reboot ?
Thank you for attention. I appreciate your advice's !
Regards,
Bogdan Florin
Issue not solved in Feb-03-2021, on shutdown or reboot all Hyper-V VM's don't get proper shutdown, even if Automatic Stop Action is set to SAVE. So if you select Save or Shutdown the result is the same = improper shutdown.
It's incredible how a critical option for Hyper-V VM's is still not working after all these years.
Windows Server 2019 with the latest updates.
the same issue! last updates are present.
2021,Feb 3 pass midnight , so the 4th, running latest Windows update on a 2019 Server with Hyper-V. and bang it hangs at this exact same problem. Note the Host was rebuild in December 2020, and has previously rebooted while running VMs on it OK and VMs did not reported dirty shutdown… Is this a returning bug in the latest update…
Hi
I have 2 HyperV with basically idenitcal HW specs. One is rebooting just fine either saving or shutdown VMs and the other HyperV just shutds down and kills the VMs – this is driving me nuts an is really annoying. I whis I knew how to fix / workaround it….
I hope my comments can help clarify the problem:
I have a Dell PE R410 running Server 2019 (eval license installed on 2/11/2021) with a VM installed using the same media and same eval license. The VM always complains about an unexpected shutdown whenever the host machine is restarted, even if it was OFF. Why is the VM complaining about an unexpected power-off when it's already off? The automatic start/stop options have no effect on the problem.
I am also experiencing a similar the same problem on a Windows 10 Pro Build 2019 and later on a 20H2 system where my Ubuntu 18.04 LTS Linux guest OS seem to shutdown/restart without any issues.
But when I attempt to shutdown a Windows 10 Pro or Windows 2019 Server guest OS on the same system the Hyper-V environment appears to lock up and report following error.
An error occurred while attempting to turn off the select virtual machine(s).
"Windows2019Server_Trial" failed to change state.
The operation cannot be performed while the object is in its current state.
When I am in state the only way I can stop my virtual machine is to hardware reboot the Windows 10 Pro host system.
I have also noticed the following message in the Windows 10 Pro host system "System Event" logs when it processing its the shutdown procedure.
"The Hyper-V Virtual Machine Management service did not shut down properly after receiving a pre-shutdown control."
I also checked the system for any corruption using the sfc /scannow command and it reported no errors.
This problem I am having seems to have developed on my system quite recently where before my Windows VMs would shutdown or restart without any issues. I am wonder whether a Microsoft upgrade has broken something recently.
I would be interested to finding out if anyone else is experiencing this problem.
Hi,
I have still problem that my VM's are not shutting down properly. The Host is a Win2022 and the Guest is a Windows 2019. It's angry because my Hyper-V will not shutdown in case of an Energyfault…….
What's up with Microsoft.
I updated the Server and still the Problem or am I'm wrong and did not see another problem?
Bye
Workaround for this is to set the VM start behavior to do nothing and put a PowerShell script into Scheduled Tasks on the Hyper-V host (a minute or so after startup) with following content (server names are the names you have set in your Hyper-V Manager, sequence is your choice too):
$VMs = @("Server01","Server02","Server03","Server05") #Server names in wanted sequence
$Delay = 30 #delay between startup of VMs
$Retry = 3 #how many retries per VM
#check for running service Hyper-V Management and loop as long as not
while ((get-service vmms).Status -ne "Running") {write-host "Waiting for service VMMS" -ForegroundColor Red;Start-Sleep -seconds 10}
#If no VM list is given perform task for all machines on host – attention touches also VMs which are down intentionally therefore listed as comment)
#if ($VMs -eq "") { $VMs = Get-VM }
foreach ($VM in $VMs) {
if ((get-vm $VM).state -eq "Running") {
Write-Host "Server $VM is already running." -ForegroundColor Green
}
else {
write-host "Starting virtunal machine $VM" -ForegroundColor Yellow
while (((get-vm $VM).state -ne "Running") -and ($Retry -gt 0)) {
start-vm $VM
$Retry —
start-sleep -Seconds $Delay
}
$Retry = 3
}
#Ende foreach
}
@Michael: Shutting down would be a one liner in PowerShell:
get-vm | stop-vm
(as long as there is no special sequence required)
Best greetings from Germany
Olaf