[German]I'll pull it out separately as a blog post. Administrators of Windows Domain Controllers should be careful about installing the January 2022 security updates. I have now received numerous reports that Windows servers acting as domain controllers will not boot afterwards. Lsass.exe (or wininit.exe) triggers a blue screen with the stop error 0xc0000005. It can hit all Windows Server versions that act as domain controllers, according to my estimation.
Advertising
January 2022 updates address Active Directory bug
I listed it in the Patchday blog posts linked at the end of the article. In all the security updates for Windows Server (e.g., Update KB5009624 (Monthly Rollup for Windows 8.1 and Windows Server 2012 R2)), it states:
Addresses a Windows Server issue in which Active Directory attributes are not written correctly during a Lightweight Directory Access Protocol (LDAP) modify operation with multiple specific attribute changes.
However, something seems to have gone wrong, because the security update can trigger a boot loop on Windows servers that act as domain controllers.
Boot loop on Windows Server DCs
German blog reader John L. contacted me via email back on January 11, 2022, and pointed out a fat problem related to the update. The module lsass.exe, version: 6.3.9600.17415, triggers an error 0xc00005 (access violation) via the library msv1_0.DLL, version: 6.3.9600.20239, so that the server gets into a boot loop.
""Name of the corrupt application: lsass.exe, version: 6.3.9600.17415, timestamp: 0x545042fe
Name of the corrupt module: msv1_0.DLL, version: 6.3.9600.20239, timestamp: 0x61c1a5c8
Exception Code: 0xc0000005
Fehleroffset: 0x0000000000002663
ID of the faulty process: 0x1f4
Start time of the faulty application: 0x01d8072ac5b2c15a
Path of the faulty application: C:\Windows\system32\lsass.exe
Path of the corrupted module: C:\Windows\system32\msv1_0.DLL
Berichtskennung: afc36fda-7320-11ec-813a-00155d012601
Full name of the corrupted package:
Application ID relative to the corrupted package: "".
I had already addressed this in the blog post Patchday: Windows 8.1/Server 2012 R2 Updates (January 11, 2022), boot loop reported, possible boot issues. John had the following advice:
I want to advise against rolling back snapshots, especially on DC's, so as not to provoke USN rollbacks.
Workaround: prevent one of the two DC's from booting, then uninstall today's hotfixes first on one and then on the other DC.
In the comments of my blog post above (and its German counterpart), other blog readers confirm this problem. The workaround is, to uninstall the January 11, 2022 security update.
Advertising
Tip: To avoid that the DC restarts too quickly during uninstall, just deactivate the network connection (pull the plug or deactivate the network driver).
German blog reader MOM20xx had the boot loop even after uninstalling the update and notes that the security-only update KB5009595 should also be uninstalled on the domain controllers.
Probably affects all versions of Windows Server DCs
German blog reader Simon wrote in this comment that it also affects Windows Server 2016/2019 Domain Controllers too. He then posted the following dump excerpt.
The process wininit.exe has initiated the restart of computer DC on behalf of user for the following reason: No title for this reason could be found
Reason Code: 0x50006
Shutdown Type: restart
Comment: The system process 'C:\Windows\system32\lsass.exe' terminated unexpectedly with status code -1073741819. The system will now shut down and restart.Faulting application name: lsass.exe, version: 10.0.14393.4704, time stamp: 0x615be0cd
Faulting module name: lsadb.dll, version: 10.0.14393.4886, time stamp: 0x61d5242f
Exception code: 0xc0000005
Fault offset: 0x000000000001be5b
Faulting process id: 0x2a8
Faulting application start time: 0x01d8077b1080a9da
Faulting application path: C:\Windows\system32\lsass.exe
Faulting module path: C:\Windows\system32\lsadb.dll
Report Id: e14067b5-aac7-46a4-9e21-cc45371c522a
Faulting package full name:
Faulting package-relative application ID:
So there wininit.exe triggers the error 0xc0000005 on the domain controller. I also have another feedback on Facebook that update KB5008873 on Windows Server 2019 is causing the restart of the AD controllers (the AD controller is restarted every 15 minutes).
Boot-Loop on Windows Server 2019
If anyone needs some more hints on how to uninstall the update in a Windows PE environment, I'll refer them to How to Remove Updates from Windows Recovery Environment (WinRE).
Note: According to this German comment, update KB5009543 causes problems with L2TP VPNs. On reddit.com there is this thread about it. See also the links below.
In addition, I got reports that VMs on Server 2012 R2 Hypervisor do not start anymore. The error message is that the hypervisor is not running: Hypervisor launch failed; The operating systems boot loader failed with error 0xC00000BB. This is probably update KB5009624 for Server 2012 R2 – just as a hint, if there should be problems under Windows Server 2016 – 2019. See also the links below.
And we have reports, that the Windows Server 2012 R2 January 11, 2022 security update removes ReFS support.
Similar articles:
Microsoft Office Updates (January 4, 2022)
Microsoft Security Update Summary (January 11, 2022)
Patchday: Windows 8.1/Server 2012 R2 Updates (January 11, 2022), boot loop reported
Patchday: Windows 10 Updates (January 11, 2022)
Patchday: Windows 11 Updates (January 11, 2022)
Patchday: Updates for Windows 7/Server 2008 R2 (January 11, 2022)
Windows Server: January 2022 security updates are causing DC boot loop
Windows VPN connections (L2TP over IPSEC) broken after January 2022 update
Windows Server 2012/R2: January 2022 Update KB5009586 bricks Hyper-V Host
Advertising
Trackback: https://threatpost.com/microsoft-yanks-buggy-windows-server-updates/177648/
I have the same problem this morning after having updated this NIGHT 2 updates:
1 SECURITY called KB5009557
1 Update for Microsoft Windows KB5008873
my server is WINDOWS Server 2019 version 1809 OS Build 17763.2366
many reboot … 10 till this morning, before the reboot a lot of kernel power (CPU 100%)
thank you
Many thanks for this. You're correct with the Server 2012 R2 Hypervisor not starting and KB update number. We were hours trying to resolve this as our HyperV virtual servers wouldn't start. Eventually came across your site, uninstalled update, rebooted and it now works fine. I wouldn't have worked this out to be honest if it wasn't for your site.
Echoing this comment – same behavior, same fix. Uninstalling Update KB5009624, reboot, then hypervisor started and we were able to start VM's. Thank you very much for this thread!
Yup, Completely BRICKED MY HYPER ADVISOR as well. Rollback worked and I avoided a coronary episode.
Thank you.
Having Hypervisor errors on a host after the January updates here too.
Life Saver! Thank you!
Not sure if it's the solution, but I added the registry key below and the server hasn't rebooted for 40 minutes now
[HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\Kdc]
"PacRequestorEnforcement"=dword:00000000
Hi Jonathan!
Is the server/servers still running ok after the regfix!
FYI – That key was introduced with a November patch for kerberos security. A later patch will enforce it and key won't do anything. But that was planned for the summer, hopefully MSFT has an actual fix by then.
KB5008380—Authentication updates (CVE-2021-42287)
to bad this did not work for me. I have 2 DC with 2012r2, the updates did fail during install, so there is nothing to remove.. But still got loops. Need te restore them from backups..
Great job MS..
Since than, has solved your problem?
I was desperately searching for hints this morning and couldn't find much until I stumbled upon your post. Thank you! One tip that helped me is that I could catch the DC before a reboot was triggered (about 2 minutes after startup) and then stop the Netlogon service. That stopped the rebooting such that I could troubleshoot and uninstall the update. Thanks again!
Dave, the net stop netlogon tip saved our bacon. Leave it to Microsoft to release a zero-day DC boot loop to their own customers.
We had similar issue this morning with Windows Server 2012 DC not R2 that kept going in reboot loop every few minutes of showing the signon screen of Windows. It is a VM running on vmware.
We booted the VM to safe mode (by pressing F8 before Windows started) and removed all last night updates and rebooted. Seems to have fixed the issue. We have changed the server to not install Windows updates automatically to avoid such issues in the future.
I will report back if anything changes
Same Here, 20 servers…
Does anyone have a comprehensive list of KB's for all server versions that are causing this issue?
Blame all Server updates from 2012R2 up to 2022 – see also the linked articles at the end of my text.
On windows server 2012 (not R2) KB5009586
Lifesaver!!! This is exactly what happened to us, it was KB5009624.
Our W2k12 DC was updated on Wed, 3:00 am. It did not reboot the hole day but restarted at night at about 9:00 pm cause of the lsass issue, and about 20min after installing exchange 2k13 updates this evening. We do have another (unpatched) W2K19 DC in a very a small enviroment.. so I guess the initial vector causing the app crash is trigged much faster in larger enviroments?
For me it always happens when user trying to authenticate with AD or join Domain.
I have the Problem since today but with other KBs installed.
Windows Server 2019
I got the same issue today…
2012 DC
Yes, VMs on Server 2012 R2 Hypervisor didn't start anymore. The error message was that the hypervisor is not running.
Uninstalled the KB5009624 updates and rebooted the HOST and now the VMs are starting just fine.
Thank you very much for this. You are the only one on the net speaking about this issue.
I got it yesterday. Luckily I was able to uninstall the latest update.
Same Issue with all Domain Controllers, Windows Server 2016 and Windows Server 2019!
KB5009546 (2016 Server)
KB5009557 (2019 Server)
KB5008373 (2019 Server)
Thank's Microsoft !!!! :( :( :( :( :(
Failed to install Monthly Rollup KB5009624, reboots still triggered by KB5009595. However, VMs on Patched Server 2012 R2 Hypervisor are still up and running, only DCs are affected.
Thank you for this, uninstalled both updates and it has stopped the unexpected reboots.
Dave Kelley, your solution works 100% for us. Thank you!
The moment you first logon.
1. Immediately run command prompt, right click, run as administrator (just incase)
2. type in "net stop netlogon" (shows The Netlogon service was stopped successfully.)
3. Program and Features
4. Select Uninstall (click on Security Update for Microsoft Windows (KB5009557)
5. You must restart your computer to apply these changes (click Restart Later)
6. Select Uninstall (click Update for Microsoft Windows (KB5008873)
7. You must restart your computer to apply these changes (click Restart Now)
Reboots and Walla.
Thanks for sharing this, I ran on the same problem today.
I had two clients this morning saying their email services were down. I tried starting the services and coincidentally noticed the Domain controller rebooting.
I did one final search for guidance and found your post.
Thank you for your help…
For your information this related to two servers with the following configurations :
Windows 2012 R2 Server with a HyperV running a Windows 2012 R2 with Exchange.
I had the same issue and managed to resolve it today by using the
NET STOP NETLOGIN command without issue.
I tried this on another server which was experiencing the same issues.
This server kept on restarting the service even though I stopped it.
I was forced to continually in elevated command prompt windows to use the NET STOP NETLOGIN to stop the service as soon as it started and this stopped the restart of the server.
So the next step that I found that would allow further troubleshooting of the issue was to goto services.msc and find the Netlogon Service and actually disable the service so it could not restart.
Once disabled I could then have an Operating System that was not restarting after boot, and I could then remove the latest updates.
What an unneccesary headache and mission, Happy New Year Microsoft..
I have disabled all Windows 2012 R2 Updates on the affected servers awaiting confirmation that any future updates will not cause the same issues.
If this helps you out, I would appreciate some feedback on how I address a circumstance where we are supporting a server overseas:
===================================================================
I have a server which I remotely control in another country which has a pending update for these updates so it has been downloaded and pre-installed but I do not want it to be installed. Does anyone know how to uninstall a half installed update as this will cause major dramas like another site once it reboots ? In the Windows Updates view this is considered to be Pending update.
It seems that this is related to the following Microsoft KB on our Windows 2012 Server :
KB890830
KB5009586
* KB5009720
KB5009721 2022-01 Security Monthly Quality Rollup for Windows Server 2012 R2 for X-64-based Systems
KB5009624 2022-01 Security Monthly Quality Rollup for Windows Server 2012 R2 for X-64-based Systems
* KB5009624 2022-01 Security Monthly Quality Rollup for Windows Server 2012 R2 for X-64-based Systems
========= I think this is the main culprit but I have not been able to confirm that it is this one
KB5008263 2021-12 Security Monthly Quality Rollup for Windows Server 2012 R2 for X-64-based Systems (seems OK)
IMPORTANT
=========
and after removing the affected updates I highly recommend that before you reboot, you re-enable the Netlogon service as you may not be able to login to the server after the rollback of the updates, if this service does not start.
We were also affected by a Microsoft Exchange update for 2016 which stopped all emails after 1 Jan 2022. So not a good start to the year really.
Good luck and hope this helps the community out.
I hope you are all and you keep well with the current COVID situation.
Regards
David Nyssen (BCompSc BFinAdmin)
DN Computer Services
Managing Director
@David: Thanks for reporting.
Just an explanation – I've deleted your other 2 of your 3 comments. I let all new comments – and comments with links – flow into moderation, to avoid SEO spam. After reading the comments, I will enable or delete them.
I uninstalled the last update and the services came back up.
Thank you !!!!
i had the same problem
two DC servers with Windows 2012 (HyperV running)
Everything worked again after removing the updates!
I have a Server 2016 RODC that is still stuck in a reboot loop even after removing all of the January patches. Has anyone else ran into this?
It's running on Hyper-V, if I disable networking the machine boots up but as soon as networking is re-enabled it reboots due to lsass.exe again.
I know why i hate Windows
All my servers are linux machines, running for ages without problems.
We only have one Windows 2019 Server DC in out network and to be honest, i had not run any updates for it for half a year. Yesterday (2022-01-14) i moved it to a different Hypervisor, ran the 4 pending updates and BAM, ran into this booting hell. Server boots up, i cannot login, it stays like this for 2-3 Minutes, then direct reboot (no BSOD)
took me a while to figure out how to boot into safe mode, then uninstalled at first
KB5008873
still reboots. then uninstalled
KB4589208
still reboots. then uninstalled
KB5009557
took almost an hour to uninstall but since then no reboots anymore.
honestly, this is an enterprise product. it costs alot of money. i payed for licenses
is there any quality control involved in these products?
It is a very simple install with no specialities.
This ruined my whole free day until now on the weekend.
lousy quality, i cannot understand how things like this pass Q&A!
sorry for the rant
regards from germany
I tried removing the above updates, still having an issue. for some reason it disables winrm service and you cannot enable to service even after removing the updates. Had restored server to previous backup. It appears any of the the updates that came in in January affect Windows server 2012 R2 and once removed there may be some services that are no longer working.
Microsoft is stupid, I uninstalled the update and then set the updates to manual.
I woke up to Windows Server 2016 making this annoying update again.
And now on Sunday I came to DC to fix what Microsoft can't do.
Bill Gates, leave me alone!
I'm going to migrate to something else, I'll pay you thousands of dollars on these licenses anyway.
You guys saved my ass, thanks a lot for your blogpost! Hope Microsoft pulls this updateds. Disabled network, uninstalled all the recent updates, restart, add network, works again.
This was the fix for us as well. 4 2012r2 DCs across 3 sites all boot looping after updates last night. Removing 9624 and 9595 fixed it for us. Server uptimes are now just over 40min, I'll report back if we see anything. Thank you so much for this information
Thanks for this info – I had the same problem on server 2016 DC.
After uninstalling KB5009546 and reboot, I started to get BSOD 21a stop code.
I almost had a heart attack.
Boot Windows Without the Driver Signature Enforcement Feature – recovered the BSOD.
You saved our lifes here thank you!
I have uninstalled the updates on my server 2012 R2 server and Hyper V is still not starting correctly. Any suggestions?
Thanks for the article, thousands, if not millions of admins were saved by it.
MS on the other hand, is STILL actively distributing this update, even after acknowledging this problem. I stopped the Windows Update during the initial stress period, but after a FEW days, I'm prompted with a reboot because of the SAME UPDATE!!
So you have to expressly DISABLE the Windows Update service for now.
Thanks for this post. One more thing that I am noticing is on some installations of the server you have to remove the KB prefix in the command. I got this tip from here – https://www.prajwaldesai.com/fix-windows-server-2012-reboot-loop-issue/
Thanks for this post! I'm running Server 2012 and now I see (3) updates as of 1/17/22. KB5009720 – .NET, KB5009586 – Security Rollup & KB890830 – MSRT. Anyone know if these are fixed updates and are the OK to install? I'm holding off for now. Thanks!
Excellent Post. Thanks alot for your work. I thought we would be forced to rebuild our server. You have saved us alot of downtime. Cant thank you enough!
Thanks Folks
Appreciate the effort. For me running 2012 R2, getting rid of KB5009624 fixed my Hypervisor & getting rid of KB5009595 stopped the DC boot loop!!
Cheers Again
I posted the below over at Spiceworks and I thought it might help:
After having spent 12 hours on this :((, a few things I'd like to add:
Uninstalling KB5009624 did not solve the problem for me. I also had to uninstall KB5009595. KB5009624 is the '2022-01 Security Monthly quality Rollup'. KB5009595 is the '2022-01 Security Only Quality Update'.
Safe mode can be used to stop the domain controller from rebooting and I was able to uninstall KB5009624 in safe mode, Uninstalling KB5009595 however, resulted in an error and a rollback of the uninstallation (effectively re-installing it). I was getting a message: 'We couldn't complete the updates. Undoing changes' I was able to uninstall KB5009624 when I exited safe mode.
We have now out-of-band updates – see Windows Out-of-band Updates fixes Jan. 2020 patch day issues (Jan. 17, 2022)
Thanks, however the OOB patch for Windows Server 2019 is still missing.
Wait till this afternoon (8 p.m. UTC)
I also got to this post from an article with this lovely issue against Windows Server 2016. I'll just comment that we're noticing that it doesn't appear to actually cause the reboot unless it's all of the DC's in a site. We have several sites that were mid patch where half of the DC's didn't have it, and half did. After we experienced it and rolled our first one back, the other stopped rebooting. We're still proceeding with uninstalls but I wanted to say thank you for posting this.
Having the same issue tonight.
Applied this updates and the issues came up:
– KB890830
– KB5009721
– KB5009624 –> can confirm that this one seems to be the culprit
This steps fixed it for me (DC is running as VM under ESXi 7)
– shutdown affected VM
– disconnect network adapter
– restart VM –> no boot loop anymore
– patching with KB5008263
– shutdown VM
– connect network adapter and restart
– VM is running stable since 15 Min
Seeing this issue on Windows 2012 R2. Having several sites with Windows 2019 as DC, they are not affected.
We spent the majority of a weekend chasing this issue. Removing KB5009624 fixed the lsass.exe crashes and subsequent reboots on all domain controllers.
Beware of any AD integrated applications such as f.e. a Synology also indirectly triggering the reboots.
Also: Fun times when scanning a file on a HP reboots a DC.
did you try installing the out-of-band KB5010794 update for Server 2012 R2?
https://www.catalog.update.microsoft.com/Search.aspx?q=5010794
domain controller crashed with this error after certificate enroll request from other server
uninstalled all January updates on 2012 r2
now its working
Cannot fix the issue, PLEASE help!
The server is stuck on the black screen with the circling dots. The server is able to boot onto Safe Mode but is impossible to uninstall KB5009555 in Safe Mode.
If I disable NetLogon service in Safe Mode and disconnect the network cable, the server progresses to the logon screen but freezes there. Ctrl-Alt-Delete not responding. I'm desperate!