VMware ESXI 7.0: Windows Server 2012 R2 shows "non-accessible boot device" (Oct. 2021)

[German]It's an issue that seems to have blown up in some administrators' faces in mid-October 2021. VMware ESXI 7.0 administrators were getting a "non-accessible boot device" error message on virtualized Windows Server 2012 R2 machines. Then the server would no longer boot. The cause was a patch from Microsoft for the paravirtualized SCSI adapters.

Blog reader Casten had written me a short mail on October 17 (thanks for that), but the first post on the Internet I found so on October 14, 2021 – so a few days after the October 2021 patchday from Microsoft. I'll post the information here on the blog, haven't gotten around to it yet. In the forum post Issues with Microsoft Windows Security Updates October 2021 – 2012 R2(OS) someone asks:

Hi All,

It seems there was an issue with Oct 21 monthly quality rollup patching. we installed the latest patches on two 2012 R2 servers and then rebooted. After that, they are all not booting up and stuck refresh\BCD screen error.

is anyone facing the issue who ever worked on 2012 R2 with Oct Patches…?

Regards
CHANDU

Another user then confirms these issues with Windows Server 2012 R2 running VMware ESXI 7.0:

I have three virtual machines exhibiting the same behavior. They are VMware VMs. Are you using the VMware Paravirtual SCSI driver? I am on all three of these VMs and the recovery-thing Command Prompt was not originally able to find my virtual HDD. I added a virtual LSI Logic SCSI adapter and moved the vHDD to it, and now the recovery-thing Command Prompt is able to see the data (C: is the Recovery partition, D: is my normal C: drive) but they are still unable to boot. The various safe mode, recovery modes, etc unfortunately do nothing.

Unfortunately I have no fix for you, but wanted to share a little about my environment and the troubleshooting I've done thus far in case we can figure out a solve.

Shortly after, the reddit.com thread Server 2012 R2 on ESXI 7.0 non-accessible boot device from Windows paravirtual SCSI patch was created. It says that the security only update for Windows Server 2012 R2 has a patch VMWare, Inc. – SCSIAdapter – 1.3.18.0 that causes problems. The machine goes into repair mode and can no longer boot.

This was a fun one, figured to write it up to hopefully help someone else. When patching Windows 2012 R2, MS Catalog is showing Windows 8, 8.1, 2012 and 2012 R2 ) directly from MS there was an update – VMWare, Inc. – SCSIAdapter – 1.3.18.0 Side note VMWare is stating 1.3.17 as the latest from 9/16 – https://kb.vmware.com/s/article/82290

When applying the update, along with other patches, it caused the machine to go into automatic repair mode (possibly blue screen but it did not display as such) and no drives would show in diskpart when opening the command prompt. Attempting to do normal BCDedit and bootrec commands to repair did not fix the issue.

The thread creator also provided a workaround right away.

In repair mode remove all pvscsi.inf drivers installed and install the one from the VMWare tools install of the ESXi host.

Take a snapshot first just in case.Boot into windows recovery and start the command promptMount the vmware tools through Vcenter either via the datastore location or guest ->install tools -> mountrun the following commandsNote: D should be the mounted tools ISO if not skip down to the diskpart command to see which drive letter you're using for the ISO

d:
cd "program files\vmware\wmware tools\drivers\pvscsi\win8\amd64"
drvload .\pvscsi.inf

Should come back as drvload: successfully loadedYou should now see your drives, you may have to assign a drive letter to the "C" drive but in my cases i didn't. To verify / view do the following

diskpart
list volume
exit

Note the letter of the C drive, mine was always E when i checked.Now run the next set of commands to get all the base drivers installed on the system. Note this will take a hot minute depending on your system

dism /image:E:\ /get-drivers

Take note of all the oem<number>.inf names that have pvscsi.inf listed by them the other ones for vmnet3,display, printer, etc should be ok to leave alone. you will want to remove all of the pvscsi.inf ones. Do the following BUT CHANGE THE NUMBER ON OEM!! Don't run it copy paste blindly

dism /image:E:\ /remove-driver /driver:oem1.inf

Once you've gone back and removed all of them, most likely there will be more than one if the server has been around a while, you will now add in the one back from vmware tools that you still have mounted. Note: at this point you still should be in the iso mounted drive on D. otherwise the following command you'll need to adjust to where the scsi driver is. Run the following

dism /image:E:\ /add-driver /driver:".\pvscsi.inf"

After success on there, you can exit via

exit

then choose continue to booting. System should now boot.Make sure to remove your snapshot, once successful, and remove tools mounted.

In the meantime VMware has also published a KB article Removal of Windows Update published PVSCSI driver version 1.3.18.0 (86053).