[German]Microsoft has now delivered a temporary fix for the year 2022 bug that prevents (on-premises) Exchange Servers to transport mails since January 1, 2022. Since I expect that Monday, Jan. 3, 2022, some people will discover that Exchange Server is "broken," I'm summarizing everything worth knowing again in this addendum.
Advertising
The problem
As of Jan. 1, 2022 00:00 (UTC), many (on-premises) Exchange Servers worldwide will no longer be able to transport email. The problem is that a signature update for the FIP-FS "Microsoft" scan engine created a conversion error so that the engine no longer starts. The event can be found in the logs:
The FIP-FS "Microsoft" Scan Engine failed to load. PID: 39268, Error Code: 0x80004005. Error Description: Can't convert "2201010003" to long. / Event ID 5300
The background is that the signature file probably contains the encoded year date (2201010001 or a higher last digit) and when converted to a signed long integer value it creates an overflow which then causes the above error abort. I first reported this in the blog post Exchange Year 2022 Problem: FIP-FS Scan Engine failed to load – Can't Convert "2201010001" to long (2022/01/01 00:00 UTC) and mentioned a temporary workaround.
Who is affected?
Microsoft writes in the Techcommunity post Email Stuck in Transport Queues that Exchange Server 2016 and Exchange Server 2019 are affected. Frank Carius writes in this German post that Exchange 2016 or Exchange 2019 must route mail to be affected. There is also the statement that Exchange Online are not affected and also Exchange 2013 is not affected if you are not hidden behind Exchange 2016/2019 servers (Hybrid Routing).
However, this is contradicted by the statements in these German comment to my blog post Exchange Year 2022 Problem: FIP-FS Scan Engine failed to load – Can't Convert "2201010001" to long (1.1.2022), where Exchange Server 2013 was mentioned as affected.
Advertising
In addition, to trigger the above error, the malware agent must be installed. In the above tweet, someone points out that the anti-spam component is not installed by default. That would be the explanation that not all Exchange servers are affected.
The temporary fix from Microsoft
Microsoft provided an explanation for the problem in the Techcommunity post Email Stuck in Transport Queues (see also my blog post Microsoft confirms Exchange Year 2022 problem that FIP-FS Scan Engine failed to load (Jan. 1, 2022)). A few hours later, a temporary fix was then provided in an update of the article, but it requires administrator intervention for affected systems.
The fix consists of a PowerShell script https://aka.ms/ResetScanEngineVersion, which can be downloaded and then executed. It should be noted that the PowerShell Execution Policy must be activated for the script to run, using the following command:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned
The script must also be run in an administrative Exchange Management Shell of the affected Exchange server. The script stops certain services, deletes the folder containing the scan engine signature files and then requests a new signature update. Microsoft has resorted to a trick and provided this update with the signature version 2112330001 or higher. This way, no conversion error occurs anymore.
If you prefer to do the whole thing manually, Microsoft has published step-by-step instructions in the Techcommunity article Email Stuck in Transport Queues.
What else you should know
In this German comment Michael reports that the download of the EngineUpdates got stuck at 92% and about 179 MByte. I also got a similar hint on Facebook to my posts as feedback. This could be due to the load of the Microsoft servers. It can take half an hour until the download is finished (see this German comment).
Important: If you used the quick workaround I mentioned in the blog post Exchange Year 2022 Problem: FIP-FS Scan Engine failed to load – Can't Convert "2201010001" to long (1.1.2022), you have to undo it afterwards. Robert writes in this German comment:
What worked for me was that after applying the fix (running the script) I first completely rebooted the server. Then I ran Enable-AntimalwareScanning.ps1 again and restarted the transport service.
Maybe this will help one or the other affected person. If there is an error 0x84004003 when running the Microsoft fix, the server running Exchange should be rebooted and the attempt repeated (see this German comment).
If the mail transport failure lasts longer, administrators will face two problems. First, the accepted mails cannot be delivered any further. Then the queue of "submissions" will be fully routed to the mailboxes. Possibly this can lead to a full partition with the Queue.dat, as Frank Carius explains here (German). And there is the possibility that received but not delivered mails older than 48 hours are deleted.
Similar articles:
Exchange Year 2022 Problem: FIP-FS Scan Engine failed to load – Can't Convert "2201010001" to long (1.1.2022),
Microsoft confirms Exchange Year 2022 problem that FIP-FS Scan Engine failed to load (Jan. 1, 2022)
Advertising
