[German]Exchange servers (on-premises) have had woldwide a Year 2022 problem since January 1 (00:00 UTC), because mails are no longer transported due to a "date error". Meanwhile, Microsoft has confirmed this issue in a separate blog post. The developers are working on a fix, which should be rolled out soon. Addendum: A fix ist available.
Advertising
The Exchange Year 2022 Problem
The year 2022 started for administrators of Exchange servers (on-premises) with a fat Year 2022 problem, because mails can no longer be transported due to a "date error". The Microsoft Scan Engine FIP-FS cannot be loaded – and in the logs the event can be found:
The FIP-FS "Microsoft" Scan Engine failed to load. PID: 39268, Error Code: 0x80004005. Error Description: Can't convert "2201010003" to long. / Event ID 5300
The background is presumably that a date value encoded in a signed long integer with 2022 generates an overflow, which then causes the above error abort. The workaround is to temporarily suspend the malware scan or malware filtering. I had published the article Exchange Year 2022 Problem: FIP-FS Scan Engine failed to load – Can't Convert "2201010001" to long (2022/01/01 00:00 UTC) with additional explanations.
What's with Exchange Emergency Mitigation Service?
There was also discussion on Twitter that the new Microsoft Exchange Emergency Mitigation Service introduced in 2021 (see Exchange Server September 2021 CU comes Sept. 28 with Microsoft Exchange Emergency Mitigation Service) was supposed to allow automatic remediation by Microsoft.
However, the deployment of such a solution takes time, as explained by Microsoft below.
Advertising
Microsoft confirms the bug
German blog reader DW points out in this comment (thanks for that) the Techcommunity post Email Stuck in Transport Queues that Microsoft has since published. It says there that Microsoft is aware of an issue that causes messages to get stuck in Exchange Server 2016 and Exchange Server 2019 transport queues. There is no mention of the fact that, according to my information, this also affects Exchange Sever 2013.
The issue relates to a bug in the year-end date check, not a bug in the AV engine itself. It is not a malware scanning or malware engine issue, and it is not a security-related issue. The signature file version check causes the malware engine to crash, which causes messages to get stuck in transport queues. So that's what I described a few hours ago in the blog post linked above.
Microsoft is working on a fix for this problem. They expect to release details on how to resolve this issue later today (at the time of writing, this was not yet the case).
In the meantime, Microsoft suggests running malware scans on messages outside of the on-premises Exchange server (e.g., by forwarding email through Exchange Online or using a third-party message hygiene solution), or disabling malware scans on affected Exchange servers and clearing transport queues. However, Microsoft writes that one of these workarounds should only be used if administrators are using a different email malware scanner than the engine in Exchange Server. For more information on disabling or bypassing malware scans, see the following articles:
- Antimalware protection in Exchange Server | Microsoft Docs
- Procedures for antimalware protection in Exchange Server | Microsoft Docs
Regarding the Microsoft Exchange Emergency Mitigation Service mentioned above, I read the following from the Microsoft article: Their engineers were indeed working on a solution that would eliminate the need for customer intervention. But it was determined that any change that did not require customer intervention would take several days to develop and deploy.
So at this time, they are saying, "We are currently working on another update that is in the final testing phase. The update will require customer intervention, but it will provide the fastest solution. So it's wait and see when this solution will come.
Microsoft's fix
Addendum: The Microsoft Techcommunity article Email Stuck in Transport Queues hast been updated. Microsoft Providers a Script and instructions to fix the issue.D etails May bei read within the article.
Similar articles:
Security updates for Exchange Server (July 2021)
Cumulative Exchange CUs June 2021 released
Exchange Server Security Update KB5001779 (April 13, 2021)
Exchange isues with ECP/OWA search after installing security update (March 2021)
Exchange security updates from July 2021 breaks ECP and OWA
Exchange 2016/2019: Outlook problems due to AMSI integration
Wave of attacks, almost 2,000 Exchange servers hacked via ProxyShell
Exchange Server 2016-2019: Custom attributes in ECP no longer updatable after CU installation (July 2021)
Exchange Server: Authentication bypass with ProxyToken
Exchange vulnerabilities: Will we see Hafnium II?
Exchange 2016/2019: Outlook problems due to AMSI integration
Exchange Server September 2021 CU comes Sept. 28 with Microsoft Exchange Emergency Mitigation Service
Exchange Server September 2021 CU (2021/09/28)
Security updates for Exchange Server (October 2021)
Tianfu Cup 2021: Exchange 2019 and iPhone hacked
Babuk gang uses ProxyShell vulnerability in Exchange for ransomware attacks
Exchange Server November 2021 Security Updates Close RCE Vulnerability CVE-2021-423
CERT warning: Compromised Exchange servers are misused for email attacks (Nov. 2021)
CERT-Federation, USA, GB warns about attacks on Exchange and Fortinet
ProxyNoShell: Mandiant warns of new attack methods on Exchange servers (Nov. 2021)
ProxyShell, Squirrelwaffle and a new PoC Exploit, patch your Exchange Server!
Examples of virus mails from a compromised Exchange server
German CERT-Bund warns about vulnerable Exchange Server with OWA reachable from Internet
Advertising
