Exchange Server 2013 Mail issues after Feb. 2020 Update

[German]A German blog reader contacted me yesterday and mentioned a serious issue he had with an Exchange Server 2013 after installing the latest February 2020 updates. After update installation Mails were accepted, but no longer sorted into the users' mailboxes. I'll post the findings here, maybe it helps other administrators.

Blog reader Andre H. shared his update experiences with the Exchange Server 2013 (CU 23) running on Windows Server 2012 R2 on Friday (14.2.2020) and suddenly complained.

Yesterday evening we installed the latest Windows updates on one of our Exchange 2013 CU23 servers on Windows Server 2012R2 – and afterwards we found issues which we want to mention in your blog.

Andre had the following updates installed in his environment on the server [this must have been last Thursday, 2/13/22020]:

KB4502496, KB4536988, KB890830, KB4537767, KB4538124, KB4484156, KB4484265, KB4537821, KB4537803, KB4537759

This is nothing unusual and after the installation of the updates, the restart and the reconnection of the first server (server in the DAG group) everything apparently went well, Andre writes. But then the problems started.

This morning we noticed that various mails, both external and internal, did not end up in the users' mailboxes – but the server accepted them.

After some checks in the event log we could not find a more exact cause. So we pulled the server out as an active member out of the DAG group and out of the upstream LoadBalancer to investigate in more details.

We contacted the Microsoft support (with a response time of 2 hours; after about 20 minutes we got a call back). The MS colleague briefly checked the environment, created a new database and 2 test users, then checked the mail flow via OWA – result: no mail flow.

In the MessageTrackingLog you could see that the mails were stuck in the shadow queue, and that this did not continue.

A service check did not show any problems either.

The transport config looked good too.

Arriving at the IIS Manager, we checked the binding settings of the two published websites; in the default website an entry was missing here: Despite an existing entry with the type https, port 443 and ip address "*" as well as the appropriate certificate, no connection from the other servers to this server could be established.

According to Microsoft another entry was missing here: Type https, port 443, as IP address explicitly 127.0.0.1 with the appropriate certificate. The Microsoft supporter could not explain why the existing "*" entry does not apply here. The solution: 

Add the missing entries, "iisreset" and wait.

Andre wrote also within his mail: This makes a lot of sense, none of us changed a similar setting in the IIS Config.

The previously "lost" mails came in after 10-15min., the Exchange Queue Viewer emptied itself, the users (and us) were happy.

Regarding the problem itself, Andre says that from Microsoft's (and his) point of view, the following updates (or one of them) most likely changed something (unintentionally?):

KB4536988 Security Update for Exchange Server 2013 CU23
KB4502496 Security Update for Windows Server 2012 R2 x64 2020-02
KB4538124 Security and Quality Rollup for .NET Framework 3.5… 2020-02

Andre states that they could find events in the event log that the IIS services as well as AppPools were stopped several times during the installation of the updates. Nothing more detailed can be said about this and for the MS Support the call was closed after adding the new entry and a successful test.

Andre notes that Microsoft Support does not perform root cause analysis – MS Premier Support would be available for this. You can only get this support on the phone, Andre, if you sign an expensive Premier Support contract of at least 1 year and pay an additional 300€ per hour for the support. And as weekend prayer he adds:

This leaves us 2 more servers where we can install the updates and see if these problems occur again.

In the world-wide web we have not found nothing similar, it's perhaps because of the quite new updates.

and concludes this with the words: We now wish everyone a continued trouble-free running environment and you, dear Mr. Born, continued strength and success and a nice weekend. There is nothing to add to this – except that I might need a cap of sleep and I will take it now. At this point I would like to thank Andre for his advice. Hope it helps administrator affected. And if you get into this mess, drop a comment.