Exchange Online: Incident EX680695 (Oct. 11, 2023)

Stop - PixabayAs of October 11, 2023, Exchange Online has experienced a severe mail traffic disruption. Several readers reported problems receiving emails from external email addresses. Microsoft has confirmed the incident and claims to have already rolled out a fix. Spoiler: They fell victim to their own SPAM protection, which they rolled out on patchday.


Advertising

I was immediately informed by several users today regarding the glitch (thanks for that), but can only react now, as I was on the road. Christoph Z. wrote for example:

Problems with M365 Exchange Online

Hello,

Our customers with M365 Exchange Online are experiencing increased problems receiving emails from external email addresses today. Apparently there is a fault on the part of Microsoft.

"Microsoft is reporting a service issue that is causing delivery delays for email coming from outside our organization. Microsoft has identified the cause of the issue and is in the process of deploying a fix. "

Perhaps your readers are also having problems with Exchange Online.

Other readers have had these problems as well. Steffen H. got in touch and wrote about it:

Hello Mr. Born,

I would just like to inform you about current problems with Microsoft's Exchange Online, if not already known.

Currently mails which are sent to Exchange Online are not accepted by MS.

These remain in the email queue of our NoSpamProxy.

Reason: "451 4.7.500 Server busy".

There is also a corresponding entry/incident for this in the Admin Center.

Thomas W. sent the following info by mail:

since this morning there are significant problems with Exchange Online.

On our MTAs hundreds of emails to recipients are hanging on Exchange Online.

Microsoft confirmed it by phone, but no official communication….

He referred to a reddit.com post, where corresponding can also be read. Thomas Z. wrote me about it:


Advertising

Hello Mr. Born,

on our mx4.tz-com.at (78.47.134.189) I have since today 5:29 am > 400 emails hanging which should go to M365. Here our customers send away all emails via relay. Our server is not on any blacklist.

All messages according to log pretty much the same
——- cut ——-
451 4.7.500 Server busy. Please try again later from [x.x.x.x]. (S77719) [DBAEUR03FT016.eop-EUR03.prod.protection.outlook.com 2023-10-11T13:37:42.355Z 08DBC945E7F99585]
<xxx@xxx.com>… Deferred: 451 4.7.500 Server busy. Please try again later from [x.x.x.x]. (S77719) [DBAEUR03FT016.eop-EUR03.prod.protection.outlook.com 2023-10-11T13:37:42.355Z 08DBC945E7F99585]
Closing connection to xxxx.mail.protection.outlook.com.

Thomas referred to this post at Microsoft, where an administrator also reported the problem. Andreas P. emailed me the relevant Microsoft excerpts in the admin status area, as always:

Published Time: 11.10.2023 19:11:15
We've completed development of an additional fix which is undergoing validation prior to deployment. We expect to complete this process within the next 60 minutes.
This quick update is designed to give the latest information on this issue.

Published Time: 11.10.2023 18:27:35
Title: Some users may encounter delays receiving external email messages in Exchange Online
User impact: Users may encounter delays receiving external email messages in Exchange Online.
More info: Impact is specific to Simple Mail Transfer Protocol (SMTP), and users may encounter delays of up to 24 hours. Additionally, the email sender may receive a Non-Delivery Report (NDR) with a "451 4.7.500 Server busy" error message, and their attempt to retry delivery of the email may be delivered up to 24 hours later.
Current status: We've determined that our previous developed fix to resolve the throttling issue has proven to be unsuccessful. We're developing a new fix that will undergo internal validation to ensure it resolves the underlying issue before deploying it to the impacted environments.

Scope of impact: Impact is specific to some users who are served through the affected infrastructure.
Start time: Monday, October 9, 2023, at 9:00 PM UTC
Root cause: A recent service update, applied to a section of infrastructure responsible for enforcing IP address anti-spam rules, contains a change which is causing impact.
Next update by: Wednesday, October 11, 2023, at 6:30 PM UTC

Published Time: 11.10.2023 12:26:01
Title: Some users may encounter delays receiving external email messages in Exchange Online
User impact: Users may encounter delays receiving external email messages in Exchange Online.
More info: Affected users may see a "451 4.7.500 Server busy" error message.
Current status: We've identified that specific IP addresses are being unexpectedly limited by our anti-spam procedures, causing inbound external email delivery to become throttled and delayed. We're reviewing if there have been any recent changes to our anti-spam rules to understand why the IP addresses are being limited. In the meantime, we're manually adding reported affected IP addresses to an allowed list to provide immediate relief.
Scope of impact: Impact is specific to some users who are served through the affected infrastructure.
Next update by: Wednesday, October 11, 2023, at 12:30 PM UTC

Published Time: 11.10.2023 10:59:51
Title: Some users may encounter delays receiving external email messages in Exchange Online
User impact: Users may encounter delays receiving external email messages in Exchange Online.
More info: Affected users may see a "451 4.7.500 Server busy" error message.
Current status: We're analyzing sample throttling IPs from simple messages to confirm whether the issue with the portion of SQL infrastructure is causing impact, before we begin formulating a remediation plan.
Scope of impact: Impact is specific to some users who are served through the affected infrastructure.
Next update by: Wednesday, October 11, 2023, at 11:00 AM UTC

Published Time: 11.10.2023 08:33:49
Title: Users are unable to receive external email messages within Exchange Online
User impact: Users are unable to receive external email messages within Exchange Online.
More info: Some users email messages may send; however, users may experience delays up to one hour.
Current status: Our review of simple messages has lead us to a potential root cause where a portion of SQL infrastructure may be receiving an unusually large number of requests that may be hindering Exchange Online's performance. We're working to confirm our findings to cultivate further troubleshooting steps.
Scope of impact: Your organization is impacted by this event, as well as users who are attempting to receive external email messages within Exchange Online.
Next update by: Wednesday, October 11, 2023, at 8:30 AM UTC

Published Time: 11.10.2023 06:52:50
Title: Users are unable to receive external email messages within Exchange Online
User impact: Users are unable to receive external email messages within Exchange Online.
Current status: We're currently reviewing simple messages to refine our understanding of this specific impact scenario and establish our troubleshooting steps.
Scope of impact: Your organization is impacted by this event, as well as users who are attempting to receive external email messages within Exchange Online.
Next update by: Wednesday, October 11, 2023, at 6:30 AM UTC

Published Time: 11.10.2023 06:22:26
Title: Users are unable to receive external email messages within Exchange Online
User impact: Users are unable to receive external email messages within Exchange Online.
Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.

Microsoft has fallen victim to its anti-spam update. A recent service update for a part of the infrastructure responsible for enforcing anti-spam rules for IP addresses contained a change that caused this impact. The problem should now be fixed.


Advertising

This entry was posted in Cloud, issue and tagged . Bookmark the permalink.

One Response to Exchange Online: Incident EX680695 (Oct. 11, 2023)

  1. Mike says:

    Worked with Apple yesterday, Microsoft today and finally got someone to tell me there was an Incident still open….

    We have received reports on this issue from few affected users and after investigating this issue at the backend we found that impact is specific to some users, who are served through the affected infrastructure, with the email client apps that utilize the Exchange Web Services (EWS) API. So, this is affecting Legacy Outlook as it uses Exchange Web Services.

    There's an Incident (EX698071) already created for this issue by O365 service team, and your IT admin can check the status here by signing in-to the admin account.

    https://portal.office.com/ServiceStatus

    The temporary workaround with sync issue in Classic Outlook, is to switch to New Outlook using the toggle at the top right. Hope this information helps. Let us know if you have any further questions. Thank you.

    Horrible support model as they had me delete Outlook, reload outlook, reboot computer, clear cache, only to end up with the same results and then FINALLY notified me of the above. Microsoft has gotten too big to support real user issues and when I asked about why the incident focuses on third party apps when OUTLOOK isn't third party, they told me they could only discuss technical issues and to call Customer Service which is another debacle.

Leave a Reply

Your email address will not be published. Required fields are marked *

Note: Please note the rules for commenting on the blog (first comments and linked posts end up in moderation, I release them every few hours, I rigorously delete SEO posts/SPAM).