{"id":31936,"date":"2023-10-12T00:45:21","date_gmt":"2023-10-11T22:45:21","guid":{"rendered":"http:\/\/159.69.82.204\/win\/?p=31936"},"modified":"2023-10-12T01:13:11","modified_gmt":"2023-10-11T23:13:11","slug":"exchange-online-incident-ex680695-nov-11-2023","status":"publish","type":"post","link":"https:\/\/borncity.com\/win\/2023\/10\/12\/exchange-online-incident-ex680695-nov-11-2023\/","title":{"rendered":"Exchange Online: Incident EX680695 (Oct. 11, 2023)"},"content":{"rendered":"<p><img decoding=\"async\" style=\"float: left; margin: 0px 10px 0px 0px; display: inline;\" title=\"Stop - Pixabay\" src=\"https:\/\/www.borncity.com\/blog\/wp-content\/uploads\/2021\/06\/Stop01.jpg\" alt=\"Stop - Pixabay\" align=\"left\" \/>As of October 11, 2023, Exchange Online has experienced a severe mail traffic disruption. Several readers reported problems receiving emails from external email addresses. Microsoft has confirmed the incident and claims to have already rolled out a fix. Spoiler: They fell victim to their own SPAM protection, which they rolled out on patchday.<\/p>\n<p><!--more--><\/p>\n<p>I was immediately informed by several users today regarding the glitch (thanks for that), but can only react now, as I was on the road. Christoph Z. wrote for example:<\/p>\n<blockquote><p>Problems with M365 Exchange Online<\/p>\n<p>Hello,<\/p>\n<p>Our customers with M365 Exchange Online are experiencing increased problems receiving emails from external email addresses today. Apparently there is a fault on the part of Microsoft.<\/p>\n<p>\"Microsoft is reporting a service issue that is causing delivery delays for email coming from outside our organization. Microsoft has identified the cause of the issue and is in the process of deploying a fix. \"<\/p>\n<p>Perhaps your readers are also having problems with Exchange Online.<\/p><\/blockquote>\n<p>Other readers have had these problems as well. Steffen H. got in touch and wrote about it:<\/p>\n<blockquote><p>Hello Mr. Born,<\/p>\n<p>I would just like to inform you about current problems with Microsoft's Exchange Online, if not already known.<\/p>\n<p>Currently mails which are sent to Exchange Online are not accepted by MS.<\/p>\n<p>These remain in the email queue of our NoSpamProxy.<\/p>\n<p>Reason: \"451 4.7.500 Server busy\".<\/p>\n<p>There is also a corresponding entry\/incident for this in the Admin Center.<\/p><\/blockquote>\n<p><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/www.borncity.com\/blog\/wp-content\/uploads\/2023\/10\/vQgY3Av.jpg\" width=\"598\" height=\"1174\" \/><\/p>\n<p>Thomas W. sent the following info by mail:<\/p>\n<blockquote><p>since this morning there are significant problems with Exchange Online.<\/p>\n<p>On our MTAs hundreds of emails to recipients are hanging on Exchange Online.<\/p>\n<p>Microsoft confirmed it by phone, but no official communication&#8230;.<\/p><\/blockquote>\n<p>He referred to a <a href=\"https:\/\/www.reddit.com\/r\/sysadmin\/comments\/1755npg\/anyone_else_seeing_outlook_mail_delivery_problems\/?rdt=60408\" target=\"_blank\" rel=\"noopener\">reddit.com post<\/a>, where corresponding can also be read. Thomas Z. wrote me about it:<\/p>\n<blockquote><p>Hello Mr. Born,<\/p>\n<p>on our mx4.tz-com.at (78.47.134.189) I have since today 5:29 am &gt; 400 emails hanging which should go to M365. Here our customers send away all emails via relay. Our server is not on any blacklist.<\/p>\n<p>All messages according to log pretty much the same<br \/>\n&#8212;&#8212;- cut &#8212;&#8212;-<br \/>\n451 4.7.500 Server busy. Please try again later from [x.x.x.x]. (S77719) [DBAEUR03FT016.eop-EUR03.prod.protection.outlook.com 2023-10-11T13:37:42.355Z 08DBC945E7F99585]<br \/>\n&lt;xxx@xxx.com&gt;&#8230; Deferred: 451 4.7.500 Server busy. Please try again later from [x.x.x.x]. (S77719) [DBAEUR03FT016.eop-EUR03.prod.protection.outlook.com 2023-10-11T13:37:42.355Z 08DBC945E7F99585]<br \/>\nClosing connection to xxxx.mail.protection.outlook.com.<\/p><\/blockquote>\n<p>Thomas referred to <a href=\"https:\/\/learn.microsoft.com\/en-us\/answers\/questions\/1388575\/e-mail-sending-to-offcie-microsoft365-customers-no\" target=\"_blank\" rel=\"noopener\">this post<\/a> at Microsoft, where an administrator also reported the problem. Andreas P. emailed me the relevant Microsoft excerpts in the admin status area, as always:<\/p>\n<blockquote><p>Published Time: 11.10.2023 19:11:15<br \/>\nWe've completed development of an additional fix which is undergoing validation prior to deployment. We expect to complete this process within the next 60 minutes.<br \/>\nThis quick update is designed to give the latest information on this issue.<\/p>\n<p>Published Time: 11.10.2023 18:27:35<br \/>\nTitle: Some users may encounter delays receiving external email messages in Exchange Online<br \/>\nUser impact: Users may encounter delays receiving external email messages in Exchange Online.<br \/>\nMore info: Impact is specific to Simple Mail Transfer Protocol (SMTP), and users may encounter delays of up to 24 hours. Additionally, the email sender may receive a Non-Delivery Report (NDR) with a \"451 4.7.500 Server busy\" error message, and their attempt to retry delivery of the email may be delivered up to 24 hours later.<br \/>\nCurrent status: We've determined that our previous developed fix to resolve the throttling issue has proven to be unsuccessful. We're developing a new fix that will undergo internal validation to ensure it resolves the underlying issue before deploying it to the impacted environments.<\/p>\n<p>Scope of impact: Impact is specific to some users who are served through the affected infrastructure.<br \/>\nStart time: Monday, October 9, 2023, at 9:00 PM UTC<br \/>\nRoot cause: A recent service update, applied to a section of infrastructure responsible for enforcing IP address anti-spam rules, contains a change which is causing impact.<br \/>\nNext update by: Wednesday, October 11, 2023, at 6:30 PM UTC<\/p>\n<p>Published Time: 11.10.2023 12:26:01<br \/>\nTitle: Some users may encounter delays receiving external email messages in Exchange Online<br \/>\nUser impact: Users may encounter delays receiving external email messages in Exchange Online.<br \/>\nMore info: Affected users may see a \"451 4.7.500 Server busy\" error message.<br \/>\nCurrent status: We've identified that specific IP addresses are being unexpectedly limited by our anti-spam procedures, causing inbound external email delivery to become throttled and delayed. We're reviewing if there have been any recent changes to our anti-spam rules to understand why the IP addresses are being limited. In the meantime, we're manually adding reported affected IP addresses to an allowed list to provide immediate relief.<br \/>\nScope of impact: Impact is specific to some users who are served through the affected infrastructure.<br \/>\nNext update by: Wednesday, October 11, 2023, at 12:30 PM UTC<\/p>\n<p>Published Time: 11.10.2023 10:59:51<br \/>\nTitle: Some users may encounter delays receiving external email messages in Exchange Online<br \/>\nUser impact: Users may encounter delays receiving external email messages in Exchange Online.<br \/>\nMore info: Affected users may see a \"451 4.7.500 Server busy\" error message.<br \/>\nCurrent status: We're analyzing sample throttling IPs from simple messages to confirm whether the issue with the portion of SQL infrastructure is causing impact, before we begin formulating a remediation plan.<br \/>\nScope of impact: Impact is specific to some users who are served through the affected infrastructure.<br \/>\nNext update by: Wednesday, October 11, 2023, at 11:00 AM UTC<\/p>\n<p>Published Time: 11.10.2023 08:33:49<br \/>\nTitle: Users are unable to receive external email messages within Exchange Online<br \/>\nUser impact: Users are unable to receive external email messages within Exchange Online.<br \/>\nMore info: Some users email messages may send; however, users may experience delays up to one hour.<br \/>\nCurrent status: Our review of simple messages has lead us to a potential root cause where a portion of SQL infrastructure may be receiving an unusually large number of requests that may be hindering Exchange Online's performance. We're working to confirm our findings to cultivate further troubleshooting steps.<br \/>\nScope of impact: Your organization is impacted by this event, as well as users who are attempting to receive external email messages within Exchange Online.<br \/>\nNext update by: Wednesday, October 11, 2023, at 8:30 AM UTC<\/p>\n<p>Published Time: 11.10.2023 06:52:50<br \/>\nTitle: Users are unable to receive external email messages within Exchange Online<br \/>\nUser impact: Users are unable to receive external email messages within Exchange Online.<br \/>\nCurrent status: We're currently reviewing simple messages to refine our understanding of this specific impact scenario and establish our troubleshooting steps.<br \/>\nScope of impact: Your organization is impacted by this event, as well as users who are attempting to receive external email messages within Exchange Online.<br \/>\nNext update by: Wednesday, October 11, 2023, at 6:30 AM UTC<\/p>\n<p>Published Time: 11.10.2023 06:22:26<br \/>\nTitle: Users are unable to receive external email messages within Exchange Online<br \/>\nUser impact: Users are unable to receive external email messages within Exchange Online.<br \/>\nCurrent status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.<\/p><\/blockquote>\n<p>Microsoft has fallen victim to its anti-spam update. A recent service update for a part of the infrastructure responsible for enforcing anti-spam rules for IP addresses contained a change that caused this impact. The problem should now be fixed.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>As of October 11, 2023, Exchange Online has experienced a severe mail traffic disruption. Several readers reported problems receiving emails from external email addresses. Microsoft has confirmed the incident and claims to have already rolled out a fix. Spoiler: They &hellip; <a href=\"https:\/\/borncity.com\/win\/2023\/10\/12\/exchange-online-incident-ex680695-nov-11-2023\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[63,463],"tags":[2260],"class_list":["post-31936","post","type-post","status-publish","format-standard","hentry","category-cloud","category-issue","tag-exchange-online"],"_links":{"self":[{"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/posts\/31936","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/comments?post=31936"}],"version-history":[{"count":0,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/posts\/31936\/revisions"}],"wp:attachment":[{"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/media?parent=31936"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/categories?post=31936"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/tags?post=31936"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}