{"id":283887,"date":"2023-07-18T12:18:39","date_gmt":"2023-07-18T10:18:39","guid":{"rendered":"https:\/\/www.borncity.com\/blog\/?p=283887"},"modified":"2023-07-19T11:36:03","modified_gmt":"2023-07-19T09:36:03","slug":"exchange-online-blockt-e-mails-18-7-2023","status":"publish","type":"post","link":"https:\/\/borncity.com\/blog\/2023\/07\/18\/exchange-online-blockt-e-mails-18-7-2023\/","title":{"rendered":"Exchange Online blockt E-Mails (18.7.2023)"},"content":{"rendered":"

\"Stop[English<\/a>]Am heutigen 18. Juli 2023 scheint seit seit dem Morgen eine St\u00f6rung in Microsofts Cloud-Dienst Exchange Online zu geben. Blog-Leser berichten, dass E-Mails blockiert w\u00fcrden. Auf St\u00f6rungsseiten finden sich verschiedene Hinweise auf Probleme und auch das Sozial Media Team von Microsoft schreibt auf Twitter, dass man Berichten \u00fcber St\u00f6rungen nachgehe. Hier ein kurzer \u00dcberblick.<\/p>\n

<\/p>\n

\"\"Die erste Information erreichte mich von Blog-Leser Tom801, der im Diskussionsbereich einen Kommentar hinterlie\u00df, den ich hier mal herausziehe:<\/p>\n

Bei Exchange Online rattert es mal wieder EX649175 \"503 5.5.1 Bad sequence of commands\" vor allem scheint es DACH und UK zu betreffen.<\/p><\/blockquote>\n

Auf Twitter meldet @MSFT365Status ebenfalls, dass man Meldungen zu einer St\u00f6rung nachgehe. Ich ziehe die betreffenden Tweets mal als Text heraus:<\/p>\n

We're investigating an issue where some users may be unable to send Exchange Online email messages due to a recent change to the free\/busy infrastructure. We're identifying a plan to resolve the impact. Further details can be found under EX649175 in the admin center.<\/p><\/blockquote>\n

Eine Information, dass die St\u00f6rung behoben sei, wird von Nutzern auf Twitter aktuell noch negiert. Eine Ursache ist aktuell unbekannt. Ich selbst kann mangels Microsoft 365-Konto nicht auf die Statusmeldung EX649175 im Administratorbereich zugreifen. Die Office365 Statusseite<\/a> meldet keine St\u00f6rung. Auf dieser Seite<\/a> hei\u00dft es aber zur St\u00f6rung EX649175:<\/p>\n

\n

Microsoft 365 Alert \u2013 Service Degradation \u2013 Exchange Online \u2013 Some users may be unable to send Exchange Online email messages \u2013 ONGOING<\/h3>\n

18\/07\/2023 08:32:00 AM (GMT)<\/p>\n

NHSmail Reference : INC37075572<\/strong><\/p>\n

Microsoft Reference : EX649175<\/strong><\/p>\n

Issue Status : ONGOING<\/strong><\/p>\n

Issue Description :<\/strong> Users may be unable to send Exchange Online email messages.<\/p>\n

More info :<\/strong> Affected users may receive the following error \"503 5.5.1 bad sequence of commands\"<\/p>\n

Current Update : 18\/07\/2023 10:22:00 AM \u2013<\/strong> Microsoft disabled an interceptor rule that we believe was causing impact to the mail flow. They are monitoring the service to ensure the mail flow is restored.<\/p>\n

Scope of impact :<\/strong> Impact is specific to some users who are served through the affected infrastructure.<\/p>\n

Next update by :<\/strong> Tuesday, July 18, 2023, at 10:00 AM UTC<\/p><\/blockquote>\n

Die St\u00f6rung scheint aber weltweit Nutzer zu tangieren – bei einer schnellen Recherche habe ich gesehen, dass die Kollegen von Bleeping Computer hier<\/a> das Thema ebenfalls aufgreifen. Auf Seiten wie allestoerungen.de gibt es ab ca. 8:00 Uhr deutscher Zeit einen Anstieg der St\u00f6rungsmeldungen, wobei das aber mit einem Peak von 25 Meldungen pro Zeiteinheit minimal ist. Jemand von euch betroffen?<\/p>\n

Erg\u00e4nzung: In den geschlossenen Facebook-Gruppen bekomme ich gemischtes Feedback, von \"nichts gemerkt\", bis hin \"nix geht mehr\". Hier einige anonymisierte Meldungen:<\/p>\n

User #1:<\/strong> Ich war betroffen. Wir nutzen ZScaler und ZCC. Seit Montag verbindet sich OneDrive nicht mehr \u00fcber ZScaler. Also SSO funktioniert nicht. Hat das sonst noch jemand im Einsatz und stellt das Problem fest?<\/p>\n

User #2:\u00a0<\/strong>Jep, wir sind betroffen. Aber nur zu bestimmten Mailadressen: Remote server returned '503 5.5.1 Bad sequence of commands ###<\/p>\n

User #3: <\/strong>Jep, sind selbst betroffen und auch div. Kunden, die bei MS365 liegen.<\/p><\/blockquote>\n

Die Kollegen von Bleeping Computer berichten hier<\/a> in einem Nachtrag, dass Microsoft das Problem behoben habe und den Eintrag f\u00fcr EX649175 aus dem Admin-Center gel\u00f6scht h\u00e4tten. Irgendwie eine komische Ma\u00dfnahme, wie soll ein Administrator reagieren, wenn er versp\u00e4tet Kundenbeschwerden erh\u00e4lt, dass etwas nicht geklappt hat, die St\u00f6rungseintr\u00e4ge aber verschwunden sind? Oder \u00fcbersehe ich hier was? Die nachfolgenden Kommentare lesen sich f\u00fcr mich ebenfalls, wie \"der Administrator liest und wundert sich\".<\/p>\n

Erg\u00e4nzung: St\u00f6rung vom 18. Juli 2023 – Andreas hat mir noch den folgenden Auszug aus dem Statusbereich zukommen lassen (danke daf\u00fcr)<\/p>\n

Share: Incident: EX648815, impacted service: Exchange Online, impacted feature: delivery, current status is: Service degradation<\/strong><\/p>\n

Published Time: 18.07.2023 19:32:23<\/p>\n

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online<\/p>\n

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.<\/p>\n

More info: Some users may encounter delivery delays of up to approximately five hours. The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.<\/p>\n

Current status: After further investigation, we're now working to implement a set of steps that will reroute the free\/busy CFMs to the appropriate mailboxes.<\/p>\n

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.<\/p>\n

Start time: Monday, July 17, 2023, at 3:00 PM UTC<\/p>\n

Root cause: A recent code change related to our free\/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free\/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.<\/p>\n

Next update by: Tuesday, July 18, 2023, at 7:30 PM UTC<\/p>\n

Published Time: 18.07.2023 17:37:14<\/p>\n

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online<\/p>\n

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.<\/p>\n

More info: Some users may encounter delivery delays of up to approximately five hours. The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.<\/p>\n

Current status: We've disabled the free\/busy publishing assistant in a small portion of affected infrastructure to test the efficacy of this mitigation and initial feedback indicates this hasn't had the positive effect we anticipated for reducing email delays. We're reviewing other potential options to mitigate impact while simultaneously assessing further long-term fix solutions.<\/p>\n

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.<\/p>\n

Start time: Monday, July 17, 2023, at 3:00 PM UTC<\/p>\n

Root cause: A recent code change related to our free\/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free\/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.<\/p>\n

Next update by: Tuesday, July 18, 2023, at 5:30 PM UTC<\/p>\n

Published Time: 18.07.2023 15:32:40<\/p>\n

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online<\/p>\n

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.<\/p>\n

More info: Some users may encounter delivery delays of up to approximately four hours. The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.<\/p>\n

Current status: We've deployed the revised fix but identified that this has only improved delays for some users and hasn't provided the expected relief for all users. We're assessing options of disabling the free\/busy publishing assistant to prevent CFMs from being generated and causing email delivery delays. This will restore email delays but users may still see outdated free\/busy information in Exchange Online. After disabling the assistant, we'll deploy a longer term fix to enable the assistant without causing an excessive load of CFMs.<\/p>\n

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.<\/p>\n

Start time: Monday, July 17, 2023, at 3:00 PM UTC<\/p>\n

Root cause: A recent code change related to our free\/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free\/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.<\/p>\n

Next update by: Tuesday, July 18, 2023, at 3:30 PM UTC<\/p>\n

Published Time: 18.07.2023 13:19:28<\/p>\n

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online<\/p>\n

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.<\/p>\n

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.<\/p>\n

Current status: We've identified that the fix had an unintended impact and resulted in mail flow issues, generating NDRs for some users. We've reverted the fix that caused impact to mail flow and are now deploying a revised fix to effectively discard the excessive CFMs that are causing email delivery delays.<\/p>\n

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.<\/p>\n

Start time: Monday, July 17, 2023, at 3:00 PM UTC<\/p>\n

Root cause: A recent code change related to our free\/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free\/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.<\/p>\n

Next update by: Tuesday, July 18, 2023, at 1:30 PM UTC<\/p>\n

Published Time: 18.07.2023 08:00:07<\/p>\n

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online<\/p>\n

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.<\/p>\n

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.<\/p>\n

Current status: We implemented an additional fix to prevent the Control Flow Messages (CFMs) that are impacting email delivery. We're continuing to monitor the environment to ensure that the delays do not reoccur. It is possible that during this time, users may see stale free\/busy information.<\/p>\n

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.<\/p>\n

Start time: Monday, July 17, 2023, at 3:00 PM UTC<\/p>\n

Root cause: A recent code change related to our free\/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free\/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.<\/p>\n

Next update by: Tuesday, July 18, 2023, at 8:30 PM UTC<\/p>\n

Published Time: 18.07.2023 06:32:58<\/p>\n

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online<\/p>\n

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.<\/p>\n

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.<\/p>\n

Current status: During our review of the system logs, we identified that Control Flow Messages (CFMs) are still sending at an increased rate. We're looking into options to prevent the additional CFMs in the affected portion of infrastructure, which we expect will mitigate the email delivery delays.<\/p>\n

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.<\/p>\n

Start time: Monday, July 17, 2023, at 3:00 PM UTC<\/p>\n

Root cause: A recent code change related to our free\/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free\/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.<\/p>\n

Next update by: Tuesday, July 18, 2023, at 6:30 AM UTC<\/p>\n

Published Time: 18.07.2023 04:29:59<\/p>\n

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online<\/p>\n

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.<\/p>\n

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.<\/p>\n

Current status: We're continuing to monitor the environment to verify that messages are delivering successfully. We're still seeing some issues with messages queueing for delivery on a portion of infrastructure. We're reviewing system logs to see if there are any message caches that are trying to resend messages.<\/p>\n

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.<\/p>\n

Start time: Monday, July 17, 2023, at 3:00 PM UTC<\/p>\n

Root cause: A recent code change related to our free\/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free\/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.<\/p>\n

Next update by: Tuesday, July 18, 2023, at 4:30 AM UTC<\/p>\n

Published Time: 18.07.2023 02:18:38<\/p>\n

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online<\/p>\n

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.<\/p>\n

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.<\/p>\n

Current status: We're continuing to monitor to ensure messages deliver as expected, and for the majority of users impact has been resolved. There's a small portion of infrastructure still experiencing a queuing issue which we're examining further to understand why it's not seeing relief where others are and determine steps to ensure all messages are successfully delivered.<\/p>\n

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.<\/p>\n

Start time: Monday, July 17, 2023, at 3:00 PM UTC<\/p>\n

Root cause: A recent code change related to our free\/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free\/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.<\/p>\n

Next update by: Tuesday, July 18, 2023, at 2:30 AM UTC<\/p>\n

Published Time: 18.07.2023 00:13:33<\/p>\n

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online<\/p>\n

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.<\/p>\n

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.<\/p>\n

Current status: We've completed our deployment of the fix and the reversion process has completed. We're monitoring mail queues to ensure they deliver as expected, resolving impact.<\/p>\n

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.<\/p>\n

Start time: Monday, July 17, 2023, at 3:00 PM UTC<\/p>\n

Root cause: A recent code change related to our free\/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free\/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.<\/p>\n

Next update by: Tuesday, July 18, 2023, at 12:30 AM UTC<\/p>\n

Published Time: 17.07.2023 22:07:09<\/p>\n

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online<\/p>\n

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.<\/p>\n

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.<\/p>\n

Current status: We've identified a recent code change related to our free\/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free\/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure which is resulting in message queuing. We're working to disable this change so no further mail becomes queued and determine if additional steps are needed to drain the current backlogged messages.<\/p>\n

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.<\/p>\n

Start time: Monday, July 17, 2023, at 3:00 PM UTC<\/p>\n

Root cause: A recent code change related to our free\/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free\/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.<\/p>\n

Next update by: Monday, July 17, 2023, at 10:30 PM UTC<\/p>\n

Published Time: 17.07.2023 21:27:00<\/p>\n

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online<\/p>\n

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.<\/p>\n

More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.<\/p>\n

Current status: We're reviewing service health telemetry information to identify possible sources of impact where we see mail queueing occurring to help formulate our next troubleshooting steps.<\/p>\n

Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.<\/p>\n

Next update by: Monday, July 17, 2023, at 8:30 PM UTC<\/p>\n

Published Time: 17.07.2023 21:03:33<\/p>\n

Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online<\/p>\n

User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.<\/p>\n

More info: The issue isn't client-specific, thus may occur in all Exchange Online connection methods.<\/p>\n

Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.<\/p>\n","protected":false},"excerpt":{"rendered":"

[English]Am heutigen 18. Juli 2023 scheint seit seit dem Morgen eine St\u00f6rung in Microsofts Cloud-Dienst Exchange Online zu geben. Blog-Leser berichten, dass E-Mails blockiert w\u00fcrden. Auf St\u00f6rungsseiten finden sich verschiedene Hinweise auf Probleme und auch das Sozial Media Team von … Weiterlesen →<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[7263,7862],"tags":[1171,7801,987],"class_list":["post-283887","post","type-post","status-publish","format-standard","hentry","category-cloud","category-stoerung","tag-cloud","tag-echange-online","tag-storung"],"_links":{"self":[{"href":"https:\/\/borncity.com\/blog\/wp-json\/wp\/v2\/posts\/283887","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/borncity.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/borncity.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/borncity.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/borncity.com\/blog\/wp-json\/wp\/v2\/comments?post=283887"}],"version-history":[{"count":0,"href":"https:\/\/borncity.com\/blog\/wp-json\/wp\/v2\/posts\/283887\/revisions"}],"wp:attachment":[{"href":"https:\/\/borncity.com\/blog\/wp-json\/wp\/v2\/media?parent=283887"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/borncity.com\/blog\/wp-json\/wp\/v2\/categories?post=283887"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/borncity.com\/blog\/wp-json\/wp\/v2\/tags?post=283887"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}