[English]Am heutigen 18. Juli 2023 scheint seit seit dem Morgen eine Störung in Microsofts Cloud-Dienst Exchange Online zu geben. Blog-Leser berichten, dass E-Mails blockiert würden. Auf Störungsseiten finden sich verschiedene Hinweise auf Probleme und auch das Sozial Media Team von Microsoft schreibt auf Twitter, dass man Berichten über Störungen nachgehe. Hier ein kurzer Überblick.
Anzeige
Die erste Information erreichte mich von Blog-Leser Tom801, der im Diskussionsbereich einen Kommentar hinterließ, den ich hier mal herausziehe:
Bei Exchange Online rattert es mal wieder EX649175 "503 5.5.1 Bad sequence of commands" vor allem scheint es DACH und UK zu betreffen.
Auf Twitter meldet @MSFT365Status ebenfalls, dass man Meldungen zu einer Störung nachgehe. Ich ziehe die betreffenden Tweets mal als Text heraus:
We're investigating an issue where some users may be unable to send Exchange Online email messages due to a recent change to the free/busy infrastructure. We're identifying a plan to resolve the impact. Further details can be found under EX649175 in the admin center.
Eine Information, dass die Störung behoben sei, wird von Nutzern auf Twitter aktuell noch negiert. Eine Ursache ist aktuell unbekannt. Ich selbst kann mangels Microsoft 365-Konto nicht auf die Statusmeldung EX649175 im Administratorbereich zugreifen. Die Office365 Statusseite meldet keine Störung. Auf dieser Seite heißt es aber zur Störung EX649175:
Microsoft 365 Alert – Service Degradation – Exchange Online – Some users may be unable to send Exchange Online email messages – ONGOING
18/07/2023 08:32:00 AM (GMT)
NHSmail Reference : INC37075572
Microsoft Reference : EX649175
Issue Status : ONGOING
Issue Description : Users may be unable to send Exchange Online email messages.
More info : Affected users may receive the following error "503 5.5.1 bad sequence of commands"
Current Update : 18/07/2023 10:22:00 AM – Microsoft disabled an interceptor rule that we believe was causing impact to the mail flow. They are monitoring the service to ensure the mail flow is restored.
Scope of impact : Impact is specific to some users who are served through the affected infrastructure.
Next update by : Tuesday, July 18, 2023, at 10:00 AM UTC
Die Störung scheint aber weltweit Nutzer zu tangieren – bei einer schnellen Recherche habe ich gesehen, dass die Kollegen von Bleeping Computer hier das Thema ebenfalls aufgreifen. Auf Seiten wie allestoerungen.de gibt es ab ca. 8:00 Uhr deutscher Zeit einen Anstieg der Störungsmeldungen, wobei das aber mit einem Peak von 25 Meldungen pro Zeiteinheit minimal ist. Jemand von euch betroffen?
Anzeige
Ergänzung: In den geschlossenen Facebook-Gruppen bekomme ich gemischtes Feedback, von "nichts gemerkt", bis hin "nix geht mehr". Hier einige anonymisierte Meldungen:
User #1: Ich war betroffen. Wir nutzen ZScaler und ZCC. Seit Montag verbindet sich OneDrive nicht mehr über ZScaler. Also SSO funktioniert nicht. Hat das sonst noch jemand im Einsatz und stellt das Problem fest?
User #2: Jep, wir sind betroffen. Aber nur zu bestimmten Mailadressen: Remote server returned '503 5.5.1 Bad sequence of commands ###
User #3: Jep, sind selbst betroffen und auch div. Kunden, die bei MS365 liegen.
Die Kollegen von Bleeping Computer berichten hier in einem Nachtrag, dass Microsoft das Problem behoben habe und den Eintrag für EX649175 aus dem Admin-Center gelöscht hätten. Irgendwie eine komische Maßnahme, wie soll ein Administrator reagieren, wenn er verspätet Kundenbeschwerden erhält, dass etwas nicht geklappt hat, die Störungseinträge aber verschwunden sind? Oder übersehe ich hier was? Die nachfolgenden Kommentare lesen sich für mich ebenfalls, wie "der Administrator liest und wundert sich".
Ergänzung: Störung vom 18. Juli 2023 – Andreas hat mir noch den folgenden Auszug aus dem Statusbereich zukommen lassen (danke dafür)
Share: Incident: EX648815, impacted service: Exchange Online, impacted feature: delivery, current status is: Service degradation
Published Time: 18.07.2023 19:32:23
Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.
More info: Some users may encounter delivery delays of up to approximately five hours. The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.
Current status: After further investigation, we're now working to implement a set of steps that will reroute the free/busy CFMs to the appropriate mailboxes.
Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.
Start time: Monday, July 17, 2023, at 3:00 PM UTC
Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.
Next update by: Tuesday, July 18, 2023, at 7:30 PM UTC
Published Time: 18.07.2023 17:37:14
Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.
More info: Some users may encounter delivery delays of up to approximately five hours. The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.
Current status: We've disabled the free/busy publishing assistant in a small portion of affected infrastructure to test the efficacy of this mitigation and initial feedback indicates this hasn't had the positive effect we anticipated for reducing email delays. We're reviewing other potential options to mitigate impact while simultaneously assessing further long-term fix solutions.
Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.
Start time: Monday, July 17, 2023, at 3:00 PM UTC
Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.
Next update by: Tuesday, July 18, 2023, at 5:30 PM UTC
Published Time: 18.07.2023 15:32:40
Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.
More info: Some users may encounter delivery delays of up to approximately four hours. The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.
Current status: We've deployed the revised fix but identified that this has only improved delays for some users and hasn't provided the expected relief for all users. We're assessing options of disabling the free/busy publishing assistant to prevent CFMs from being generated and causing email delivery delays. This will restore email delays but users may still see outdated free/busy information in Exchange Online. After disabling the assistant, we'll deploy a longer term fix to enable the assistant without causing an excessive load of CFMs.
Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.
Start time: Monday, July 17, 2023, at 3:00 PM UTC
Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.
Next update by: Tuesday, July 18, 2023, at 3:30 PM UTC
Published Time: 18.07.2023 13:19:28
Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.
More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.
Current status: We've identified that the fix had an unintended impact and resulted in mail flow issues, generating NDRs for some users. We've reverted the fix that caused impact to mail flow and are now deploying a revised fix to effectively discard the excessive CFMs that are causing email delivery delays.
Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.
Start time: Monday, July 17, 2023, at 3:00 PM UTC
Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.
Next update by: Tuesday, July 18, 2023, at 1:30 PM UTC
Published Time: 18.07.2023 08:00:07
Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.
More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.
Current status: We implemented an additional fix to prevent the Control Flow Messages (CFMs) that are impacting email delivery. We're continuing to monitor the environment to ensure that the delays do not reoccur. It is possible that during this time, users may see stale free/busy information.
Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.
Start time: Monday, July 17, 2023, at 3:00 PM UTC
Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.
Next update by: Tuesday, July 18, 2023, at 8:30 PM UTC
Published Time: 18.07.2023 06:32:58
Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.
More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.
Current status: During our review of the system logs, we identified that Control Flow Messages (CFMs) are still sending at an increased rate. We're looking into options to prevent the additional CFMs in the affected portion of infrastructure, which we expect will mitigate the email delivery delays.
Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.
Start time: Monday, July 17, 2023, at 3:00 PM UTC
Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.
Next update by: Tuesday, July 18, 2023, at 6:30 AM UTC
Published Time: 18.07.2023 04:29:59
Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.
More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.
Current status: We're continuing to monitor the environment to verify that messages are delivering successfully. We're still seeing some issues with messages queueing for delivery on a portion of infrastructure. We're reviewing system logs to see if there are any message caches that are trying to resend messages.
Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.
Start time: Monday, July 17, 2023, at 3:00 PM UTC
Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.
Next update by: Tuesday, July 18, 2023, at 4:30 AM UTC
Published Time: 18.07.2023 02:18:38
Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.
More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.
Current status: We're continuing to monitor to ensure messages deliver as expected, and for the majority of users impact has been resolved. There's a small portion of infrastructure still experiencing a queuing issue which we're examining further to understand why it's not seeing relief where others are and determine steps to ensure all messages are successfully delivered.
Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.
Start time: Monday, July 17, 2023, at 3:00 PM UTC
Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.
Next update by: Tuesday, July 18, 2023, at 2:30 AM UTC
Published Time: 18.07.2023 00:13:33
Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.
More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.
Current status: We've completed our deployment of the fix and the reversion process has completed. We're monitoring mail queues to ensure they deliver as expected, resolving impact.
Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.
Start time: Monday, July 17, 2023, at 3:00 PM UTC
Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.
Next update by: Tuesday, July 18, 2023, at 12:30 AM UTC
Published Time: 17.07.2023 22:07:09
Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.
More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.
Current status: We've identified a recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure which is resulting in message queuing. We're working to disable this change so no further mail becomes queued and determine if additional steps are needed to drain the current backlogged messages.
Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.
Start time: Monday, July 17, 2023, at 3:00 PM UTC
Root cause: A recent code change related to our free/busy publishing APIs has resulted in an influx of Control Flow Messages (CFMs), which are sent during certain free/busy operations such as calendar replication, to be sent for other instances where CFMs are not necessary. This is resulting in a spike in messages on Exchange Online infrastructure, which is resulting in message queuing.
Next update by: Monday, July 17, 2023, at 10:30 PM UTC
Published Time: 17.07.2023 21:27:00
Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.
More info: The issue isn't client-specific, thus this impact may occur in all Exchange Online connection methods.
Current status: We're reviewing service health telemetry information to identify possible sources of impact where we see mail queueing occurring to help formulate our next troubleshooting steps.
Scope of impact: This issue may affect any user sending or receiving mail in Exchange Online.
Next update by: Monday, July 17, 2023, at 8:30 PM UTC
Published Time: 17.07.2023 21:03:33
Title: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
User impact: Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online.
More info: The issue isn't client-specific, thus may occur in all Exchange Online connection methods.
Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.
Anzeige
Zu diesem Fehler konnte ich nichts finden, allerdings gibt es seit gestern Abend den Fehler: EX648815, Users' inbound and outbound email delivery may be delayed for 15 minutes or longer in Exchange Online
Hier dürfte aber schon ein Fix implementiert sein, der aktuell beobachtet wird.
Mir selbst sind keine Probleme aufgefallen, mails funktionierten eigentlich wie gewohnt.
Dto. kein EX649175 dafür EX648815.
Jedoch keine Auswirkungen festgestellt.
Fast alle unsere Kunden waren betroffen (Schweiz). Kurz nach 12 war die Störung gem. Microsoft behoben.
Ich kann das nicht mal in unserem Admin Portal finden. Es existiert schlicht weg nicht in meinem Dienststatus.
MS hat mW die Möglichkeit die Meldungen zielreichtet auf die Tentants einzuspielen.
Ich habe die Mitteilung auch nicht.
Also vrmtl. nicht betroffen.
/servicehealth/:/alerts/EX649175
"Sie haben keine Berechtigungen".
Ist auf jedenfall bei uns im Portal nicht mehr sichtbar nirgendwo. Kein Hinweis.
Spurenbeseitigung?
Ein zu scharfer Spamfilter …
Nicht wirklich. Und es traf auch nicht alle.
"We've mitigated impact for EX649175. Please refer to the admin centre for more details. As the change was only applied to a limited portion of Exchange infrastructure, we've now scoped down our communication to only the customers who were affected by this issue"
Daher wurde es bei einigen zurück gezogen. Nur wenn man anklickte, dass man auch betroffen sei, ging der IM nicht weg.
Gestern ist uns nichts aufgefallen aber dafür gibts heute seit ca. 14Uhr viele Störung mit von uns zur Microsoft Cloud. Teils mit abstrusen NDRs seitens Microsoft.