[German]As of June 5, 2023, Microsoft's Exchange Online seems to have experienced a massive disruption. Administrators could no longer get into the Admin Center and Outlook could no longer be reached on the web. A user alerted me to this disruption, which began shortly before 3:00 p..m. UTS, but has been largely resolved 3 hours later. Microsoft has confirmed the problem in the meantime.
Advertising
A reader's message
A German blog reader emailed me shortly before 5:00 p.m. UTC to report that Exchange Online was experiencing a glitch. According to the reader, there was probably "a really big Exchange Online outage." The reader sent along the following screenshot of the Service Heath Dashboard showing the status page. In addition, the reader wrote:
Is noticeable with us despite Azure Location West (Netherlands/Ireland) that Outlook does not get a connect or Outlook Online only fragmented what displays. Sometimes it works, sometimes it doesn't, then it doesn't work again :-)
Let's see if there will be more than 100 …
In the Admin Center the whole thing looks like a red alert on the Enterprise :-D
(Click to zoom)
At reddit.com there is also this thread, where someone asks around 4:00 p.m. MEZ whether there is a malfunction. Other affected people confirmed the outage. I asked the downdetector (or the German site allestoerungen.de) and got this outage confirmed – there is a peak in the failure reports.
Advertising
Microsoft confirms the outage
On Twitter, there is now confirmation from Microsoft about a problem. On June 5, 2023, Microsoft365 status writes the following at around 16:47 German time:
We're investigating an issue with accessing Outlook on the web. Further details can be found under EX571516 in the admin center.
Later this was added as follows:
We're reviewing our networking systems and recent updates in an effort to identify the underlying root cause of the issue. Additional information can be found in the admin center under EX571516.
And the Microsoft claims to have identified the issue.
We've identified downstream impact for Microsoft Teams, SharePoint Online and OneDrive for Business. We're providing full impact details and updates for those services via MO571683. Updates pertaining to Exchange and Outlook on the web will continue to be provided via EX571516.
We've halted an ongoing deployment and are monitoring services to see if that provides relief to the environment. Further information can be found in the admin center under EX571516 and MO571683.
So there was a glitch that prevents access to Outlook on the web. This has also identified impacts to Microsoft Teams, SharePoint Online, and OneDrive for Business. Microsoft plans to provide full details on the impact and updates for these services via MO571683. Updates for Exchange and Outlook on the Web will continue to be provided via EX571516. In the meantime, Microsoft has stated in the following message that an ongoing deployment has been stopped.
We've reverted the update and telemetry shows service improvement. We're continuing to monitor the service to ensure recovery and performing actions to address any residual impact. Further information can be found under EX571516 and MO571683 in the admin center.
Microsoft started monitoring the services and hoped that the disruption will slowly subside. Whithin the following hours, some German users confirmed that there ongoing service disruptions in Exchange Online. A reader has send me the following status message from message center:
Share: Incident: EX571516, impacted service: Exchange Online, impacted feature: E-Mail and calendar access, current status is: Service restored
Published Time: 05.06.2023 19:58:46
Title: Some users were unable to access Outlook on the web, and may have experienced issues with other Exchange Online services
User impact: Users were unable to access Outlook on the web, and may have experienced issues with other Exchange Online services.
More info: Additionally, other Microsoft 365 services also experienced impact:
Microsoft Teams
– Users may have had difficulties scheduling meetings and/or live events
– Users may have had trouble loading people profile cards
– Users may have had issues loading file lists
– Users may have been unable to create new teams & channels
– Users may have been unable to install apps
– Users may have experienced issues performing searches
– Users may have seen delays in admin policy changes taking effect
– Users might have seen errors when using messaging extensions
– Users may have been unable to view up to date Presence information
– Teams Graph APIs may have been impacted
– Assignments tab in Teams may have not loaded
– Teams Admin Center functionalities were not performing as expectedSharePoint Online and OneDrive for Business
– Users may have been unable to use Search functionalityMicrosoft Power Automate in Microsoft 365
– Users were unable to trigger flows using Office 365 connectors in Microsoft Power AutomatePower Apps in Microsoft 365
– Users were unable to run apps in Power Apps that utilize Office 365 connectors
– Flows using the Office 365 Groups, Groups Mail, Outlook, Users, and Videos connectors may have been impactedFinal status: We've completed the restarts to address the residual impact, and after an extended period of monitoring, we've confirmed that the issue is resolved. We continue to investigate why the update caused impact and further details will be provided within the Post-Incident Report.
Scope of impact: Impact was primarily to users hosted within infrastructure that received the update prior to the deployment halt. However, any user may have experienced some degree of impact as described in the "More Info" section above.
Start time: Monday, June 5, 2023, at 2:10 PM UTC
End time: Monday, June 5, 2023, at 5:13 PM UTC
Preliminary root cause: A recent update, intended to add additional resilliencies against downstream request remaining in a hanging state, inadvertently caused impact.
Next steps:
– We're reviewing dump files gathered from impacted components during the incident to better understand why impact occurred.
– We're investigating to content of the update to identify the underlying root cause.We'll publish a Post-Incident report within five business days.
Published Time: 05.06.2023 19:07:56
Telemetry indicates that service availability is at approximately 99.9% across the service. We're continuing to isolate a small number of components which remain in an unhealthy state and are performing targeted restarts to remediate the residual impact.
This quick update is designed to give the latest information on this issue.
Published Time: 05.06.2023 18:19:40
Title: Some users are unable to access Outlook on the web, and may experience issues with other Exchange Online services
User impact: Users are unable to access Outlook on the web, and may experience issues with other Exchange Online services.
More info: Additionally, other Microsoft 365 services are also experiencing impact:
Microsoft Teams
– Users may have difficulties scheduling meetings and/or live events
– Users may have trouble loading people profile cards
– Users may have issues loading file lists
– Users may not be able to create new teams & channels
– Users may not be able to install apps
– Users may have trouble issuing searches
– Users may see delays in admin policy changes taking effect
– Users might see errors when using messaging extensions
– Users may not see up to date Presence information
– Teams Graph APIs may be impacted
– Assignments tab in Teams may not load
– Teams Admin Center functionalities may not perform as expectedSharePoint Online and OneDrive for Business
– Users may be unable to use Search functionalityCurrent status: We've reverted a change which occurred just prior to impact starting and after service replication took place, telemetry is showing significant improvement in availability across the service. Additionally, we've received confirmation from previously impacted users that the issue is no longer occurring. We're continuing to monitor to ensure recovery and are isolating any sections of the service showing residual impact which would require additional recovery actions.
Scope of impact: Impact is specific to users routed through the affected infrastructure.
Next update by: Monday, June 5, 2023, at 6:00 PM UTC
Published Time: 05.06.2023 17:58:00
We've prevented the potentially problematic update from propagating further across the service and are reviewing options to revert the change quickly and safely in those portions of our infrastructure where it has been applied.
This quick update is designed to give the latest information on this issue.
Published Time: 05.06.2023 17:14:38
We've identified an update which was applied around the time that impact started. We're reverting the update to potentially mitigate impact whilst we continue to investigate the underlying cause of the issue.
This quick update is designed to give the latest information on this issue.
Published Time: 05.06.2023 17:09:33
Title: Some users are unable to access Outlook on the web.
User impact: Users are unable to access Outlook on the web.
More info: Additionally, other Microsoft 365 services are also experiencing impact:
Microsoft Teams
– Users may have difficulties scheduling meetings and/or live events
– Users may have trouble loading people profile cards
– Users may have issues loading file lists
– Users may not be able to create new teams & channels
– Users may not be able to install apps
– Users may have trouble issuing searches
– Users may see delays in admin policy changes taking effect
– Users might see errors when using messaging extensions.
– Teams Graph APIs may be impactedCurrent status: We're continuing to review monitoring health systems targeted at networking, Client Access Front End (CAFE) routing, and database storage to determine the underlying cause of the issue.
Scope of impact: Impact is specific to users routed through the affected infrastructure.
Next update by: Monday, June 5, 2023, at 4:30 PM UTC
Published Time: 05.06.2023 16:52:56
Title: Some users may be unable to access Outlook on the web.
User impact: Users may be unable to access Outlook on the web.
Current status: We're reviewing monitoring telemetry to isolate the source of the issue.
Scope of impact: Impact is specific to users routed through the affected infrastructure.
Next update by: Monday, June 5, 2023, at 4:00 PM UTC
Published Time: 05.06.2023 16:32:40
Title: We're looking into a potential problem.
User Impact: We're investigating the potential end-user impact.
Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 60 minutes.
And here is the 2nd status entry
Share: Incident: MO572252, impacted service: Microsoft 365 suite, impacted feature: Portal, current status is: Service degradation
Published Time: 05.06.2023 22:11:47
Title: Can't access Outlook on the web and other Microsoft services and features
User impact: Users may be unable to access Outlook on the web or other Microsoft 365 services and features.
More info: Impacted services and features may include, but are not limited to:
Exchange Online
– Users are unable to access Outlook on the webMicrosoft Teams
– Users may have difficulties scheduling meetings and/or live events
– Users may have trouble loading people profile cards
– Users may have issues loading file lists
– Users may not be able to create new teams & channels
– Users may not be able to install apps
– Users may have trouble issuing searches
– Users may see delays in admin policy changes taking effect
– Users might see errors when using messaging extensions
– Users may not see up to date Presence information
– Teams Graph APIs may be impacted
– Assignments tab in Teams may not load
– Teams Admin Center functionalities may not perform as expectedSharePoint Online, OneDrive for Business, Microsoft Power Automate in Microsoft 365 and Power Apps in Microsoft 365 may also be experiencing some impact related to this issue.
Current status: We've determined that impact associated with MO571683 and EX571516 is reoccurring. We suspect that impact to our services is similar, though we're investigating to determine the scope of impact and the source of the issue.
Scope of impact: Users routed through the affected infrastructure are affected by this issue, however, impact can vary depending on where users are hosted.
Next update by: Monday, June 5, 2023, at 9:00 PM UTC
Published Time: 05.06.2023 19:57:28
Title: Some users experienced impact to multiple Microsoft 365 services and features
User impact: Users experienced impact to multiple Microsoft 365 services and features.
More info: Impacted services and features include, but were not limited to:
Exchange Online
– Users were unable to access Outlook on the web (as communicated under EX571516)Microsoft Teams
– Users may have had difficulties scheduling meetings and/or live events
– Users may have had trouble loading people profile cards
– Users may have had issues loading file lists
– Users may have been unable to create new teams & channels
– Users may have been unable to install apps
– Users may have experienced issues performing searches
– Users may have seen delays in admin policy changes taking effect
– Users might have seen errors when using messaging extensions
– Users may have been unable to view up to date Presence information
– Teams Graph APIs may have been impacted
– Assignments tab in Teams may have not loaded
– Teams Admin Center functionalities were not performing as expectedSharePoint Online and OneDrive for Business
– Users may have been unable to use Search functionalityMicrosoft Power Automate in Microsoft 365
– Users were unable to trigger flows using Office 365 connectors in Microsoft Power AutomatePower Apps in Microsoft 365
– Users were unable to run apps in Power Apps that utilize Office 365 connectors
– Flows using the Office 365 Groups, Groups Mail, Outlook, Users, and Videos connectors may have been impactedFinal status: We've completed the restarts to address the residual impact, and after an extended period of monitoring, we've confirmed that the issue is resolved. We continue to investigate why the update caused impact and further details will be provided within the Post-Incident Report.
Scope of impact: Impact was primarily to users hosted within infrastructure that received the update prior to the deployment halt. However, any user may have experienced some degree of impact as described in the "More Info" section above and in Service Health Dashboard post MO571683.
Start time: Monday, June 5, 2023, at 2:10 PM UTC
End time: Monday, June 5, 2023, at 5:13 PM UTC
Preliminary root cause: A recent update, intended to add additional resilliencies against downstream request remaining in a hanging state, inadvertently caused impact.
Next steps:
– We're reviewing dump files gathered from impacted components during the incident to better understand why impact occurred.
– We're investigating to content of the update to identify the underlying root cause.We'll publish a Post-Incident report within five business days.
Published Time: 05.06.2023 19:07:52
Telemetry indicates that service availability is at approximately 99.9% across the service. We're continuing to isolate a small number of components which remain in an unhealthy state and are performing targeted restarts to remediate the residual impact.
This quick update is designed to give the latest information on this issue.
Published Time: 05.06.2023 18:21:37
Title: Some users are experiencing impact to multiple Microsoft 365 services and features
User impact: Users are experiencing impact to multiple Microsoft 365 services and features.
More info: Impact services and features include, but are not limited to:
Exchange Online
– Users are unable to access Outlook on the web (as communicated under EX571516)Microsoft Teams
– Users may have difficulties scheduling meetings and/or live events
– Users may have trouble loading people profile cards
– Users may have issues loading file lists
– Users may not be able to create new teams & channels
– Users may not be able to install apps
– Users may have trouble issuing searches
– Users may see delays in admin policy changes taking effect
– Users might see errors when using messaging extensions
– Users may not see up to date Presence information
– Teams Graph APIs may be impacted
– Assignments tab in Teams may not load
– Teams Admin Center functionalities may not perform as expectedSharePoint Online and OneDrive for Business
– Users may be unable to use Search functionalityCurrent status: We've reverted a change which occurred just prior to impact starting and after service replication took place, telemetry is showing significant improvement in availability across the service. Additionally, we've received confirmation from previously impacted users that the issue is no longer occurring. We're continuing to monitor to ensure recovery and are isolating any sections of the service showing residual impact which would require additional recovery actions.
Scope of impact: Impact is specific to users routed through the affected infrastructure.
Next update by: Monday, June 5, 2023, at 6:00 PM UTC
Published Time: 05.06.2023 17:58:21
We've identified an update which was applied around the time that impact started. We've prevented the potentially problematic update from propagating further across the service and are reviewing options to revert the change quickly and safely in those portions of our infrastructure where it has been applied.
This quick update is designed to give the latest information on this issue.
Published Time: 05.06.2023 17:29:32
Title: Some users are experiencing impact to multiple Microsoft 365 services and features
User impact: Users are experiencing impact to multiple Microsoft 365 services and features.
More info: Impact services and features include, but are not limited to:
Exchange Online
– Users are unable to access Outlook on the web (as communicated under EX571516)Microsoft Teams
– Users may have difficulties scheduling meetings and/or live events
– Users may have trouble loading people profile cards
– Users may have issues loading file lists
– Users may not be able to create new teams & channels
– Users may not be able to install apps
– Users may have trouble issuing searches
– Users may see delays in admin policy changes taking effect
– Users might see errors when using messaging extensions.
– Teams Graph APIs may be impactedSharePoint Online and OneDrive for Business
– Users may be unable to use Search functionalityCurrent status: We're reviewing monitoring health systems targeted at networking, Client Access Front End (CAFE) routing, and database storage to determine the underlying cause of the issue.
Scope of impact: Impact is specific to users routed through the affected infrastructure.
Next update by: Monday, June 5, 2023, at 4:30 PM UTC
Were any of you affected? If anyone has access to the status messages in the Admin Center, they can post them here.
not fully resolved yet, guenni
Outlook.com was sometimes not responding around 6/5 – 6/7:
https://www.neowin.net/news/microsoft-confirms-massive-issues-and-outage-across-exchange-outlook-teams-onedrive/
{see "Update 4" section of the Neowin article}
https://www.bleepingcomputer.com/news/microsoft/outlookcom-hit-by-outages-as-hacktivists-claim-ddos-attacks/