[German]Since June 5, 2023, there have been repeated problems with the availability of the Microsoft cloud and the services offered there. I had speculated in my blog posts that there might be an attack behind it, especially since a hacktivist group Anoynmous Sudan had claimed that. From Microsoft circles it was said uniso "nothing known". Now I have a Post Incident Report For Microsoft 365, that indicates, that there must have been DDoS attacks on Microsoft's cloud.
Advertising
Microsoft cloud disrupted multiple times
When I was informed about a disruption at Microsoft Exchange Online on June 5, 2023, I still assumed a technical problem – although my blog post Exchange Online down for hours (June 5, 2023) mentioned a potential attac. The reason was that a hacktivist group called Anonymous Sudan claimed responsibility for the disruption.
The hacktivists claimed to have overloaded Microsoft's servers via a DDoS attack. However, I was told by Microsoft that nothing is known about this. But somewhere in the next few days, at the beginning of June 2023, there was more service disruptions in the Microsoft cloud. On June 8, 2023, the blog post Outlook.com and OneDrive down – consequence of cyber attacks? (June 8, 2023) reported the next disruption. And on June 9, 2023, my blog post Microsoft Azure outage (June 9, 2023); what's going on? talked about a service disruption of the Azure portal.
On June 14, the Microsoft 365 portal was down for German users (see my German blog post Microsoft 365-Portal erneut mit Schluckauf (14.6.2023)).
In all posts I had included corresponding status notes from Microsoft, which were sent to me by a blog reader (thanks again for that). The question of what caused the problems remained open – there was no comment from Microsoft regarding the question of whether Anonymous Sudan was involved. Also my sources told me, that nothing was known about a hack. Today a German blog reader notified me in a private message, that he got the information early from Microsoft by phone, that the service disruption is caused by DDoS attacks, and nobody can say how long the cloud will be affected. But he wasn't able to get an email confirmation about that.
Advertising
This could also be read indirectly from the status reports in the administrator dashboard, where there was talk of traffic spikes.
A quote from the reader info: "At some point, there was even a tip to move critical apps to on-prem, if possible. Absolute nightmare the whole thing."
It was a DDoS attack after all
Somebody emailed me a post-incident report for Microsoft 365 dated June 15, 2023 for incidents EX571516, MO571683 and MO572252 at the end of the week (thanks for that), which sheds some light. I have compiled already the content below times briefly on. The disturbances (on the dates I mentioned above) affected the following services:
Users may have been unable to access Outlook on the web or other Microsoft 365 services and features.
Impacted services and features include, but were not limited to:Exchange Online
– Users were unable to access Outlook on the web
– Users may have experienced issues using the Outlook mobile application– Users may have experienced issues with the search function in Outlook on the web, Outlook desktop client and the Outlook mobile application
Microsoft Teams
– Users may have experienced difficulties scheduling meetings and/or live events
– Users may have had trouble loading people profile cards
– Users may have experienced issues loading file lists
– Users may have been unable to create new teams & channels
– Users may have been unable to install apps
– Users may have experienced issues performing searches
– Users may have seen delays in admin policy changes taking effect
– Users might have seen errors when using messaging extensions
– Users may not have seen up to date Presence information
– Teams Graph APIs may have been impacted
– Assignments tab in Teams may have not loaded
– Teams Admin Center functionalities may have not performed as expected
– Users may have been unable to view personal or channel calendar events
SharePoint Online and OneDrive for Business
– Users may have been unable to use Search functionality
Microsoft Bookings, Microsoft Power Automate and Power Apps in Microsoft 365 may have also experienced some impact related to this issue.
The impacts were specific to users served by the affected infrastructure. Over a two-day period, there were four specific impairment windows, each lasting approximately two hours. Telemetry showed that the first outage period had the greatest impact on Outlook on the Web and Representational State Transfer (REST) connections to the Exchange Online service, with service availability dropping to about 89% during the outage window.
Microsoft cloud disruptions, source: Microsoft
Traffic analysis revealed anomalous spikes in HTTP requests directed to a portion of Microsoft 365's front-end components that bypassed existing automatic recovery measures. Front-end components began to operate below acceptable thresholds, impacting features such as Outlook on the Web, REST and search capabilities. Outlook on the Web availability dropped to 89% at times.
Microsoft provided more specific time frames for the events in question, which affected cloud services worldwide, for June 5 and 6, 2023:
- First impact window – June 5, 2023, from 2:12 PM to 4:00 PM UTC – lowest point of availability was 89% for Outlook on the Web.
- Second impact window – June 5, 2023, from 7:24 PM to 10:03 PM UTC – lowest point of availability was 93% for Outlook on the Web.
- Third impact window – June 6, 2023, from 7:59 AM to 9:30 AM UTC – lowest point of availability was 95% for Outlook on the Web.
- Forth impact window – June 6, 2023, from 3:01 PM to 5:10 PM UTC– lowest point of availability was 98% for Outlook on the Web and REST.
For Microsoft, it was unfortunate that an update distribution was timed to coincide with the glitch, and they related the problems to the update. It was later discovered that the error patterns continued to occur despite fixes, and anomalous traffic pattern spikes were causing the underlying problem. Microsoft's administrators began rolling out several fixes to address the underlying issue. This included developing changes aimed at better mitigating the impact, while in parallel developing fixes to address the underlying cause and eliminate the impact. The changes affected three areas of the service:
- Improvements to load balancing operations and logic (three separate changes).
- Optimizations to specific code within applications to better handle the specific anomalous requests (two separate changes)
- Optimizations to front-end components to improve overall request processing (three separate changes)
On June 8, 2023, the third fix to handle such loads in the Microsoft Network for the Cloud was distributed. Also on June 8, 2023, traffic analysis revealed anomalous spikes in HTTP requests directed to a portion of Microsoft 365 front-end components that bypassed existing automatic recovery measures. Front-end components began to operate below acceptable thresholds, impacting features such as Outlook on the Web, REST, and search capabilities.
In the meantime, Microsoft believes that it has optimized the cloud infrastructure in such a way that such load peaks are absorbed and the services are available to users worldwide. To return to the initial information: It looks as if the claims of the hacktivists from Anonymous Sudan that they were responsible for the outages of the Microsoft Cloud were by no means made up out of thin air.
For me, the question is: What if a state with experienced hackers drives a cyberattack on the Microsoft Cloud.
Microsoft confirms Storm-1359 DDoS attack
Addendum: Microsofts has published a blog post Microsoft Response to Layer 7 Distributed Denial of Service (DDoS) Attacks and confirmed a DDos attack from Anonymus Sudan (Microsoft name it Storm-1359) with details.
Similar articles
Exchange Online down for hours (June 5, 2023)
Outlook.com and OneDrive down – consequence of cyber attacks? (June 8, 2023)
Microsoft Azure outage (June 9, 2023); what's going on?
Microsoft 365-Portal erneut mit Schluckauf (14.6.2023) (German)
Advertising
