On August 10, 2022, Microsoft 365 services experienced an outage that specifically affected North America but also EMEA. Users had problems with Office 365, Outlook and other services. It was probably due to a false alarm within the Cisco Meraki firewall used by Microsoft.
Advertising
Microsoft has admitted to problems with its cloud services on Twitter, as the following tweets show. The initial message already confirmed that network traffic was blocked across various regions.
It quickly became clear that the whole thing was related to firewall solutions used by Microsoft. Blog reader Markus pointed me to this post from Bleeping Computer via email (thanks for that). The outage was triggered by a false alarm in the firewall in use, which prevented users from connecting to Exchange Online, Microsoft Teams, Outlook desktop clients and OneDrive for Business.
A Cisco employee addressed this in this forum post. A vulnerability reported by Microsoft, CVE-2022-35748, triggers SNORT rule 1-60381, which caused problems with communication through the firewall. In the meantime, however, Microsoft has probably fixed the problem (see also the notes at Bleeping Computer).
Blog reader Andreas P. sent me the following excerpts from the status area of the Admin Center (thanks for that).
Advertising
Published Time: 10.08.2022 19:56:28
The firewall partner is currently reviewing options to remediate impact.
This quick update is designed to give the latest information on this issue.
Published Time: 10.08.2022 19:02:39
Title: Some users may be unable to connect to multiple Microsoft 365 services.
User Impact: Users may be unable to connect to multiple Microsoft 365 services.
More info: Impacted services include, but are not limited to:
– Outlook desktop client
– OneDrive for Business
– Microsoft Teams
Affected customers have reported that disabling firewall rules blocking TLS 1.2 is mitigating impact. Some firewall vendors have published guidance on disabling the impacting rules, and we recommend contacting your firewall vendor for further assistance.
Current status: We continue to work with the firewall partner to investigate a Snort rule which is contributing to impact. Our focus remains on mitigation and from user reports, disabling the specific firewall rule provides immediate relief. Additionally, we continue to investigate recent changes within the Microsoft-managed environment to rule out potential causes of impact.
Scope of impact: At this time, impact appears to be specific to some users who are served through the affected infrastructure.
Next update by: Wednesday, August 10, 2022, at 6:30 PM UTC
Published Time: 10.08.2022 18:41:46
We're continuing to work with the firewall partner to investigate the issue. Additionally, we monitoring feedback from impacted organizations that disabling a specific firewall rule, which blocks TLS 1.2, is mitigating impact.
This quick update is designed to give the latest information on this issue.
Published Time: 10.08.2022 17:25:42
Title: Some users may be unable to connect to multiple Microsoft 365 services.
User Impact: Users may be unable to connect to multiple Microsoft 365 services.
More info: Impacted services include, but are not limited to:
– Outlook desktop client
– OneDrive for Business
– Microsoft Teams
Affected customers have reported that disabling firewall rules blocking TLS 1.2 is mitigating impact.
Current status: We've identified an increase in errors related to TLS 1.0 and 1.1 across Microsoft 365 services. We've confirmed that there have not been any recent changes to the service feature which is blocking the traffic. We're continuing to engage with the firewall partners to assist our investigation into the potential blocking of legitimate traffic. Additionally, we're working with impacted users to gather client logs.
Scope of impact: At this time, impact appears to be specific to some users who are served through the affected infrastructure.
Next update by: Wednesday, August 10, 2022, at 5:00 PM UTC
Published Time: 10.08.2022 17:01:14
We're directly working with some of the affected users to aid in our investigation while continuing to engage with our firewall partners. Analysis into Microsoft 365 client endpoints is ongoing.
This quick update is designed to give the latest information on this issue.
Published Time: 10.08.2022 16:28:30
We're looking at recent changes made within the Microsoft-managed infrastructure and reviewing endpoints that are leveraging TLS 1.2. Additionally, we're contacting firewall partners to assist our investigation.
This quick update is designed to give the latest information on this issue.
Published Time: 10.08.2022 15:53:29
Title: Some users may be unable to connect to multiple Microsoft 365 services.
User Impact: Users may be unable to connect to multiple Microsoft 365 services.
More info: Impacted services include, but are not limited to:
– Outlook desktop client
– OneDrive for Business
– Microsoft Teams
Current status: After analyzing system telemetry and Fiddler logs from impacted users, we suspect that third-party firewall devices are potentially blocking legitimate Microsoft traffic. Affected customers have reported that disabling firewall rules blocking TLS 1.2 is mitigating impact. We're continuing our investigation into the underlying cause.
Scope of impact: At this time, impact appears to be specific to some users who are served through the affected infrastructure.
Next update by: Wednesday, August 10, 2022, at 3:30 PM UTC
Published Time: 10.08.2022 15:25:09
Some customers are able to mitigate impact by disabling a firewall rule that is blocking TLS 1.2.
This quick update is designed to give the latest information on this issue.
Published Time: 10.08.2022 14:57:17
We're reviewing Exchange trace logs (ETL) from users who are experiencing impact. We believe the issue may be related to Active Directory (AD) services and are investigating this further.
This quick update is designed to give the latest information on this issue.
Published Time: 10.08.2022 14:41:52
Title: Some users may be unable to connect to multiple Microsoft 365 services.
User Impact: Users may be unable to connect to multiple Microsoft 365 services.
More info: Impacted services include, but are not limited to:
– Outlook desktop client
– OneDrive for Business
– Microsoft Teams
Current status: We're reviewing system telemetry to isolate the source of the issue. Additionally, we're working with impacted users to gather network trace logs to assist our investigation.
Scope of impact: At this time, impact appears to be specific to some users who are served through the affected infrastructure.
Next update by: Wednesday, August 10, 2022, at 2:00 PM UTC
Note: Wrong product name used initially has been amended.
Advertising
