[German]Microsoft's cloud service Exchange Online seems to have been disrupted the day before yesterday and yesterday (March 28/29, 2023). The disruption began as early as March 28, 2023, at around 2:00 p.m., when users were unable to log in. Microsoft has confirmed this and writes that there is probably also a service disruption, so that emails on Exchange Online are quarantined. Blog-Oser Andreas P. informed me by mail and sent me the MS status information (thanks for that). Here's a quick overview of what's going on.
Advertising
It started already on 03/28/2023 at 14:51:55 when Microsoft noticed a problem. Incoming emails from users were occasionally and unexpectedly quarantined in Exchange Online. Microsoft suspected abld that this issue was caused by residual effects of a previous Azure Active Directory (Azure AD) issue that affected users with geolocation-based conditional access policies. I had reported on this issue in the GErman blog post Microsoft 365: Nutzer versehentlich per Geo-Location ausgesperrt (23. März 2023). Microsoft then confirmed this cause in further status updates.
We have confirmed a number of affected IPs with incorrect geolocation classifiers and are in the process of applying a rule to ensure that all new messages are routed and delivered via the expected geolocation. Once this process is complete, we will focus on replaying the previously affected messages to release them from quarantine and fully resolve the issue.
The cause was likely a geolocation data tagging error that resulted in incorrect geolocation classifiers being assigned to a number of affected IPs. Microsoft then rolled back backups to fix the error. Here are still the relevant status messages about the incident.
Issue: EX534197, affected service: Exchange Online, affected feature: signin, current status: Service degradation
Published Time: 29.03.2023 11:09:30
Title: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
User impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
Current status: The deployment of the fix is taking longer than expected due to additional validation steps. We're continuing to monitor the progress of the fix as it deploys to all affected environments. We anticipate its completion time by our next scheduled update.
Scope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
Start time: Tuesday, March 28, 2023, at 10:00 AM UTC
Root cause: A recent issue affecting our geolocation data marking resulted in a range of affected IPs being assigned incorrect geolocation classifiers, resulting in impact.
Next update by: Wednesday, March 29, 2023, at 10:00 PM UTC
Published Time: 29.03.2023 06:23:28
Title: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
User impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
Current status: We're continuing to monitor the progress of the fix and we anticipate its completion by the next scheduled update.
Scope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
Start time: Tuesday, March 28, 2023, at 10:00 AM UTC
Root cause: A recent issue affecting our geolocation data marking resulted in a range of affected IPs being assigned incorrect geolocation classifiers, resulting in impact.
Next update by: Wednesday, March 29, 2023, at 3:00 PM UTC
Published Time: 28.03.2023 23:44:28
Title: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
User impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
Current status: We've successfully gathered the full list of affected IPs and initiated the process of replaying the impacted messages, which will release the incorrectly sorted email from quarantine. We're expecting this process to fully resolve the impact of this issue, and it may complete as early as the time of our next scheduled update.
Scope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
Start time: Tuesday, March 28, 2023, at 10:00 AM UTC
Root cause: A recent issue affecting our geolocation data marking resulted in a range of affected IPs being assigned incorrect geolocation classifiers, resulting in impact.
Next update by: Wednesday, March 29, 2023, at 10:00 AM UTC
Published Time: 28.03.2023 21:19:34
Title: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
User impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
Current status: We've utilized a recent backup of our IP geolocation data to restore the affected IPs' incorrect geolocation classifiers, preventing further impact from the issue. Going forward, we're currently working to gather a full list of affected IPs so we can replay the impacted messages, releasing incorrectly sorted email from quarantine. We're aiming to begin the process by the time of our next scheduled update, at which point we should be able to provide a timeline for full remediation of the impact.
Scope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
Root cause: A recent issue affecting our geolocation data marking resulted in a range of affected IPs being assigned incorrect geolocation classifiers, resulting in impact.
Next update by: Wednesday, March 29, 2023, at 4:00 AM UTC
Published Time: 28.03.2023 19:04:37
Title: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
User impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
Current status: We've confirmed a range of affected IPs with incorrect geolocation classifiers, and we're in the process of applying a rule to ensure any new messages are routed through the expected geolocation and delivered. Once this process is complete, we will focus on replaying the previously affected messages to release them from quarantine and fully resolve this issue. We anticipate having a better estimation for impact resolution by our next update.
Scope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
Next update by: Wednesday, March 29, 2023, at 12:30 AM UTC
Published Time: 28.03.2023 17:18:21
Title: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
User impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
Current status: We suspect that this issue is caused by residual impact from a previous Azure Active Directory (Azure AD) issue in which users with geolocation-based conditional access policies were impacted. We're reviewing samples of affected users to investigate the affected API calls, confirm our hypothesis, and determine our next troubleshooting steps.
Scope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
Next update by: Tuesday, March 28, 2023, at 10:30 PM UTC
Published Time: 28.03.2023 15:19:58
Title: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
User impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
Current status: We're reviewing system logs and samples of affected messaged to isolate the origin of this issue.
Scope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
Next update by: Tuesday, March 28, 2023, at 8:30 PM UTC
Published Time: 28.03.2023 14:51:55
Title: We're looking into a potential problem
User impact: We're checking for potential impact to your users.
Current status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.
Advertising
