{"id":29206,"date":"2023-03-30T08:14:23","date_gmt":"2023-03-30T06:14:23","guid":{"rendered":"http:\/\/159.69.82.204\/win\/?p=29206"},"modified":"2023-03-30T08:14:23","modified_gmt":"2023-03-30T06:14:23","slug":"outage-ex534197-with-exchange-online-march-28-29-2023","status":"publish","type":"post","link":"https:\/\/borncity.com\/win\/2023\/03\/30\/outage-ex534197-with-exchange-online-march-28-29-2023\/","title":{"rendered":"Outage EX534197 with Exchange Online (March 28\/29, 2023)"},"content":{"rendered":"

\"Stop[German<\/a>]Microsoft's cloud service Exchange Online seems to have been disrupted the day before yesterday and yesterday (March 28\/29, 2023). The disruption began as early as March 28, 2023, at around 2:00 p.m., when users were unable to log in. Microsoft has confirmed this and writes that there is probably also a service disruption, so that emails on Exchange Online are quarantined. Blog-Oser Andreas P. informed me by mail and sent me the MS status information (thanks for that). Here's a quick overview of what's going on.<\/p>\n

<\/p>\n

\"\"It started already on 03\/28\/2023 at 14:51:55 when Microsoft noticed a problem. Incoming emails from users were occasionally and unexpectedly quarantined in Exchange Online. Microsoft suspected abld that this issue was caused by residual effects of a previous Azure Active Directory (Azure AD) issue that affected users with geolocation-based conditional access policies. I had reported on this issue in the GErman blog post Microsoft 365: Nutzer versehentlich per Geo-Location ausgesperrt (23. M\u00e4rz 2023)<\/a>. Microsoft then confirmed this cause in further status updates.<\/p>\n

We have confirmed a number of affected IPs with incorrect geolocation classifiers and are in the process of applying a rule to ensure that all new messages are routed and delivered via the expected geolocation. Once this process is complete, we will focus on replaying the previously affected messages to release them from quarantine and fully resolve the issue.<\/p><\/blockquote>\n

The cause was likely a geolocation data tagging error that resulted in incorrect geolocation classifiers being assigned to a number of affected IPs. Microsoft then rolled back backups to fix the error. Here are still the relevant status messages about the incident.<\/p>\n

Issue: EX534197, affected service: Exchange Online, affected feature: signin, current status: Service degradation<\/p>\n

Published Time: 29.03.2023 11:09:30
\nTitle: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
\nUser impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
\nCurrent status: The deployment of the fix is taking longer than expected due to additional validation steps. We're continuing to monitor the progress of the fix as it deploys to all affected environments. We anticipate its completion time by our next scheduled update.
\nScope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
\nStart time: Tuesday, March 28, 2023, at 10:00 AM UTC
\nRoot cause: A recent issue affecting our geolocation data marking resulted in a range of affected IPs being assigned incorrect geolocation classifiers, resulting in impact.
\nNext update by: Wednesday, March 29, 2023, at 10:00 PM UTC
\nPublished Time: 29.03.2023 06:23:28
\nTitle: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
\nUser impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
\nCurrent status: We're continuing to monitor the progress of the fix and we anticipate its completion by the next scheduled update.
\nScope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
\nStart time: Tuesday, March 28, 2023, at 10:00 AM UTC
\nRoot cause: A recent issue affecting our geolocation data marking resulted in a range of affected IPs being assigned incorrect geolocation classifiers, resulting in impact.
\nNext update by: Wednesday, March 29, 2023, at 3:00 PM UTC
\nPublished Time: 28.03.2023 23:44:28
\nTitle: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
\nUser impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
\nCurrent status: We've successfully gathered the full list of affected IPs and initiated the process of replaying the impacted messages, which will release the incorrectly sorted email from quarantine. We're expecting this process to fully resolve the impact of this issue, and it may complete as early as the time of our next scheduled update.
\nScope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
\nStart time: Tuesday, March 28, 2023, at 10:00 AM UTC
\nRoot cause: A recent issue affecting our geolocation data marking resulted in a range of affected IPs being assigned incorrect geolocation classifiers, resulting in impact.
\nNext update by: Wednesday, March 29, 2023, at 10:00 AM UTC
\nPublished Time: 28.03.2023 21:19:34
\nTitle: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
\nUser impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
\nCurrent status: We've utilized a recent backup of our IP geolocation data to restore the affected IPs' incorrect geolocation classifiers, preventing further impact from the issue. Going forward, we're currently working to gather a full list of affected IPs so we can replay the impacted messages, releasing incorrectly sorted email from quarantine. We're aiming to begin the process by the time of our next scheduled update, at which point we should be able to provide a timeline for full remediation of the impact.
\nScope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
\nRoot cause: A recent issue affecting our geolocation data marking resulted in a range of affected IPs being assigned incorrect geolocation classifiers, resulting in impact.
\nNext update by: Wednesday, March 29, 2023, at 4:00 AM UTC
\nPublished Time: 28.03.2023 19:04:37
\nTitle: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
\nUser impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
\nCurrent status: We've confirmed a range of affected IPs with incorrect geolocation classifiers, and we're in the process of applying a rule to ensure any new messages are routed through the expected geolocation and delivered. Once this process is complete, we will focus on replaying the previously affected messages to release them from quarantine and fully resolve this issue. We anticipate having a better estimation for impact resolution by our next update.
\nScope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
\nNext update by: Wednesday, March 29, 2023, at 12:30 AM UTC
\nPublished Time: 28.03.2023 17:18:21
\nTitle: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
\nUser impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
\nCurrent status: We suspect that this issue is caused by residual impact from a previous Azure Active Directory (Azure AD) issue in which users with geolocation-based conditional access policies were impacted. We're reviewing samples of affected users to investigate the affected API calls, confirm our hypothesis, and determine our next troubleshooting steps.
\nScope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
\nNext update by: Tuesday, March 28, 2023, at 10:30 PM UTC
\nPublished Time: 28.03.2023 15:19:58
\nTitle: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online
\nUser impact: Users' incoming email is intermittently and unexpectedly quarantined in Exchange Online.
\nCurrent status: We're reviewing system logs and samples of affected messaged to isolate the origin of this issue.
\nScope of impact: Your organization is affected by this event and any user may notice that some email that they view as legitimate are quarantined.
\nNext update by: Tuesday, March 28, 2023, at 8:30 PM UTC
\nPublished Time: 28.03.2023 14:51:55
\nTitle: We're looking into a potential problem
\nUser impact: We're checking for potential impact to your users.
\nCurrent status: We're investigating a potential issue and checking for impact to your organization. We'll provide an update within 30 minutes.<\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"

[German]Microsoft's cloud service Exchange Online seems to have been disrupted the day before yesterday and yesterday (March 28\/29, 2023). The disruption began as early as March 28, 2023, at around 2:00 p.m., when users were unable to log in. Microsoft … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[63,463],"tags":[64,47],"class_list":["post-29206","post","type-post","status-publish","format-standard","hentry","category-cloud","category-issue","tag-cloud","tag-issue"],"_links":{"self":[{"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/posts\/29206","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/comments?post=29206"}],"version-history":[{"count":0,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/posts\/29206\/revisions"}],"wp:attachment":[{"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/media?parent=29206"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/categories?post=29206"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/tags?post=29206"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}