{"id":25898,"date":"2022-07-28T23:06:35","date_gmt":"2022-07-28T21:06:35","guid":{"rendered":"http:\/\/159.69.82.204\/win\/?p=25898"},"modified":"2022-08-03T21:05:07","modified_gmt":"2022-08-03T19:05:07","slug":"cisco-webex-down-28-7-2022","status":"publish","type":"post","link":"https:\/\/borncity.com\/win\/2022\/07\/28\/cisco-webex-down-28-7-2022\/","title":{"rendered":"Cisco Webex has been down (July 28, 2022)"},"content":{"rendered":"

\"Stop[German<\/a>]German blog reader Gerald just emailed me to let me know (thanks for that) that the Cisco Webex video conferencing service was down. I had a quick look, the disruption started at about 19:00 German time and lasted at about 22:00. The Webex Room Systems service is currently (22:41) still listed with a \"Major Impact\" and Cisco is also still listing a disruption with the Webex Clod Registered Device. So the disruption seems to be slowly subsiding.<\/p>\n


\nGerald has send the following screenshot with the WebEx status a couple of hours ago.<\/p>\n

\"Cisco<\/p>\n

\"\"The following graphic is available on allestoerungen.de<\/a> and shows the start and end of the disruption on July 28, 2022 here in Germany (Central European Time, CET, UTC +1).<\/p>\n

\"Webex<\/a><\/p>\n

On Twitter, Webex support confirmed<\/a> problems with the service two hours ago. It says there:<\/p>\n

We're aware of issues affecting several Webex services and our engineering team is hard at work. We apologize for the inconvenience. Regular updates will be shared here as soon we as have them. We appreciate your patience.<\/p>\n

In addition, users may experience difficulties logging into Webex Control Hub and managing their Webex Cloud device. All hands are on deck to restore services and we apologize for the inconvenience this may cause.<\/p>\n

 <\/p>\n

Update to service (1\/2) Engineering is investigating an issue which is causing connectivity issues with the Webex App. This may include logging in, sending messages and files, presence issues, and starting or joining meetings in the Webex App.<\/p><\/blockquote>\n

But the technicians seem to have quickly got to grips with the problem, because 44 minutes ago they said:<\/p>\n

Engineering is continuing to take remediation steps to restore services. We appreciate everyone's patience while we are all hands on deck to address the incident.<\/p>\n

For users who are still unable to log into the Webex App, we recommend that you close and re-launch the Webex App. We will continue to provide updates as they become available. Once again, we appreciate your patience today.<\/p><\/blockquote>\n

And the last message a few minutes ago confirms that the service is probably largely working again.<\/p>\n

The majority of services have been restored and are operational. Some users may still experience intermittent issues accessing their Webex board and devices.<\/p><\/blockquote>\n

Here is the overview of the Cisco Webex status page, which can be accessed here<\/a>, where only residual faults can be seen<\/p>\n

\"Webex<\/a><\/p>\n

Interesting in this context is the following tweet<\/a>, which also reports an AWS problem on the East Coast of the USA.<\/p>\n

\"AWS<\/a><\/p>\n

Addendum: Cisco has released the following report about the incident.<\/p>\n

Webex Services Incident<\/p>\n

Incident Number: INC0047261
\nIncident Duration: July 28, 2022, 16:55 \u2013 19:55 UTC<\/p>\n

Incident Details
\nAt 16:55 UTC on July 28 th , a service provider hosting some of the Webex services experienced a significant outage. Services for messaging,
\ndevices, authentication, and analytics were hosted in the affected service provider data center, which caused multiple microservices to
\nfail. Users were unable to authenticate to Webex, register or connect to meetings consistently from Webex devices, use messaging, or log into the Control Hub administration console. The Webex status page was also hosted in the same service provider data center, which caused delayed access to the status page.<\/p>\n

Root Cause
\nThe Webex engineering team attempted to redirect services outside of the affected environment; however, the redirects were not
\nsuccessful due to the outage affecting multiple redundant service zones. Due to a core component of the service architecture becoming
\nunavailable, the redirects to alternative zones were ineffective. Engineering was unable to successfully stop the services in the hosted
\ndatacenter due to connectivity failures. This caused services to remain unstable as device and software clients continued to send
\nconnection retries. The combination of incomplete service termination, the unhealthy data center, and the multiple client retries caused the connections to overload the available service capacity, and the capacity of the edge and microservices in the redundant environment was unable to process the traffic.<\/p>\n

Corrective Actions
\nEngineering worked with the service provider to restore the core service architecture and scaled up the edge and service capacity within
\nthe environment which allowed the clients to successfully connect, and the additional traffic generated by the retries to stop. This caused
\nservices to stabilize, which allowed engineering to take additional remediation steps, including restarting unhealthy instances and
\nadditional load balancing, leading to full service recovery. Engineering completed remaining service clean-up and load balancing, and services were fully recovered at 19:55 UTC.
\nThe service team has identified areas of improvement for our incident response time, including changes to the service metrics for faster root cause identification, improved runbooks to speed up scale-up and deployment of additional micro-services, and revisiting client retry values which will improve service restoration should a similar incident occur in the future. The messaging architecture team is also engaged to identify service architecture improvements which will allow Webex services to better withstand a multi-zone failure.<\/p>\n

Timeline
\n16:55: Webex alerts indicate multiple service failures; incident process began
\n17:15: Engineering began service redeploys
\n17:33: Services fully stopped in the affected DC
\n17:45: Service provider reported partial recovery
\n18:00: Metrics indicate high traffic volume due to client retries
\n18:30: Multiple scale-up efforts completed
\n19:10: Capacity increase across both pools completed; 80% increase in traffic observed. Service redeploys began
\n19:45: Vendor confirms services fully restored. Additional capacity deploys completed in all three zones
\n19:55: Final redeploys completed; services restored.<\/p><\/blockquote>\n","protected":false},"excerpt":{"rendered":"

[German]German blog reader Gerald just emailed me to let me know (thanks for that) that the Cisco Webex video conferencing service was down. I had a quick look, the disruption started at about 19:00 German time and lasted at about … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[463],"tags":[47],"class_list":["post-25898","post","type-post","status-publish","format-standard","hentry","category-issue","tag-issue"],"_links":{"self":[{"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/posts\/25898","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/comments?post=25898"}],"version-history":[{"count":0,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/posts\/25898\/revisions"}],"wp:attachment":[{"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/media?parent=25898"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/categories?post=25898"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/borncity.com\/win\/wp-json\/wp\/v2\/tags?post=25898"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}