[German]After the CrowdStrike Falcon software paralyzed 8.5 million Windows computers some time ago, the provider has now issued a second statement. According to the statement, 99% of the sensors are now back in operation. Otherwise, some of those affected are threatening to sue for damages. Delta Air Lines' approach has made it into the media. Now there is the first counterattack: Microsoft had offered Delta Air Lines free support, but this was rejected. Here is a summary of the relevant information.
Advertising
The CrowdStrike incident in retrospect
On July 19, 2024, a faulty signature update in the CrowdStrike Falcon security software caused 8.5 million Windows systems to fail. Most of them remained in a blue screen loop and could no longer be booted in some cases. Corporate computers were affected, as the above-mentioned CrowdStrike Falcon software is not used by private individuals. As a result, airports came to a standstill, trains, radio stations, petrol stations, stores and banks were affected.
Administrators at the affected companies were told to try booting the Windows computers up to 15 times and hope that the faulty update would be replaced by a working version via the Internet. Or manually remove the faulty update from the computers on site and make them work again. I also reported on the problems with Bitlocker recovery key queries in a timely manner here in the blog (see article links at the end of the post).
Technical analysis of the incident (Channel File 291)
CrowdStrike had already published an initial analysis of the cause of the BlueScreens a few days after the incident. I took this up in the blog post CrowdStrike: Investigation report; amount of damages and compensation; attribution of blame. Microsoft had also published an analysis (see Microsoft's analysis of the CrowdStrike incident and recommendations).
CrowdStrike has now published a full report entitled External Technical Root Cause Analysis — Channel File 291 as a 12-page PDF file. I was made aware of this on Facebook and via the above tweet. The following letter to customers and partners was sent to me by a reader (thanks for that).
Advertising
According to CrowdStrike, the PDF report expands on the information already provided in the preliminary post-incident review. It explains in more detail what was incorrect in the logic of the interpreter for the sensors and what was wrong in the development process. The beef can be found on this CrowdStrike page in a post dated 6.8.2024, hidden in a jumble of text. I'll extract the relevant sentences.
On July 19, 2024, a Rapid Response Content Update was delivered to certain Windows hosts, which further developed the new feature first released in February 2024. The sensor expected 20 input fields, while the update delivered 21 input fields. In this case, the discrepancy led to an unauthorized memory access that caused a system crash.
According to the report, the provider wants to make improvements to avoid such errors in the future.
What is also new for me is the information that as of July 29, 2024, ~99% of Windows sensors are online again compared to the previous week or before the incident. The provider states that there is normally a deviation of approx. 1% from week to week in the sensor connections. Interested readers can read the details in the linked document. In general, the question arises as to what other problems lie dormant that will soon fall on the feet of users. The following tweet takes up the idea and addresses two major failures.
Insurance, compensation and counterclaims
Let's move on to the subject of damages, compensation from insurance companies as well as the alluded to claims for damages against CrowdStrike. The Delta Air Lines case in particular has been in the news because they are claiming enormous damages.
Damages in the billions, and uninsured
In the blog post CrowdStrike: Investigation report; amount of damages and compensation; attribution of blame, I ventured an initial assessment of the damage caused by the incident. The worldwide outage of Windows systems caused by CrowdStrike has already cost this US Fortune 500 company 5.4 billion US dollars.
The insurer Parametrix writes that banking and healthcare companies as well as large airlines are likely to suffer the greatest losses. According to this neowin.net article, the global damage caused by the CrowdStrike incident is estimated at 15 billion US dollars.
Bloomberg states in this article (paywall) that insurance companies will have to pay between 300 million and 1.5 billion US dollars to those affected. In other words, a large part of the damage caused by the Windows outages is uninsured. It wasn't a cyber attack, and whether an insurance company will pay out, for example in the context of business interruption, can only be found in the small print.
The matter of compensation
In the blog post CrowdStrike: Investigation report; amount of damages and compensation; attribution of blame, I discussed initial thoughts on the topic of compensation for damages caused by CrowdStrike. The CrowdStrike T&Cs are likely to leave many claims for damages in the lurch, as applications where major damage could be caused by a failure are explicitly excluded.
But there is the airline Delta, which claims to have suffered 500 million US dollars in damages. Delta wants to have this damage compensated by CrowdStrike and Microsoft (see this CNBC article).
Answers tp Delta Air Lines' claims for damages
Delta Air Lines' claims for damages against CrowdStrike and Microsoft have already provoked reactions and a counterattack by the latter two companies. CrowdStrike is quoted by The Register in this article as saying that it is "very disappointed" by the threat of legal action. It rejects claims by Delta and its lawyers that the provider was grossly negligent in the events that led to the global IT outage. Statement from CrowdStrike according to The Register:
If Delta continues down this path, Delta will have to explain to the public, its shareholders and ultimately a jury why CrowdStrike took responsibility for its actions – quickly, transparently and constructively – and Delta did not.
CrowdStrike also plays the card of why the airline took so much longer than its competitors to recover from the same problem (Neowin. And it raises the question of why Delta turned it down the free on-site assistance offered by CrowdStrike. The Verge picked it up here. The letter from CrowdStrike to Delta can be viewed on X in this tweet.
Microsoft is also taking the same line, as you can read on CNBC in this article. There, Microsoft is countered with the statement "Our preliminary review suggests that Delta, unlike its competitors, has not modernized its IT infrastructure for the benefit of its customers or its pilots and flight attendants". Delta argues that it is making long-term investments and has spent billions of dollars on its IT infrastructure since 2016.
Microsoft's Attorney Cheffo wrote that Microsoft had offered to help Delta free of charge. Every day from July 19 to July 23, Microsoft employees offered to help repair the damage. But Delta rejected the offer each time, the lawyer's letter states. Microsoft CEO Satya Nadella had sent an email to Delta CEO Bastian, "who never responded", the lawyer's letter states. CrowdStrike also argues that its CEO, George Kurtz, contacted his counterpart at Delta but received no response.
Cheffo quotes a letter from Microsoft to a Delta employee dated July 22, where help was offered. The response from the Delta employee was: "All good. Cool, we'll let you know and thanks for the offer. CNBC mentions that Delta signed a deal with IBM in 2021 to implement a hybrid cloud architecture running on Red Hat's OpenShift software. In 2022, Amazon announced that Delta had selected the digital commerce company's Amazon Web Services unit as its preferred cloud provider.
The disaster at Delta – some details
The Delta CEO states that the company had to manually reset 40,000 servers. Microsoft's attorney Cheffo wrote to Delta: "It quickly becomes clear that Delta likely refused Microsoft's help because the IT system that had the most trouble recovering – the crew tracking and scheduling system – was maintained by other technology vendors, such as IBM, because it runs on those vendors' systems and not on Microsoft Windows or Azure."
Delta CEO Bastian told CNBC "If you're going to have access, priority access, to the Delta ecosystem in terms of technology, you have to test these things. You can't come into a mission-critical 24/7 operation and tell us we have a bug. It's not working."
Regardless of the last statement, Microsoft's response is also a cheap retort of the "if you had Windows 11 and Microsoft Azure, you would have been up and running faster" variety. Just a note from my side: As far as I know, only Windows systems were affected – if Delta relied on Red Hat, the Linux servers were not affected. So there must have been 40,000 Windows servers that were bricked by the CrowdStrike Falcon update. It took five days to fix them.
There is a blog post by Bastian stating that Delta uses a significant number of applications that rely on Windows. This includes one of the tools used to track the location and duty indicators of crews. This system was affected by the CrowdStrike incident and could not effectively handle the unprecedented number of changes triggered by the [Windows] system shutdown, it says. Delta teams have been working around the clock to restore the system and ensure full functionality.
You can also boil the above down to "get out the popcorn and watch the beat exchange". We may see some more reports on this here on the blog.
Similar articles:
Worldwide outage of Microsoft 365 (July 19, 2024)
Windows systems throw BSOD due to faulty CrowdStrike update
Why numerous IT systems around the world failed due to two errors on July 19, 2024
CrowdStrike analysis: Why an empty file led to BlueSceen
Review of the CrowdStrike incident, the biggest computer glitch of all time
CrowdStrike incident: sensor failure as a previously unknown side effect?
CrowdStrike: Investigation report; amount of damages and compensation; attribution of blame
Microsoft's analysis of the CrowdStrike incident and recommendations
Advertising
