Review of the CrowdStrike incident, the biggest computer glitch of all time

[German]It is considered to be the world's biggest computer glitch to date, paralyzing 8.5 million Windows systems and causing billions in damage. In the meantime, the clean-up work, the finger-pointing and the "washing of hands in innocence" have begun. I would like to collect some more information as part of a review. From the extent of the impact, to recovery measures, to questions and answers that others are now asking.

The CrowdStrike incident

On July 19, 2024, there were numerous disruptions to IT systems with Windows worldwide. Operations at airports came to a standstill, banks could no longer work, trains were canceled and companies sent their employees home (e.g. Tegut) because the IT systems stopped working. German municipalities, companies and banks were also affected. I reported on this in the articles Worldwide outage of Microsoft 365 (July 19, 2024) and Windows systems throw BSOD due to faulty CrowdStrike update.

However, it was not a cyberattack, but a faulty update for an EDR security solution from the US provider CrowdStrike, which caused Windows systems to crash with a blue screen. An initial analysis of what might have happened can be found in my blog post CrowdStrike analysis: Why an empty file led to BlueSceen. In this tweet, someone cites that the flawed update was intended to target newly observed malicious "named pipes" used by common C2 frameworks in cyberattacks.

But it went wrong – and there was a lot going on at Microsoft and in some companies – this tweet contains a video that was sent to me.

How big was the incident?

The interesting question is how big the impact was. Experts are calling the incident the biggest computer glitch of all time

Only 8.5 million Windows systems affected

On July 20, 2024, Microsoft published the Techcommunity article Helping our customers through the CrowdStrike outage and reported on its efforts to help affected customers out of the patch. The figure of 8.5 million affected Windows systems was also mentioned.

This is less than one percent of all Windows computers, but has far-reaching economic and social implications, reflecting the use of CrowdStrike by companies.

This relativization of "less than 1% of Windows systems affected" immediately made me think of the following: There must be around 850 million Windows systems in use. A large proportion of these are used by private users or small companies.

Only larger companies will use an EDR security solution like CrowdStrike Falcon. This puts the "less than 1%" into perspective – the majority of CrowdStrike customers were probably affected.

Consequences of the outage

I had mentioned in various blog posts that a wide range of companies and organizations were affected. HackManac has listed some of the affected entities worldwide in a tweet on X.

The following tweet from Flight Radar about the flight movements over the USA shows a good picture of the extent of the impact. Simply click on the image to start the simulation.

It turns out that air traffic comes to a standstill for hours because no more planes can be handled.

Microsoft says: "EU forced us"

Meanwhile, the fallout from the flawed CrowdStrike update for Windows, which corrupted systems around the world, is once again fueling concerns about Microsoft's monopoly on government and corporate IT systems. The Guardian, for example, has published an article on the subject.

Microsoft has since pointed out that the EU forced the company to open its internal interfaces to third-party providers in 2009 – in other words, that it was effectively to blame for the debacle. Neowin.net, for example, points out the issue here (the Wall Street Journal has published an article, but I am not linking to it because of a paywall). A spokesman is said to have told the WSJ that an agreement with the European Commission from 2009 is the reason why Microsoft cannot secure its operating system more strongly.

Following a complaint, the spokesperson said, Microsoft agreed with the European Commission in 2009 to grant the manufacturers of security software the same access to Windows as Microsoft. This decision means that security software providers have more opportunities to manipulate systems

Let's sort this out a bit – it's the game of "pointing the finger at others – it's their fault". The agreement with the EU Commission had a competition law background, because Microsoft slowed down and hindered the competition by not disclosing vulnerabilities. The EU forced the opening for third-party users – but the implementation of the APIs in question is probably so lousy that the debacle has now occurred.

Nobody forced Microsoft to implement it this way and not to put it on a "more stable footing" since then. Just a reminder: Since 2009, we have seen Windows 8, Windows 8.1, Windows 10 and now Windows 11 in various feature update variants. But Microsoft is apparently continuing to work with old concepts that would have fallen on Redmond's feet in exactly the same way in the event of a supply chain attack.

Microsoft CrowdStrike Recovery Tool

In the meantime, Microsoft has published a CrowdStrike recovery tool. This is intended to make it easier for administrators to recover machines that are stuck in a blue screen loop. The following tweet refers to the relevant Techcommunity article with further information.

And John Hammond points out in the following tweet that someone has written a PowerShell script to extract BitLocker keys from an NTDS.dit file. This could be useful if the domain controller is damaged by the CrowdStrike disaster, but WinRE requests the BitLocker recovery key on boot.

Important information: Cyber criminals are taking advantage of the chaos and setting up CrowdStrike websites offering fake repair tools. The warning from the BSI and other organizations is: Only use official information and tools from CrowdStrike and Microsoft.

Similar articles:
Worldwide outage of Microsoft 365 (July 19, 2024)
Windows systems throw BSOD due to faulty CrowdStrike update
Why numerous IT systems around the world failed due to two errors on July 19, 2024
CrowdStrike analysis: Why an empty file led to BlueSceen
Review of the CrowdStrike incident, the biggest computer glitch of all time