[German]A faulty update to the CrowdStrike Falcon software brought around 8.5 million Windows computers to a standstill on July 19, 2024. The incident is already considered to be the world's largest computer failure to date and is likely to have caused billions in damage. Since Friday, administrators have been struggling to repair the failed systems and "clean-up operations" are underway. A blog reader contacted me at the weekend because, as an administrator, he had noticed some side effects that had not yet been mentioned publicly. Even after the repair of the BlueScreen behavior, Falcon Sensor agents may no longer be able to work. I'll put together some information in the blog post.
Advertising
The CrowdStrike incident
On July 19, 2024, there were numerous disruptions to IT systems running Windows around the world. Operations at airports came to a standstill, banks could no longer work, trains were canceled and companies sent their employees home (e.g. Tegut) because the IT systems no longer worked. German municipalities, companies and banks were also affected. I reported on this in the article Windows systems throw BSOD due to faulty CrowdStrike update.
However, it was not a cyberattack, but a faulty update for an EDR security solution from the US provider CrowdStrike, which caused Windows systems to crash with a blue screen. An initial analysis of what might have happened can be found in my blog post wdStrike analysis: Why an empty file led to BlueSceen.
In that tweet, someone cites that the flawed update was intended to target newly observed malicious "named pipes" used by common C2 frameworks in cyberattacks. I reported on the incident here in the blog promptly and followed it up in various posts (see links at the end of the article).
New CrowdStrike problems/side effects?
A blog reader contacted me by e-mail because he is also responsible for CrowdStrike as an administrator and was affected by the incident. He noticed new problems or side effects in the CrowdStrike case. The reader wrote to me that the company had a large part of its landscape back in operation.
However, when he looked at the company's servers in the CrowStrike console, he realized that about 10% of the servers have not reported since Friday (although the servers themselves are running). If the CrowdStrike icon was activated in the CrowStrike console, the icon shows this error.
Advertising
Die Konsole meldet, dass der Treiber gestoppt wurde, obwohl der Dienst läuft. Sprich: Die CrowdStrike Falcon Sensoren liefern keine Rückmeldungen an die Konsole.
Agent funktioniert nicht mehr
Nach der Aktualisierung der Version funktioniert er wieder, aber was ist die Ursache? Der Leser hat daraufhin recherchiert und ist dem Problem auf den Grund gegangen. Der CrowdStrike-Support berichtet Folgendes.
File not found: %SYSTEMROOT%\system32\drivers\CrowdStrike\csagent.sys
it seems that CSAgent.sys has been renamed to CSAgent.sys.old:
CSAgent.sys.old 09.07.2024 14:50:46 7.15.18513.0
The reason why an upgrade solved it, it is probably because that also triggered a repair and renamed CSAgent.sys.old to its original name.
Current suggestion is to check the the impacted Channel File has been removed from the System and then rename the CSAgent.sys.old back to its original name CSAgent.sys and reboot the host.
This means that even if the BlueScreen (BSOD) has been repaired under Windows due to the faulty update, the Falcon Sensor Agent may not be able to run. However, affected servers can be easily filtered in the CrowdStrike console using the "last seen" query. Servers should report regularly due to their continuous operation.
Trouble with clients
The reader also wrote that the situation is different for clients, as they are online irregularly and therefore do not report regularly. He asked the question: "Who wants to find out which client has been repaired but is not reporting?" This is rather difficult.
If the above-mentioned icon was activated, users could check for themselves whether the Falcon software was working on the client. In his email, the reader also noted that he only started the client repair yesterday, Monday, and wants to include this as a test step. Thanks to the reader for the information – anyone else with this observation?
Similar articles:
Worldwide outage of Microsoft 365 (July 19, 2024)
Windows systems throw BSOD due to faulty CrowdStrike update
Why numerous IT systems around the world failed due to two errors on July 19, 2024
CrowdStrike analysis: Why an empty file led to BlueSceen
Review of the CrowdStrike incident, the biggest computer glitch of all time
CrowdStrike incident: sensor failure as a previously unknown side effect?
Advertising
