CrowdStrike has released a Root Cause Analysis (RCA) report detailing what caused the July 19, 2024, systems crash that caused global outages.
As part of regular operations, CrowdStrike released a content configuration update (via channel files) for the Windows sensor that caused the system crash.
“As of 8:00 p.m. EDT on July 29, 2024, around 99% of Windows sensors were back online, compared to before the content update and using a week-over-week comparison,” said CrowdStrike Founder and CEO George Kurtz. “To any customers still affected, please know we will not rest until all systems are restored.”
In February 2024, CrowdStrike introduced a new sensor capability to enable visibility into possible novel attack techniques that may abuse certain Windows mechanisms. This capability pre-defined a set of fields for Rapid Response Content to gather data. As outlined in the RCA, this new sensor capability was developed and tested according to CrowdStrike’s standard software development processes.
On March 5, 2024, following a successful stress test, the first Rapid Response Content for Channel File 291 was released to production as part of a content configuration update, with three additional Rapid Response updates deployed between April 8, 2024 and April 24, 2024. These performed as expected in production.
On July 19, 2024, a Rapid Response Content update was delivered to certain Windows hosts, evolving the new capability first released in February 2024. The sensor expected 20 input fields, while the update provided 21 input fields. In this instance, the mismatch resulted in an out-of-bounds memory read, causing a system crash. CrowdStrike’s analysis, together with a third-party review, confirmed this bug was not exploitable by a threat actor.
While this scenario with Channel File 291 is now incapable of recurring, it informs the process improvements and mitigation steps that CrowdStrike is deploying to ensure further enhanced resilience.
“I want to express my sincere gratitude for the incredible round-the-clock efforts of our customers and partners who, working alongside our teams, mobilized immediately to restore systems and bring many back online within hours,” added Kurtz.
