Updated 24 July
Crowdstrike has released a Preliminary Post Incident Review (PIR): Content Configuration Update Impacting the Falcon Sensor and the Windows Operating System (BSOD)
What Happened?
On Friday, July 19, 2024 at 04:09 UTC, as part of regular operations, CrowdStrike released a content configuration update for the Windows sensor to gather telemetry on possible novel threat techniques.
These updates are a regular part of the dynamic protection mechanisms of the Falcon platform. The problematic Rapid Response Content configuration update resulted in a Windows system crash.
Systems in scope include Windows hosts running sensor version 7.11 and above that were online between Friday, July 19, 2024 04:09 UTC and Friday, July 19, 2024 05:27 UTC and received the update. Mac and Linux hosts were not impacted.
The defect in the content update was reverted on Friday, July 19, 2024 at 05:27 UTC. Systems coming online after this time, or that did not connect during the window, were not impacted.
What Went Wrong and Why?
CrowdStrike delivers security content configuration updates to our sensors in two ways: Sensor Content that is shipped with our sensor directly, and Rapid Response Content that is designed to respond to the changing threat landscape at operational speed.
The issue on Friday involved a Rapid Response Content update with an undetected error.
Read the full release here
Updated 22 July
David Weston – Vice President, Enterprise and OS Security for Microsoft released a statement outlining the steps taken with CrowdStrike and others to remediate and support customers. Steps taken have included:
– Engaging with CrowdStrike to automate their work on developing a solution. CrowdStrike has recommended a workaround to address this issue and has also issued a public statement. Instructions to remedy the situation on Windows endpoints were posted on the Windows Message Center.
– Deploying hundreds of Microsoft engineers and experts to work directly with customers to restore services.
– Collaborating with other cloud providers and stakeholders, including Google Cloud Platform (GCP) and Amazon Web Services (AWS), to share awareness on the state of impact we are each seeing across the industry and inform ongoing conversations with CrowdStrike and customers.
– Quickly posting manual remediation documentation and scripts found here.
– Keeping customers informed of the latest status on the incident through the Azure Status Dashboard here.
Additionally, CrowdStrike helped Microsoft develop a scalable solution that helped Microsoft’s Azure infrastructure accelerate a fix for CrowdStrike’s faulty update. Microsoft also worked with both AWS and GCP to collaborate on the most effective approaches.
Weston said, “While software updates may occasionally cause disturbances, significant incidents like the CrowdStrike event are infrequent. We currently estimate that CrowdStrike’s update affected 8.5 million Windows devices, or less than one percent of all Windows machines. While the percentage was small, the broad economic and societal impacts reflect the use of CrowdStrike by enterprises that run many critical services.
This incident demonstrates the interconnected nature of our broad ecosystem — global cloud providers, software platforms, security vendors and other software vendors, and customers. It’s also a reminder of how important it is for all of us across the tech ecosystem to prioritize operating with safe deployment and disaster recovery using the mechanisms that exist. As we’ve seen over the last two days, we learn, recover and move forward most effectively when we collaborate and work together. We appreciate the cooperation and collaboration of our entire sector, and we will continue to update with learnings and next steps.”
19 July
Following a widespread IT outage reported to impact critical industry sectors, including media, airlines, transport providers and retail outlets, coincidently timed on a Friday afternoon (Australian Eastern standard time), CrowdStrike reports it is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. The update clarifies, “This is not a security incident or cyberattack.”
“The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.
We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels.
Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.”
A Microsoft spokesperson told MySecurity Media, “We’re aware of an issue affecting Windows devices due to an update from a third-party software platform. We anticipate a resolution is forthcoming.”
In a social media post, Australian Home Affairs and Minister for Cyber Security The Hon Clare O’Neil confirmed, “The Australian Government has conducted a National Coordination Mechanism meeting, which I joined earlier this evening. CrowdStrike attended the meeting and we can confirm there is no evidence that this is a cyber-security incident.”
An update from Deputy Secretary, Hamish Hansford, following this evening’s National Coordination Mechanism meeting relating to the CrowdStrike technical incident. pic.twitter.com/WkUh2UtHIL
— Cyber and Infrastructure Security Centre (@CISC_AU) July 19, 2024
Omdia Senior Director, Cybersecurity Maxine Holt stated, “Omdia’s Cloud and Data Center analysts have long warned about over-reliance on cloud services. Today’s outages will make enterprises rethink moving mission-critical applications off-premises. The ripple effect is massive, hitting CrowdStrike, Microsoft, AWS, Azure, Google, and beyond. CrowdStrike’s shares have plummeted by more than 20% in unofficial pre-market trading in the US, translating to a staggering $16 billion loss in value.
Looking forward, there’s a shift towards consolidating security tools into integrated platforms. However, as one CISO starkly put it, “Consolidating with fewer vendors means that any issue has a huge operational impact. Businesses must demand rigorous testing and transparency from their vendors.
CrowdStrike’s testing procedures will undoubtedly be scrutinized in the aftermath. For now, the outages continue to rise, and the tech world watches as the fallout unfolds.”
In a ‘critical’ Australian Cyber Security Centre Alert organisations or individuals that have been impacted or require assistance can contact 1300 CYBER1 (1300 292 371) and access alerts at https://www.cyber.gov.au
