Statement on Falcon content update for Windows hosts

Updated at 9:13 p.m. ET, July 19, 2024

CrowdStrike is actively working with customers affected by a flaw discovered in a single content update for Windows hosts. Mac and Linux hosts were not affected. This was not a cyberattack.

The issue has been identified, isolated, and a fix has been implemented. We are referring customers to the support portal for the latest updates and will continue to provide ongoing, comprehensive public updates on our blog.

Additionally, we recommend that organizations ensure they communicate with CrowdStrike representatives through official channels.

Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.

We understand the severity of the situation and deeply regret the inconvenience and disruption. We are working with all affected customers to ensure that systems are back up and running and able to provide the services their customers count on.

We assure our customers that CrowdStrike is operating normally and that this issue does not affect our Falcon platform systems. If your systems are operating normally, there will be no impact on your protection if the Falcon sensor is installed.

Below is the latest CrowdStrike technical alert with more information about the issue and remediation steps organizations can take. We will continue to provide updates to our community and the industry as they become available.

Summary

Details

Symptoms include hosts experiencing a bugcheck error or blue screen related to the Falcon sensor.
Unaffected Windows hosts do not require any action as the problematic channel file has been reverted.
Windows hosts that come online after 0527 UTC will also not be affected.
This issue does not affect Mac or Linux-based hosts.
The channel file “C-00000291*.sys” with timestamp 0527 UTC or later is the reverted (good) version.
The channel file “C-00000291*.sys” with timestamp 0409 UTC is the problematic version.

Note: It is normal to have multiple “C-00000291*.sys” files in the CrowdStrike directory, provided that one of the files in the folder have a timestamp of 0527 UTC or later, that will be the active content.

Current action

CrowdStrike Engineering has identified a content implementation related to this issue and reverted those changes.
If hosts continue to fail and cannot stay online to receive changes to the channel file, the workaround steps below can be used.
We assure our clients that CrowdStrike is operating normally and this issue does not affect our Falcon platform systems.If your systems are operating normally, your protection will not be affected by installing the Falcon sensor. Falcon Complete and OverWatch services are not affected by this incident.

Query to identify affected hosts through advanced event search

Please see this Knowledge Base article: How to identify hosts potentially affected by Windows crashes (PDF) or log in to view it on the support portal.

Panel

Similar to the query mentioned above, a dashboard is now available showing the affected channels and CIDs and affected sensors. Depending on your subscriptions, it is available in the console menu at:

Next-Generation SIEM > Dashboard or;
Research > Dashboards
Named as: hosts_possibly_impacted_by_windows_failures

Note: The Dashboard cannot be used with the “Live” button

Articles on automated recovery:

Please see this article: Automatic Blue Screen Recovery on Windows Instances on GCP (PDF) or sign in to view it on the support portal.

Workaround steps for individual hosts:

Reboot the host to give it a chance to download the reverse channel file. We strongly recommend connecting the host to a wired network (rather than WiFi) before rebooting, as the host will gain Internet connectivity considerably faster via Ethernet.
If the host fails again, then:

Boot Windows in Safe Mode or Windows Recovery Environment

NOTE: Placing the host on a wired network (instead of WiFi) and using Safe Mode with Networking may help with the resolution.

Go to the directory %WINDIR%\System32\drivers\CrowdStrike

Windows Recovery defaults to X:\windows\system32

First, navigate to the appropriate partition (default is C:\) and navigate to the crowdstrike directory:

C:
cd windows\system32\drivers\crowdstrike

Note: In WinRE/WinPE, navigate to the Windows\System32\drivers\CrowdStrike directory on the operating system volume

Locate the file that matches “C-00000291*.sys” and delete it.

No delete or change any other files or folders

Cold boot of the host

Turn off the host.
Boot the host from the powered off state.

Note: BitLocker-encrypted hosts may require a recovery key.

Workaround steps for public cloud or similar environment, including virtual:

Option 1:

Detach the operating system disk volume from the affected virtual server
Create a snapshot or backup of the disk volume before proceeding as a precaution against unwanted changes.
Attach/mount volume to new virtual server
Go to the directory %WINDIR%\System32\drivers\CrowdStrike
Locate the file that matches “C-00000291*.sys” and delete it.
Disconnect the volume from the new virtual server
Reconnect the fixed volume to the affected virtual server

Option 2:

Return to a snapshot prior to 0409 UTC.

AWS Specific Documentation:

Azure Environments:

User Access Recovery Key in the Workspace ONE Portal

When this setting is enabled, users can retrieve the BitLocker recovery key from the Workspace ONE portal without having to contact the HelpDesk for assistance. To activate the recovery key in the Workspace ONE portal, follow the steps below. See this Omnissa article for more information.