Windows CrowdStrike BSOD Incident: Steps for Recovery and Mitigation

Posted on by Dhiren Bhardwaj
Spread The Knowledge 😌
        
Post Views: 12

On July 19, 2024, many organizations in the IT world encountered a Blue Screen of Death (BSOD) issue. While it was a work-free Friday for many folks, causing some initial excitement, this incident severely impacted airlines, banks, stock markets, and other businesses across the globe. Let’s dive deep to understand what happened, why it happened, and how we can stop it.

Understanding CrowdStrike

CrowdStrike’s Falcon platform offers endpoint detection and response (EDR) capabilities, meaning it can detect threats and take immediate action to stop them. Unlike traditional systems that merely alert users, Falcon can actively intervene.

What Happened: Background

According to CrowdStrike’s official site:

“On July 19, 2024, at 04:09 UTC, as part of ongoing operations, CrowdStrike released a sensor configuration update to Windows systems. Sensor configuration updates are an ongoing part of the protection mechanisms of the Falcon platform. This configuration update triggered a logic error resulting in a system crash and blue screen (BSOD) on impacted systems.”
The sensor configuration update that caused the system crash was remediated on Friday, July 19, 2024, at 05:27 UTC.

Observations

Microsoft:

As an OEM, Microsoft is responsible for the overall system’s stability and integration of other apps.

CrowdStrike:

As a vendor, CrowdStrike is responsible for the functionality and compatibility of released updates.

Mitigation Steps

How to Regain Access in Windows PCs

1. Boot into Safe Mode or Windows Recovery Environment:

  • Restart the computer and press the F8 key before Windows loads.
  • Select Safe Mode or Windows Recovery Environment from the menu.

2. Navigate to the CrowdStrike Directory:

  • Go to `C:\Windows\System32\drivers\CrowdStrike`.

3. Delete the Specific Driver File:

  • Locate and delete the file that matches `C-00000291*.sys`.

4. Reboot Normally:

  • Restart the computer normally.

Mitigation for Cloud Resources

AWS

Step 1: Log in to AWS Management Console:

  • Open your web browser and navigate to the AWS Management Console.
  • Enter your credentials to log in.

Navigate to the EC2 Dashboard:

  • In the console, go to Services and select EC2 under the Compute category.
  • Select the impacted instance in the EC2 Dashboard.

Detach the EBS Volume:

  • Click on the Description tab to see the instance details.
  • Under Block devices, click on the EBS volume ID.
  • Click on Actions and select Detach Volume. Confirm the detachment.

Step 2: Attach the EBS Volume to a New EC2 Instance:

  • Select or launch a new EC2 instance.
  • Go to the Elastic Block Store section, click on Volumes.
  • Select the detached volume, click on Actions, and choose Attach Volume. Specify a device name (e.g., /dev/sdf) and click Attach.

Step 3:  Fix the CrowdStrike Driver Folder:

  • Access the new instance via SSH or Remote Desktop.
  • If using a Linux instance, mount the volume:
  • Navigate to the CrowdStrike Directory:
  • Delete the problematic driver file:
  • Unmount the volume:

Step 4: Detach the EBS Volume from the New EC2 Instance:

  • In the AWS Management Console, go back to the EC2 Dashboard.
  • Detach the volume from the new instance.

Step 5: Attach the EBS Volume Back to the Impacted EC2 Instance:

  • Attach the fixed volume to the impacted instance.
  • Specify the original device name (e.g., /dev/sda1) and click Attach.
  • Reboot the impacted instance.

Azure

Using the Azure Portal:

Attempt ‘Restart’ on affected VMs.

Using the Azure CLI or Azure Shell:

Follow the [Azure CLI documentation](https://learn.microsoft.com/en-us/cli/azure/vm?view=azure-cli-latest#az-vm-res).

Additional Options for Recovery:

Option 1:
  • Restore from a backup preferably from before July 19, 2024, at 04:09 UTC.
Option 2:
  • Attempt to remove the `C-00000291*.sys` file directly:
Option 3:
  • Troubleshoot a Windows VM by attaching the OS disk to a repair VM through the Azure portal.
  • Delete the problematic file: `Windows/System32/Drivers/CrowdStrike/C-00000291*.sys`.

CrowdStrike has confirmed that the affected update has been pulled. Customers experiencing issues should reach out to CrowdStrike for additional assistance. We are continuing to investigate further mitigation options and will share more information as it becomes available.

This comprehensive guide should help mitigate the issues caused by the recent CrowdStrike BSOD incident and ensure system stability.

 
       

Leave a Reply