On July 19, 2024, CrowdStrike experienced a significant issue that caused a global IT outage, affecting many of its customers. This incident is a stark reminder of the vulnerabilities inherent in our increasing reliance on SaaS (Software as a Service) offerings and public cloud infrastructures. Here’s a detailed overview of the incident, its impact, the technical steps taken for resolution, and the lessons learned.

The Incident: What Happened?

The outage was triggered by a defective content update for CrowdStrike’s Falcon sensor on Windows hosts. This faulty update led to widespread system failures, with many Windows systems experiencing “blue screen of death” (BSOD) errors, rendering devices inoperable. The issue primarily affected Windows hosts running Falcon sensor versions 7.15 and 7.16.

Root Cause Analysis

CrowdStrike’s CEO, George Kurtz, clarified that the incident was not a result of a cyberattack but a software update problem. The defective update caused a logic error, leading to system crashes. The company quickly identified, isolated, and deployed a fix, advising users to boot Windows in safe mode or the Windows Recovery Environment and delete a specific file as a workaround.

Technical Steps and Debugging Process

1. Initial Identification:
   • Defective Update Detection: CrowdStrike detected that the faulty content update for Falcon sensor versions 7.15 and 7.16 caused system crashes.
2. Immediate Response:
   • Reverting the Update: The problematic update file “C-00000291*.sys” was identified and reverted.
   • Customer Advisory: Users were advised to boot Windows in Safe Mode or the Windows Recovery Environment.
3. Manual Intervention Steps:
   • Booting in Safe Mode:
     1. Reboot the Windows system and press F8 or Shift+F8 during startup to enter Safe Mode.
   • Navigating to the Falcon Sensor Directory: 2. Open Windows Explorer and navigate to C:\Windows\System32\drivers\CrowdStrike.
   • Renaming the Faulty File: 3. Locate the file C-00000291-00000000-00000032.sys, right-click and rename it to C-00000291-00000000-00000032.renamed.
   • Restarting the System: 4. Reboot the system normally.
4. For Virtual and Cloud Environments:
   • Detach the Operating System Disk Volume:
     1. Detach the OS disk volume from the impacted virtual server.
     2. Create a snapshot or backup of the disk volume as a precaution.
   • Attach to a New Virtual Server: 3. Attach the volume to a new virtual server. 4. Navigate to C:\Windows\System32\drivers\CrowdStrike.
   • Deleting the Faulty File: 5. Locate the file matching C-00000291*.sys and delete it.
   • Reattaching the Fixed Volume: 6. Detach the volume from the new virtual server and reattach it to the impacted virtual server.
   • Rolling Back to a Snapshot: 7. Roll back to a snapshot taken before 0409 UTC.
5. Azure via Serial Console:
   • Accessing Serial Console:
     1. Log in to Azure Console, select the VM, and access “Serial Console”.
     2. Type cmd and press enter, then type ch -si 1.
   • Booting in Safe Mode: 3. Enter bcdedit /set {current} safeboot minimal. 4. Enter bcdedit /set {current} safeboot network.
   • Restarting VM: 5. Restart the VM and confirm the boot state by running wmic COMPUTERSYSTEM GET BootupState.

Impact of the Outage

The CrowdStrike outage profoundly impacted various sectors, illustrating the interconnectedness and vulnerability of modern digital infrastructure.

Aviation Sector:

• Airports worldwide experienced chaos, with over a thousand flights canceled in the United States alone.
• Major airlines like American Airlines, Delta Airlines, and United Airlines halted operations.
• Swiss air traffic decreased by 30%, KLM suspended most flights, and Berlin Airport was paralyzed.

Retail Sector:

• Australian supermarkets had to close due to checkout system failures.

Events and Hospitality:

• Paris Olympics organizers resorted to manual security checks due to accreditation system failures.
• Disneyland Paris also experienced disruptions.

Healthcare Sector:

• Hospital operations were significantly disrupted, with many reporting difficulties in scheduling and patient management systems.
• Electronic Health Records (EHR) systems were affected, impacting access to medical records.
• Emergency services in some U.S. states experienced issues with 911 call centers.
• Hospitals reverted to manual processes, slowing down operations.

The Double-Edged Sword of Agility and Convenience

SaaS and public cloud solutions have revolutionized businesses’ operations, providing unparalleled flexibility and scalability. Companies can quickly deploy new services, scale operations, and access cutting-edge technology without substantial capital investment. However, this convenience comes at a cost. The recent CrowdStrike disruption highlights a crucial vulnerability: when a single point of failure occurs, it can cascade across numerous organizations, leading to widespread operational paralysis.

Building Resilience in a Cloud-Dependent World

Diversification of Providers:

• Avoid placing all eggs in one basket. By diversifying cloud and SaaS providers, businesses can mitigate the risk of a single point of failure.

Hybrid and Multi-Cloud Solutions:

• Implement a hybrid multi-cloud approach that combines various cloud services with on-premises infrastructure. This strategy provides a safety net in case of cloud service disruptions and enhances operational flexibility.

Enhanced Security Measures for Hybrid Environments:

• Integrate advanced security protocols across both cloud and on-premises systems. This includes utilizing encryption, multi-factor authentication, and continuous monitoring to safeguard against vulnerabilities, even when security providers face issues.

Comprehensive Risk Management:

• Develop risk management frameworks that account for potential disruptions in cloud services. This includes identifying critical assets, evaluating the impact of potential threats, and establishing contingency plans for maintaining operations during service outages.

Resilient IT Operations:

• Design IT operations to be robust and adaptable, leveraging automation and orchestration tools to switch between cloud quickly and on-premises resources in response to disruptions. This ensures continuous service availability and minimizes downtime.

Conclusion

The CrowdStrike update disruption is a crucial lesson in the risks associated with overreliance on SaaS and public cloud solutions. While these technologies offer significant benefits, they also introduce vulnerabilities that can have widespread implications. It is imperative for organizations to critically evaluate their dependence on these services and implement strategies to build resilience and safeguard their operations and data. By prioritizing these measures, businesses can better safeguard against disruptions and ensure seamless operations in an increasingly digital landscape.

Team Prime is always at your support with specialized incident responders and research team help to stay ahead of the Proactive Defence journey in a complex digital space.