How Action1 Preempts the Risk of IT Outages

On July 18, 2024, an incident occurred that the media dubbed “the largest IT outage in history.” CrowdStrike’s XDR product, Falcon, running on Windows, received a botched update that caused the Windows OS to crash and display the Blue Screen of Death (BSOD). The tedious recovery process required physical access to each affected system to repair. This incident turned into a global outbreak, affecting 8.5 million Windows machines worldwide and causing critical disruptions and downtime in numerous organizations, including banks, hospitals and airports.

This situation understandably raises many questions. It is highly unusual for a trusted cybersecurity company like CrowdStrike, with a long history of quality product development, to allow a flawed update to be rapidly rolled out worldwide. Such a company is expected to have a robust development process that includes rigorous testing in a staging environment before pushing updates to production. Phased rollouts to different customer segments should have also prevented a widespread outbreak. It is hard to imagine conditions where all best practices and safeguards for testing and validating updates were ignored—an investigation will eventually reveal the cause.

Nonetheless, we understand that this incident may raise concerns among some customers regarding the reliability of using Action1 as an agent-based patch management solution. In this post, we want to explain what Action1 does to preempt the risk of such outages within our global customer base.

At Action1, we have a multi-layered approach to how we develop, test, and validate updates before release:

• Non-intrusive design: The Action1 agent is designed to ensure it cannot physically “take down” an endpoint. Unlike some agent-based solutions, our agent does not have any kernel-mode components, which eliminates the risk of causing a BSOD. In the worst-case scenario, patching stops working while the rest of the system operates as usual.
• Rigorous testing: Our development process includes thorough unit, integration, and system testing, including stress and load testing. We utilize automated testing pipelines to ensure reliable quality checks and reduce any possibility of human error.
• Update-then-run: The agent always checks for an updated version of itself before trying to execute after a restart. This eliminates the possibility of any manual repair work even in the unlikely event of Action1 releasing a broken agent update.
• Phased rollout: The deployment process is always carried out in a staged approach, as we roll out new agent versions to multiple separate cloud instances. Specifically, we release updates to the agent in batches over multiple days, constantly analyzing their behavior. This gradual approach allows us to quickly “step on the break” before it affects too many systems.
• Regional isolation: We have distributed data centers globally, which not only ensures compliance with data privacy and residency requirements but also adds additional granularity to the phased update deployment process.

By maintaining rigorous development and testing processes and deployment in a controlled, phased manner, we ensure that Action1 remains a reliable and outage-proof agent-based patch management solution.

We believe this proactive statement will help reassure our customers and highlight our commitment to IT security and reliability.

We also believe that the IT community worldwide will not go down the path of postponing update deployment for operating systems and applications due to stability concerns and fear of new widespread IT outages. Instead, we hope the much-needed reduction of Mean-Time-to-Remediation (MTTR) will gradually happen due to patch management automation. This is the only way of staying ahead of threat actors exploiting known vulnerabilities at an ever-accelerating pace. Action1’s patch management solution is here to help you with this arduous task.

About Action1

Action1 reinvents patch management with an infinitely scalable, highly secure, cloud-native platform configurable in 5 minutes—and it just works, with no VPN needed. Featuring unified OS and third-party patching with peer-to-peer patch distribution and integrated real-time vulnerability assessment, it enables autonomous patch compliance that preempts ransomware and security risks, all while eliminating costly routine labor. Trusted by thousands of enterprises managing millions of endpoints globally, Action1 is certified for SOC 2 and ISO 27001.