While the impact on military organizations, and the defense sector is still unknown, Friday’s incident was a wakeup call on how a broad internet outage can impact the entirety of society, and how it could severely compromise national security, when military networks fall victim to such an incident.
The Incident Unveiled
On Friday, July 29th, 2024 CrowdStrike, a major cybersecurity firm, which also provides security for military networks, rolled out an update to its “Falcon” product. “Falcon” is known as an endpoint detection and response (EDR) solution. An EDR is a cybersecurity technology focussed on identifying, and (autonomously) responding to potential threats on endpoints such as computers, servers, and mobile devices.
The faulty update rolled out led to computers, and servers running Microsoft Windows going into an infinite reboot loop, rendering them unresponsive, and unusable.
While quickly identified, and addressed by CrowdStrike, the disruption had widespread consequences on critical infrastructure like aviation, and public safety, bringing various businesses to a grinding halt.
Assessing the Damage
According to their website, CrowdStrike Falcon is widely deployed to protect military networks, as well, achieving IL5 authorization in May 2023 to secure DoD unclassified National Security Systems (NSS), and therefore being allowed to deploy their Falcon software to protect systems mentioned above.
While the incident did not result in data breaches, and the leaking of sensitive data, a compromise of military readiness is a realistic scenario considering that a majority of the world’s militaries run Microsoft Windows-based computers, and servers, which were downed by this faulty update.
Since the CrowdStrike Falcon product is a unified, cloud-security solution the update was centrally, and proactively pushed by the company, and without the involvement of system administrators. Therefore, the administrators did not stand a chance to intervene or decide whether or not to deploy the update or delay its rollout.
While automatic updating has become quite common, mostly for convenience purposes, this incident should be treated as a lesson learned to only rely on software, in which updates/upgrades are deployed (semi-)manually to ensure the military/defense readiness of a nation.
Strategic Implications for the Defense Sector
With the results of this incident trust in software, which is auto-updatable should be withdrawn from such vendors. A return to manually deployed software updates/upgrades, which can be as secure when stringent, fail-proof rollout protocols are in place, as I have repeatedly proven in cases with government organizations, and militaries should be the only logical conclusion.
Having the armed forces of a nation down due to a software engineer not having had the appropriate amount of sleep, or coffee, or whatever else is totally unacceptable!
However, you also have to keep in mind that the parts of the defense sector rely on propriety vendors to ensure the security, and availability of their software solutions. Therefore products, and services they offer to militaries, and government organizations might also be compromised, risking national security.
Technological and Operational Adaptations
I think we have all been observing the increased focus on artificial intelligence in the past couple of years and explored (theoretical) opportunities on where AI, machine learning, etc. can enhance cybersecurity efforts, and national security neglecting the fact that a majority of today’s available solutions are cloud-based, and therefore prone to the same type of outage CrowdStrike’s update presented us with.
One of the many possible reactions could of course be the implementation of a zero-trust policy, which is a security model based on the principle of maintaining the strictest access controls, and not trusting anyone by default, even those already inside the network. Yet even such a strict protocol would have not prevented Friday’s incident, because CrowdStrike’s Falcon has “kernel level access”, so access to the “heart” of the operating system to prevent the said kernel from being exploited.
It is therefore arguable that very advanced incident response plans have to be created to ensure the re-establishment of defense readiness happens as swiftly as possible, leaving the only detrimental fact that the defense readiness of a nation was disturbed in the first place, and could have been taken advantage of by an adversary.
We therefore inevitably return to the human factor…. You know, these pesky creatures who fall ill, and want vacation days, etc. we’re so desperately trying to replace with machine counterparts? Because the machines are less prone to making mistakes? Oh, never mind!
The fact is that a well-trained team of cybersecurity professionals, and a very aware team of talented personnel would have run the update in a contained environment before releasing it into the wild, and this incident would have never happened, and we would have all had a nice summer weekend. Call it a second line of defense, since the proprietary engineers did such a good job at it.
The Bigger Picture: Geopolitical and Economic Repercussions
Let’s leave the national security aspect alone for a moment, and speak about the economic impact… The way it appears some 60 hours later the aviation industry was hit hardest, with hundreds of flights canceled/delayed, and entire airports being shut down.
Banks, like aviation, belong to the “critical infrastructure” segment and were also among those severely affected. Probably worth mentioning were emergency dispatch systems.
There are no numbers in yet as to the actual cost of this incident, yet I’m betting there are insurance professionals out there who didn’t get the best sleep over the weekend.
Also worth mentioning in a geopolitical context are the elaborate network infiltrations like the Bangladesh Bank hack, and that they may be state-sponsored. And what this incident has shown to any hacker in the world is that you can stop the globe from spinning by infiltrating third-party software providers, and screwing up their software… a very worrying outlook.
It’s important that you put your preconceived friend/foe notions aside for a while, and start collaborating better on a global scale with other nations, and exchange threat intelligence to put yourself into a situation in which your defense apparatus can prevent such incidents from happening in the first place.
Looking Forward: Building a Resilient Defense Cybersecurity Posture
The key takeaway is that auto-updates suck. And that auto-updates issued by a propriety vendor without you being able to control suck even more.
If you’re so susceptible to convenience, at least make sure that you have appropriate crisis-reaction protocols in place to have your organization business-ready as quickly as humanly possible.
Then again, I’m talking to defense professionals, and you should entirely overthink your choices in terms of hard- and software, replacing products and/or services, which take away your possibility to intervene with auto-deployment updates… for the sake of military as end-users, yet also for the defense industry offering SaaS products to improve the mission-readiness of their clients, a downed system can cripple the entire national security.
Any future investment into cybersecurity efforts, infrastructure resilience, etc. should keep these lessons learned in mind.
Conclusion
In an ideal world, this incident would have been avoidable. But this ain’t Canada… or Bhutan, for that matter.
We have allowed software to propagate into the most sensitive environments, and justifiably so, to make our lives easier, and more convenient. Unfortunately, we have often done so without proper risk assessment.
Ever since the advent of “as a Service” products I have warned against relying entirely on proprietary cloud vendors, thereby forfeiting autonomy for convenience, and begged the people I work with not to listen to the promises of software vendors ensuring us with nothing more than “what could possibly go wrong” arguments.
A large group of people I’ve worked with has considered my stringent concerns, an exorbitantly larger crowd criticized my (partially) alarmist views.
Now, shall we talk? Call +49 173 5433491 or write me an email at lars.hilse@gmail.com (PGP 17FFC660)
60 Second Bio on the Author
Lars G. A. Hilse (*1979) is an independent political/corporate advisor, and expert/witness in information security with a focus on the risks of cyber terrorism, and cyber warfare, and their impact on the security of critical, national/global infrastructure with degrees in engineering, and finance.
He maintains close cooperations with law enforcement, and the intelligence community, and actively contributes to some of the most difficult, and notable cybercrime investigations. He acts as a cyber-crisis-manager, and through his vast network is able to deploy highly specialised cyber-incident-response teams globally at breathtaking speed.
In his capacity as an information security consultant he performs state-of-the-art risk assessments & mitigation directives, crisis response protocols, and establishes cyber security maturity models.
Among his other talents are intelligence gathering, accessing fortified infrastructure/networks, forensics, all with a focus on social engineering and other unorthodox access methods.
Since his first exposure to the internet at age 13, Hilse has constantly broadened his skillset in cybersecurity, focussing on cyber crime, cyber terrorism, and cyber defence.
Due to his precise foresight of all developments in the digital realm for over 25 years, he is repeatedly referred to as a global thought leader in cyber security, and digital strategy. Hilse has privately funded research in cybersecurity worth over USD $1.000.000 since 2011. Some of the results hereof were the Advanced Cybersecurity Risk Assessment Checklist (ACRAC), and the advancement of several information security maturity models, and numerous papers and books.
He acts as a political/military/intelligence/police advisor among others to the European Parliament, proposing highly individualised, complex, multi-phase approaches in which a threat-landscape is established, and risk-mitigation-projects resulting thereof, after which a (legislative) framework will reduce the potential fallout of cyberattacks.
One Response
I would definitely say that Mr Lars is highly professional and most experienced cyber security expert who has build his name all over