Wait … it’s not just the bad guys and hackers we have to worry about anymore?
By R. Dana Barlow
January 2025 – The Journal of Healthcare Contracting
What transpired this past summer on July 19 clearly was unexpected and unfathomable.
Perhaps the Associated Press categorized this aptly with the keen weekend headline, “Technology’s grip on modern life is pushing us down a dimly lit path of digital land mines.”
Many acknowledge that traditional cybersecurity measures are designed to help you defend and prevent illegal and unauthorized hacking and intrusions from bad guys. In short, they are designed to prevent – not cause – computers to crash.
Unfortunately, when a “trusted business partner” in the cybersecurity arena (like, for example, CrowdStrike) suffers a problem that negatively impacts programming from one of the world’s largest companies (Microsoft), seriously impeding all sorts of communication, commerce and care on a global scale, you might wonder whether you now must worry about and protect yourselves against authorized vendors, too.
In his story that was updated on July 27, AP Technology Writer Michael Liedtke called this incident, which reportedly affected an estimated 8.5 million Windows devices around the world that slowed or stopped operations among airlines and airports, businesses, hospitals and others, a “telltale moment – one that illustrates the digital pitfalls looming in a culture that takes the magic of technology for granted until it implodes into a horror show that exposes our ignorance and vulnerability.” (SOURCE: “Technology’s grip on modern life is pushing us down a dimly lit path of digital land mines,” AP News, July 27, 2024.)
Liedtke quoted Paul Saffo, identified as a Silicon Valley forecaster and historian: “We are utterly dependent on systems that we don’t even know exist until they break. We have become a little bit like Blanche DuBois in that scene from ‘A Streetcar Named Desire,’ where she says, ‘I have always depended on the kindness of strangers.’” Liedtke included a YouTube video link to the scene.
One southeastern healthcare supply chain executive attempted to squeeze lemonade out of lemons with a cheerily optimistic observation of a key benefit from facing the electronic abyss.
“We got to see all of our IT folks show up here at work at the same time!” the executive chimed.
The Journal of Healthcare Contracting reached out to a variety of supply chain executives to learn how their organization dealt with the digital dilemma and established defensive as well as offensive strategies and tactics to combat future occurrences. Unfortunately, few were willing to share their observations on the record due to the sensitive nature of the event, its impact on their organization and the publicity safeguards their respective media communications teams erected.
However, supply chain executives at two prominent integrated delivery networks (IDNs) were willing to provide a glimpse into what happened at their organizations, how they handled it and how they are working to avert future incidents, if JHC granted them anonymity. Here’s what they shared.
JHC: What hardware and software products/systems specifically at your organization were affected by CrowdStrike’s action(s) and how did that affect your operations and services?
PROVIDER 1: Generally, a significant portion of workstations and servers were impacted. Each has to have a hand on them to apply a fix, so this took significant time and resources. Most systems were back up in 48 hours. All were up in five days. In supply chain, our ERP and handhelds worked, but the middleware between them was impacted so orders to distribution did not go out in the first 24 hours. But this was quickly resolved.
PROVIDER 2: Like most, travel. We worked with our travel partners to prioritize workload, diverted work to other tools, such as phone and email and limited access to the digital platform. There were also indirect impacts in the supply chain as a whole and some disruption. Unfortunately, we have gotten quite good at dealing with various supply-related disruptions post-COVID.
JHC: How did you (try to) maintain operations and services – either through alternative technologies or reversion to manual processes?
PROVIDER 1: We had existing plans to cut over to down-time procedures that supported a continuation of operations. It was painful, nonetheless. Examples include clinical documentation, how to place and prioritize lab orders, etc. For supply chain, we replicated a prior day order to get products from the distributor. This has its own challenges as you are not necessarily getting the supplies you need or too many of others.
PROVIDER 2: We have a long tradition of having downtime procedures for all critical functions of our supply chain. Digital risk is not necessarily new, albeit with cloud/multi-tenant/multi-enterprise solutions and consolidations, the risk is far greater as an industry.
JHC: What did you learn from this incident in how to respond to future challenges like this? What did this crisis teach you about trusting your business partners?
PROVIDER 1: The biggest learning was the need for a more detailed plan on what system needed to be brought up first in the restoration plan. This would be agreed upon by service. With limited resources and capacity, a plan to return the most critical systems first allows the best use of resources. We have Level 1 systems but not prioritized more granular than that. We also learned a lot on how to mobilize additional resources to put hands on keyboards to apply any fix.
Organizations need to know their risk tolerance and own their quality control. This is key to the ability to trust partners. The way CrowdStrike-type software works, this event could easily happen with any software that does this type of service. The security software is in the guts of operating systems and software. It monitors and is constantly learning what is happening in the environment and making decisions on what looks suspicious and proactively turns parts of applications off.
PROVIDER 2: It is easy to get into a pattern of deprioritizing risk detection and preventive controls. The most recent issues were a good reminder that technology is far from bulletproof, and interruptions from time to time (while not welcome) are expected. We all need to make sure we are prepared with effective DR plans that are regularly reviewed and tested.
JHC: Why do (or don’t) you believe CrowdStrike’s solution to the problem and prevention from happening again is enough?
PROVIDER 1: They have increased transparency on how quality control/assurance works, how it failed and what the go-forward correction is. They also have provided organizations more flexibility and options on rollout of the service to allow early/mid/late adopter status and additional capability to run in part of system as a test prior to launching across entire network.
PROVIDER 2: Digital systems are complex, vulnerable and have some level of inherent risk. We need to continue to improve the quality and assurance of digital products, isolate and protect our most critical assets (including the ability to rapidly roll-back of any changes) and continue to improve our risk identification and mitigation. Risk as well as outages are not going to go away, and we should be aware, make best efforts to prevent as well as prepare in the event we are impaired.
JHC: Irony aside, how much – if at all – should healthcare supply chain pros be concerned about companies that create cybersecurity products designed to prevent disruption/interruption of operations and services, but then actually “cause” the problem(s) themselves and why? What should organizations – suppliers and providers, vendors and customers – take away from this?
PROVIDER 1: In overall context, CrowdStrike-type services have saved our organization from bad actors and downtime far more than the impact of this one instance. The benefit/risk is without a doubt worth it. That being said, organizations need to know how these systems work and understand third-party risk. Only with a solid understanding can we assess the true risk and level of protection, etc. needed. We are further consolidating our third-party risk management into a streamlined Vendor Credentialling process that centralizes workflow, risk assessment and risk management.
PROVIDER 2: As a consumer in the healthcare space – where we have a high degree of technology – we should be aware of these types of risks or have partnerships with those that specialize in this area. Having a strong third-party risk management (TPRM) program is a must in today’s environment. TPRM processes should be continuous, particularly when change is introduced, and not for just new suppliers/partners. In other words, “trust” but “verify.” Along with TPRM you need a very strong supply chain risk function that is continuously monitoring risks and mitigation. Also, while not new to anyone, supplier relations should be built on transparency, frequent communication and discussion as to not just what has been accomplished and needs to be done but also any risks that either party sees and how to best mitigate the