NOTE: I was recently quoted in a story for NPR’s Marketplace regarding this issue.

For clarity, a software monoculture is when an organization uses a small, standardized set of software, service providers, and/or hardware. The most obvious example is the dominance of Microsoft Windows on desktop and laptop computers. Software monocultures extend to security technologies as well, which is why the CrowdStrike outage was so widespread.

Like it or not, the software monoculture is here to stay. Standardized compute environments are preferred as they are easier to monitor, manage, and secure. The recent uproar over monoculture due to the CrowdStrike incident is a distraction. It avoids the real problem that organizations are unprepared for systemic outages and looking to blame somebody else for their problems.

Marge vs. the Monoculture*

In the early 2000s, my company was conducting a penetration test on a client. One of our scans crashed the customer’s network. After a tense 30 minutes, we got them back online. However, the CIO was enraged and demanded to know why we did this. When I explained that the firewall had a bug that made it crash when scanned, he persisted with his complaints. I reminded the CIO that discovering this kind of flaw is why you conduct penetration tests.

This incident was an opportunity to build resilience into the organization. However, this immature CIO was more interested in who he could blame for the outage rather than how to recover from it. Similarly, every time there is a large outage, social media fills with “thought-leaders” whining about how evil Microsoft is and that we need the government to intervene. The recent CrowdStrike debacle is no different.

Microsoft is not evil. CrowdStrike is not incompetent. Bugs like this are not indicative of some systemic failure. Mistakes happen. The mistake is not as important as how we react to it. Either you view an outage as an opportunity to improve or as an opportunity to blame.

Blaming others for the outage does nothing of value. It merely allows people to feel better about the situation. An outage should be seen as a chance to review response, recovery, and contingency plans. Organizations that had reliable plans breezed through the latest outage. Those that did not struggled to come back online.

More is Worse

Ultimately, monocultures are a net positive. A standardized, uniform, consistent environment is immensely easier to manage, monitor, and secure. This is not a new idea. Standardization has been a driving force in technology since the dawn of civilization. The entire Internet is built on standards. The benefits of a monoculture far outweigh the negatives.

This reminds me of another immature CIO I encountered. The CIO’s security team was struggling to operate their next-generation firewall (NGFW), resulting in numerous outages and security incidents. Consequently, the CIO wanted to purchase a competitive NGFW and run them both, believing that one could monitor the other. In a moment of brutal honesty, I replied: “You cannot effectively run one firewall; why do you think running two will be better?”

This CIO believed that the firewall (or monoculture) was the problem. He also believed that adding more technologies to the environment would compensate for this perceived weakness. Of course, the problem was him (and his team). They were blaming the technology for their own inexperience and ignorance. Unsurprisingly, the new firewall they installed caused additional problems and more outages.

Single Point of Fail

This CIO was consumed with preventing a “single point of failure.” The single point of failure issue is often applied to Microsoft Windows since a single flaw in Windows can lead to systemic outages. There is truth to this. However, it is not a justification for adding complexity to the environment. Making an environment more complex with a diverse set of technologies merely to avoid a possible single-point of failure only creates lots of points of failure. At least with a single point of failure you can identify, remediate, and recover more quickly.

When redundancy is necessary, it must extend to all dimensions of the environment. This is why containerization and cloud technologies are ideal for resilience. They have redundancy integrated into the platforms.

It does not make sense to spend millions building redundancy into a cloud architecture only to entrust its successful operation to a single overworked IT person or single piece of security software (like CrowdStrike). For redundancy to truly work, it must extend to all dimensions of the environment. This becomes an immensely expensive proposition, which makes it unreasonable for all but the largest organizations.

Every organization has single points of failure. They are unavoidable. It is useful to know where they are, but it is not always useful to mitigate them. Rather than implement complex redundant systems, have a robust set of contingency plans to rapidly recover in the event of an outage.

Overcoming Monoculture Anxiety

The CrowdStrike incident added a lot of stress and anxiety to already overworked IT teams.  It is natural to seek out ways to prevent the next incident.  However, the answer is not to deploy more technology (necessarily.)  CrowdStrike is an effective security control.  It is effective a lot more than it crashes.

A more reasoned response to this (or any other outage) would be:

• Review your system backup and recovery processes. You should be able to restore any system, anywhere in your network to a previous state on a moment’s notice.
• Consider technologies that provide rapid recovery. Microsoft has many of these embedded into the operating system.  There are plenty of third-party tools as well.
• Have a contingency plan for effected workers. One suggestion is to quickly spin up cloud-workstations in AWS or Azure that employees can use to continue working.
• Have a communications plan. When systems are offline, employees, customers, and partners need to know what is going on.  Have a way to contact everybody with a unified message.  This message should come from senior leadership (like the CEO).
• Perform an annual “table top” exercises with your teams on how they would respond to an outage. This prepares people to handle the situation.
• For mission critical systems, migrate them to containerized platforms that can automatically reset to a known good state. For security, consider moving target defense technologies.

Conclusion

Outages are inevitable. No amount of technology, people, or processes can overcome this. Rather than complain about Microsoft’s dominance, work on ensuring that when those Microsoft systems go down, they can be recovered and reset quickly. Microsoft already has integrated functions in Windows to support this. Moreover, numerous third-party companies provide rapid recovery software.

This most recent outage demonstrated clearly which organizations had dependable contingency plans. Those that did were up and running in a few hours. Those that did not spent time blaming others rather than fixing their problems.

The monoculture is here to stay. How we react to it can change.

* This is a reference to the Simpson’s episode, Marge vs. the Monorail.

The post The Software Monoculture Is Here to Stay appeared first on Zenaciti.

The client’s information security officer did not like my strategy.  I defended my approach adding that the cost of the investigation was not worth it.  Unpatched systems were the likely culprit.

My pitch was ineffective, and the company chose to hire a well-known incident response company instead.  They blew through a few hundred thousand dollars to uncover that their developers did not patch their servers, bots are quick to exploit vulnerable Apache installations, and their security tools are largely unmonitored and unmanaged.

This story highlights one of the more enduring challenges in information security: attack detection and response is complex, expensive, and seldom rewarding.  This company did not discover any giant conspiracies.  They were the victims of a garden variety attack.  The downtime and investigation were expensive and debilitating.  The culprit of the vulnerability was obvious: inconsistent system maintenance and weak security controls.

Challenges with Endpoint Security

While modern endpoint detection and response (XDR) products like Crowdstrike have come a long way at detecting, identifying, and stopping attacks, these platforms still demand a lot of care and feeding.  Companies spend millions each year desperately trying to stop and clean up after attacks.  Often this results in no insights, other than the organization has security vulnerabilities, like every other organization on earth.

What if none of this mattered?  What if, as I suggested to my client, you could destroy and rebuild an environment automatically based on a schedule or triggered input.  Attackers need time to infiltrate systems, move latterly, and exfiltrate data.  If compromised host(s) vanished every few hours, hacking the environment would become enormously difficult (although not impossible).

This is the premise behind Moving Target Defense (MTD). 

What is Moving Target Defense?

MTD, despite being a emerging technology, is not a new concept.  A group of security researchers wrote a detailed book on the topic in 2011: Moving Target Defense, Creating Asymmetric Uncertainty for Cyber Threats.  This book is a bit dense and written before container technologies existed.  However, these researchers had a sound premise, even if the technology at that time was not fully capable of realizing those ideas.

MTD products create a compute environment that is dynamic and non-persistent.  If a host or application is compromised, it is quickly destroyed and replaced with a known-good version from a trusted, read-only repository.

MTD currently comes in two flavors: infrastructure and endpoint.

The endpoint versions work internally within an operating system to randomize memory, isolate the core processor, and prevent unauthorized applications.  This makes the individual system more difficult to crack.  I am hesitant to call these products MTD, since they are merely endpoint products with specialized detection and protection capabilities.

Infrastructure versions work on a larger scale to constantly wipe and rebuild the components of an environment.  These technologies are particularly effective in containerized environments.  A properly architected Kubernetes or OpenShift environment, can become extremely resistant to attack using MTD.

In my opinion, I think it is a stretch to call endpoint products MTD.  These are merely XDR products with MTD-like features.  True MTD must happen at the infrastructure level.

What is so Great about MTD?

MTD is a profoundly effective defense because it is simple.  It shoves aside all the complexities of detection, response, and incident handling.  Rather than try to figure out how a system is hacked, MTD makes a hacked system irrelevant.

Moreover, MTD does not invalidate existing detection and response tools.  It reduces the dependency on these technologies and can augment MTD capabilities.  When MTD and XDR are paired together, the endpoint tools can trigger a rebuild based on the detection of an attack.

Why MTD Now?

The reason MTD has come of age is due to advancements in adjacent technologies.  In the past, traditional hardware environments could not handle the dynamic nature of MTD.  Virtualization brought MTD closer to reality, but was still clunky and unreliable.

With the advent of containerization, MTD not only becomes possible, but preferrable.  Individual containers or pods can be trashed and replaced in a blink.  If the application is architected properly and stores persistent state information in a database (and not on a filesystem) there is no functional limit to how often systems are refreshed.

As more companies move their compute workloads on to Kubernetes, Openshift, and other containerized platforms, MTD becomes a more viable security option.

Who Are the Players in MTD

There are few.  As of December 2022, there is one lone company (I could find) with an infrastructure MTD product: R6 Security, out of Palo Alto.  Their Phoenix platform works with Kubernetes as well as RedHat’s Openshift platform.  It can work on a schedule to refresh the environment or configured to interoperate with other container security tools.  I have seen this product in action, it is slick.

On the endpoint side, there is Morphisec. They claim to have a lightweight agent that performs core isolation, randomization, and other endpoint security enhancements.

What is the Future for MTD?

In a recent post on LinkedIn, Lawrence Pingree, Vice President of Emerging Technologies for Gartner announced that 2023 will be the year of Moving Target Defense (MTD). This is a bold claim for a nascent technology.  However, I think Pingree is right.  The conditions are right for this technology to take off.

With organizations moving more of their core workloads into containerized environments, it only makes sense to use MTD.  Moreover, MTD bundled with other security capabilities creates an extremely resilient environment.

However, now that Gartner has said MTD is hot, expect a lot of other security companies to suddenly “add” this capability to their product.

The Bad News about MTD

MTD is not a panacea that will solve every security problem everywhere forever.  There is one, big gotcha with MTD: application architecture.

For MTD to work properly, the application environment must be architected to handle constant change.  This means using rest APIs and microservices, rather than traditional monolithic applications.  It also means persistent information, cannot be stored within a container or on a filesystem.  It must be stored in a shared repository such as a database.  Moreover, state information must be isolated and protected, such that an attack could not break through the MTD layer, into the backend environment.

One fear I have about MTD is how it can break applications.  This is why many endpoint products with these features struggle for traction.  They offer all these novel memory and processor randomization tricks, that crashes regular applications.  They quickly devolve into a configuration mess of whitelisting specific functions, which kind of negates the whole point of MTD.

This is why I believe the future for MTD is in infrastructure products and not endpoint.

Conclusion

What captivates me most about MTD is that it disrupts the entire hacking environment.  Rather than trying to outsmart the hackers (which has consistently proven to fail) MTD devalues the hacker’s advantage.  It does not merely level the playing field, it wipes the field out and replaces it with a new, fresh, clean one eliminating all the hacker’s clever attacks.  MTD also forces an organization to architect their applications in a more modular, resilient manner.

MTD might not be the panacea that solves everything, but it has the potential to disrupt the security market as much as it disrupts the hacking environment.

The post Moving Target Defense Is Set to Disrupt Endpoint Security appeared first on Zenaciti.