What is the Problem

Complexity

Building and maintaining software is fundamentally about managing complexity. As in nature, the more complex a system is, the more prone to decay and failure.

While any system has a minimum necessary complexity (related to W. Ross Ashby’s Law of Requisite Variety) to achieve its purpose, having complexity above this threshold is a lability that is likely to result in weird machine behaviors and thus security vulnerabilities.

Weird Machines

So what is the essence of a cyberattack? An adversary would like to make a system perform actions contrary to the wishes of the owner/administrator of said system. One way to achieve this objective is to load new code onto the machine, however if the attacker does not have the ability to load code this is not feasible.

Fortunately for the attacker, overly complex systems tend to have “excess” compute built-in. There are effectively weird machines hidden within. If the weird assembly instructions can be discovered, these weird machines can be programmed to perform a wide variety of tasks, including loading new code.

Invalid Input and LangSec

Language-theoretic Security is an interesting approach to restricting the input space and thus reducing program errors and opportunities to find exploits.

Invalid State Transitions

Make invalid states unrepresentable (2023)

Constraints and PoLA (ZeroTrust)

You Need More Constraints

Lack of boundary contracts

Currently, there is no universal method for restricting the access and capabilities of external dependencies and other software components included into a system. Ideally, we could limit the access that a given component has to a subset of the code or data in the system. While permissions systems like SELinux and sandboxing, these exist at the program level or at the container/virtual machine level.

Software Defects

The other issue that can be prevented via correct by construction software methodologies is defective software.

While the state of cybersecurity and is deteriorating rapidly and cyberwarfare is on the rise, more deaths are currently attributable to software defects. These incidents include the infamous Therac-25, the Patriot Missile System, the Boeing 787 Max 8, self driving car crashes, and the rapidly rising incidence of people being jailed due to broken fraud detection algorithms.

While these incidents are painted as “only mistakes”, the fact that so much harm is occurring without an external malicious party should be far more concerning. The threat is coming from inside the industry, and it’s rising.

A companion site and video series studying these Doomsday Machines is in the works.

Vulnerable Topologies

Certain organizational and system topologies are inherently vulnerable. Centralized systems are inherently vulnerable to compromise, denial of service, and other forms of abuse.