The Future of Software Security

Software security has always been a big deal. It can really ruin things for your stuff to be exploited. In spaces like cryptocurrency it can mean the end of your organization. But not all things are that complete. Lots of software has a risk profile where prioritizing never-making-a-mistake isn't going to be sensible. The advent of AI agents has demonstrated this with particular significance.

The previous model was something like this:

1. Write some software
2. Do some kind of security analysis on it - pentesting, code review, etc.
3. Release the software and forget about it entirely until your annual security audit

Lots of organizations relied on this approach because companies win customers by providing functionality and only rarely by withholding functionality due to a security review. Consequently, the gating function of security was treated as an afterthought made worse by hiring specialists in security who spend most of their time ensuring that they play the role of gatekeeper. This was never secure, but it is even less secure today now that modern AI agents outrace security engineers all over the world.

A future model of secure software should take advantage of the fact that we now have intelligent and tireless agents that can try to break one's software before others break your software. We no longer need to gate features on a thorough security review. Instead, our security team should be a near fully-automated system that is constantly attempting intrusion and exfiltration.

The most common flaws are likely going to be:

1. Out-of-date software, specifically that which has public CVEs (and therefore known exploits)
2. Library software bugs that hit a broad category of software
3. "Supply-chain attacks" that hit broadly used libraries or frameworks

Hackers are constrained by the market as well and their gains are best when they have single large targets that they can attack prior to detection.

The truth is that we probably should have been doing security this way in the first place. Getting things right today don't mean anything if a novel exploit has been discovered. The idea of a full security review being sufficient to allow software to exist in public was false. Whether or not software engineers believed this, this is how security teams and software engineers operated in practice. Critical bugs or exploits were discovered by accident. This is not going to be the case in the future.

What we would all have liked to have done is had software engineers of the calibre of Tavis Ormandy constantly attempting to find flaws in our software and the software it depends upon, so that they may be detected out-of-band from actual development. This is not to say that it is defensible to write in string concatenated SQL and so on, but these classes of human-detectable bugs that are visible to the average software engineer are also the class of bugs that are easily detectable by an AI code review agent.

Modern -cyber models are incredibly capable. They can exploit unbelievable numbers of things through a combination of knowledge of obscure tricks, the ability to not tire, and an exhaustive memory of what software systems are vulnerable. The only way for us to survive the coming security apocalypse is constantly vigilant agents both attempting to emulate our enemies but also watching for them. The idea that humans can play this role is foolish. We need machines for this.