Relevant Tony Hoare quote: “There are two approaches to software design: make it so simple there are obviously no deficiencies, or make it so complex there are no obvious deficiencies”.

I'm starting to think that Opus and Mythos are the same model (or collection of models) whereas Mythos has better backend workflows than Opus 4.6. I have not used Mythos, but at work I have a 5 figure monthly token budget to find vulnerabilities in closed-source code. I'm interested in mythos and will use it when it's available, but for now I'm trying to reverse engineer how I can get the same output with Opus 4.6 and the answer to me is more tokens.

It looks like proof of work because:

> Worryingly, none of the models given a 100M budget showed signs of diminishing returns. “Models continue making progress with increased token budgets across the token budgets tested,” AISI notes.

So, the author infers a durable direct correlation between token spend and attack success. Thus you will need to spend more tokens than your attackers to find your vulnerabilities first.

However it is worth noting that this study was of a 32-step network intrusion, which only one model (Mythos) even was able to complete at all. That’s an incredibly complex task. Is the same true for pointing Mythos at a relatively simple single code library? My intuition is that there is probably a point of diminishing returns, which is closer for simpler tasks.

In this world, popular open source projects will probably see higher aggregate token spend by both defenders and attackers. And thus they might approach the point of diminishing returns faster. If there is one.

Although not an escape from the "who can spend the most on tokens" arms race, there is also the possibility to make reverse engineering and executable analysis more difficult. This increases the attacker's token spend if nothing else. I wonder if dev teams will take an interest.

Better to write good, high-quality, properly architected and tested software in the first place of course.

Edited for typo.

I discussed this in more detail in one of my earlier comments, but I think the article commits a category error. In commercial settings, most of day-to-day infosec work (or spending) has very little to do with looking for vulnerabilities in code.

In fact, security programs built on the idea that you can find and patch every security hole in your codebase were basically busted long before LLMs.

There's still the question of access to the codebase. By all accounts, the best LLM cyber scanning approaches are really primitive - it's just a bash script that goes through every single file in the codebase and, for each one and runs a "find the vulns here" prompt. The attacker usually has even less access than this - in the beginning, they have network tools, an undocumented API, and maybe some binaries.

You can do a lot better efficiency-wise if you control the source end-to-end though - you already group logically related changes into PRs, so you can save on scanning by asking the LLM to only look over the files you've changed. If you're touching security-relevant code, you can ask it for more per-file effort than the attacker might put into their own scanning. You can even do the big bulk scans an attacker might on a fixed schedule - each attacker has to run their own scan while you only need to run your one scan to find everything they would have. There's a massive cost asymmetry between the "hardening" phase for the defender and the "discovering exploits" phase for the attacker.

Exploitability also isn't binary: even if the attacker is better-resourced than you, they need to find a whole chain of exploits in your system, while you only need to break the weakest link in that chain.

If you boil security down to just a contest of who can burn more tokens, defenders get efficiency advantages only the best-resourced attackers can overcome. On net, public access to mythos-tier models will make software more secure.

  I would think, the golden age of criminal hacking is drawing to a close. This assume companies smart enough to do this however.

people biting into what companies say about their own products had always been the frustration in cyber. now more than ever.

nothing is better or worse, basically as its always been.

if you think otherwise, stop ignoring the past.

It looks like it, but it isn't. It's the work itself that's valued in software security, not the amount of it you managed to do. The economics are fundamentally different.

Put more simply: to keep your system secure, you need to be fixing vulnerabilities faster than they're being discovered. The token count is irrelevant.

Moreover: this shift is happening because the automated work is outpacing humans for the same outcome. If you could get the same results by hand, they'd count! A sev:crit is a sev:crit is a sev:crit.

By using these services, you're also exfiltrating your entire codebase to them, so you have to continuously use the best cyber capabilities providers offer in case a data breach allows somebody to obtain your codebase and an attacker uses a better vulnerability detector than what you were using.

I've said for decades that, in principle, cybersecurity is advantage defender. The defender has to leave a hole. The attackers have to find it. We just live in a world with so many holes that dedicated attackers rarely end up bottlenecked on finding holes, so in practice it ends up advantage attacker.

There is at least a possibility that a code base can be secured by a (practically) finite number of tokens until there is no more holes in it, for reasonable amounts of money.

This also reminds me of what I wrote here: https://jerf.org/iri/post/2026/what_value_code_in_ai_era/ There's still value in code tested by the real world, and in an era of "free code" that may become even more true than it is now, rather than the initially-intuitive less valuable. There is no amount of testing you can do that will be equivalent to being in the real world, AI-empowered attackers and all.

> Cybersecurity looks like proof of work now

Imo, cybersecurity looks like formally verified systems now.

You can't spend more tokens to find vulnerabilities if there are no vulnerabilities.

Security has always been a game of just how much money your adversary is willing to commit. The conclusions drawn in lots of these articles are just already well understood systems design concepts, but for some reason people are acting like they are novel or that LLMs have changed anything besides the price.

For example from this article:

> Karpathy: Classical software engineering would have you believe that dependencies are good (we’re building pyramids from bricks), but imo this has to be re-evaluated, and it’s why I’ve been so growingly averse to them, preferring to use LLMs to “yoink” functionality when it’s simple enough and possible.

Anyone who's heard of "leftpad" or is a Go programmer ("A little copying is better than a little dependency" is literally a "Go Proverb") knows this.

Another recent set of posts to HN had a company close-sourcing their code for security, but "security through obscurity" has been a well understand fallacy in open source circles for decades.

The article heavily quotes the "AI Security Institute" as a third-party analysis. It was the first I heard of them, so I looked up their about page, and it appears to be primarily people from the AI industry (former Deepmind/OpenAI staff, etc.), with no folks from the security industry mentioned. So while the security landscape is clearly evolving (cf. also Big Sleep and Project Zero), the conclusion of "to harden a system we need to spend more tokens" sounds like yet more AI boosting from a different angle. It raises the question of why no other alternatives (like formal verification) are mentioned in the article or the AISI report.

I wouldn't be surprised if NVIDIA picked up this talking point to sell more GPUs.

Dijkstra would shake his head at our folly.

I'm curious to see if formally verified software will get more popular. I'm somewhat doubtful, since getting programmers to learn formally math is hard (rightfully so, but still sad). But, if LLMs could take over the drudgery of writing proofs in a lot of the cases, there might be something there.

Everything eventually turns into Bitcoin. That’s what I plan to see in the future years and decades.

If you have a limited budget of tokens as a defender, maybe the best thing to spend them on is not red teaming, but formalizing proofs of your code's security. Then the number of tokens required roughly scales with the amount and complexity of your code, instead of scaling with the number of tokens an attacker is willing to spend.

(It's true that formalization can still have bugs in the definition of "secure" and doesn't work for everything, which means defenders will still probably have to allocate some of their token budget to red teaming.)

Everything eventually turns into Bitcoin. That’s what I plan to see in the future years and decades. Satoshi just saw it first.

> You don’t get points for being clever. You win by paying more.

And yet... Wireguard was written by one guy while OpenVPN is written by a big team. One code base is orders of magnitude bigger than the other. Which should I bet LLMs will find more cybersecurity problems with? My vote is on OpenVPN despite it being the less clever and "more money thrown at" solution.

So yes, I do think you get points for being clever, assuming you are competent. If you are clever enough to build a solution that's much smaller/simpler than your competition, you can also get away with spending less on cybersecurity audits (be they LLM tokens or not).

we did a lot of thinking around this topic. and distilled it into a new way to dynamically evaluate the security posture of an AI system (which can apply for any system for that matter). we wrote some thoughts on this here: https://fabraix.com/blog/adversarial-cost-to-exploit

Please. Are we going to rely now in Anthropic et al to secure our systems? Wasn’t enough to rely on them to build our systems? What’s next? To rely on them for monitoring and observability? What else? Design and mockups?

If you run this long enough presumably it will find every exploit and you patch them all and run it again to find exploits in your patches until there simply... Are no exploits?

In other news, token seller says tokens should be bought

>You don’t get points for being clever. You win by paying more.

Really depends how consistently the LLMs are putting new novel vulnerabilities back in your production code for the other LLMs to discover.

> to harden a system you need to spend more tokens discovering exploits than attackers will spend exploiting them.

If we take this at face value, it's not that different than how a great deal of executive teams believe cybersecurity has worked up to today. "If we spend more on our engineering and infosec teams, we are less likely to get compromised".

The only big difference I can see is timescale. If LLMs can find vulnerabilities and exploit them this easily (and I do take that with a grain of salt, because benchmarks are benchmarks), then you may lose your ass in minutes instead of after one dedicated cyber-explorer's monster energy fueled, 7-week traversal of your infrastructure.

I am still far more concerned about social engineering than LLMs finding and exploiting secret back doors in most software.

Security always had “defender’s dilemma” (an attacker needs to find one thing, but defender needs to fix everything) problem, nothing is new in terms of AI’s impact just application of different resources and units.

Cybersecurity has always been proof of work. Fuck, most of software development is proof of work by that logic. Thats why many attacks originate from countries were the cost of living is a fraction of the COL in the United States. They can throw more people at the problem because its cheaper to do so.

But I don't really get the hype, we can fix all the vulnerabilities in the world but people are still going to pick up parking-lot-USBs and enter their credentials into phishing sites.

All of the recent news read like something that could happen in a cyberpunk novel - AIs that defend vs AIs that do the attacks.

I think were are already here. I wrote something about this, if you are interested: https://go.cbk.ai/security-agents-need-a-thinner-harness

> Classical software engineering would have you believe that dependencies are good (we’re building pyramids from bricks)

Would it? I’m old school but I’ve never trusted these massive dependency chains.

That’s a nit.

We’re going to have to write more secure software, not just spend more.

> to harden a system you need to spend more tokens discovering exploits than attackers will spend exploiting them.

I, for the NFL front offices, created a script that exposed an API to fully automate Ticketmaster through the front end so that the NFL could post tickets on all secondary markets and dynamic price the tickets so if rain on a Sunday was expected they could charge less. Ticketmaster was slow to develop an API. Ticketmaster couldn't provide us permission without first developing the API first for legal reasons but told me they would do their best to stop me.

They switched over to PerimeterX which took me 3 days to get past.

Last week someone posted an article here about ChatGPT using Cloudflare Turnstile. [0] First, the article made some mistakes how it works. Second, I used the [AI company product] and the Chrome DevTools Protocol (CDP) to completely rewrite all the scripts intercepting them before they were evaluated -- the same way I was able to figure out PerimeterX in 3 days -- and then recursively solve controlling all the finger printing so that it controls the profile. Then it created an API proxy to expose ChatGPT for free. It required some coaching about the technique but it did most of the work in 3 hours.

These companies are spending 10s of millions of dollars on these products and considering what OpenAI is boasting about security, they are worthless.

[0] https://news.ycombinator.com/item?id=47566865

Does this mean all code written before Mythos is a liability?

I don't know about Mythos but the chart understates the capability of the current frontier models. GPT and Claude models available today are capable of Web app exploits, C2, and persistence in well under 10M tokens if you build a good harness.

The benchmark might be a good apples-to-apples comparison but it is not showing capability in an absolute sense.

There are never ending ways to make agents better at hacking. Defense is clearly behind. At my startup we are constantly coming up with new defensive measures to put our hacking agent Sable against, and I've determined that you basically need to be air gapped in the future for a chance of survival. A SOC of AI agents can't keep up with 1 AI hacker on a network that is even remotely stealthy. it is a disaster. wrote an article about it: https://blog.vulnetic.ai/evading-an-ai-soc-with-sable-from-v...

My first thought seeing the title: "always has been"

https://imgflip.com/memetemplate/Always-Has-Been

Everything in modern corporate is just proof of work. Security is filling out forms. Engineering is just endless talking. Token-maxing is the new meta.

Trusted software will be so expensive that it will effectively kill startups for infrastructure, unless they can prove they spent millions of dollars hardening their software.

I predict the software ecosystem will change in two folds: internal software behind a firewall will become ever cheaper, but anything external facing will become exponential more expensive due to hacking concern.

Maybe I’m missing something, but there’s also the idea that you don’t need to be perfectly secure, you just need to be secure enough that it’s not worth the effort to break in.

In the case of crooks (rather than spooks) that often means your security has to be as good as your peers, because crooks will spend their time going with the best gain/effort ratio.

> If corporations that rely on OSS libraries spend to secure them with tokens, it’s likely going to be more secure than your budget allows.

That's a really big "if". Particularly since so many companies don't even know all of the OSS they are using, and they often use OSS to offload the cost of maintaining it themselves.

My hope is when the dust settles, we see more OSS SAST tools that are much better at detecting vulnerabilities. And even better if they can recommend fixes. OSS developers don't care about a 20 point chained attack across a company network, they just want to secure their one app. And if that app is hardened, perhaps that's the one link of the chain the attackers can't get past.

What do they mean when they say "no diminishing returns?" does this essentially mean the code you are testing has no bounded state space and you continue to find infinite paths?

Because we have tools and techniques that can guarantee the absence of certain behavior in a bounded state space using formal methods (even unbounded at times)

Sure, it's hard to formally verify everything but if you are dealing with something extremely critical why not design it in a way that you can formally verify it?

But yeah, the easy button is keep throwing more tokens till you money runs out of money

I mostly agree with the article.

> You don’t get points for being clever

Not sure about this framing, this can easily lead to the wrong conclusions. There is an arms race, yes, and defenders are going to need to spend a lot of GPU hours as a result. But it seems self-evident that the fundamentals of cybersecurity still matter a lot, and you still win by being clever. For the foreseeable future, security posture is still going to be a reflection of human systems. Human systems that are under enormous stress, but are still fundamentally human. You win by getting your security culture in order to produce (and continually reproduce) the most resilient defense that masters both the craft and the human element, not just by abandoning human systems in favor of brute forcing security problems away as your only strategy.

Indeed, domains that are truly security critical will acquire this organizational discipline (what's required is the same type of discipline that the nuclear industry acquires after a meltdown, or that the aviation industry acquires after plane crashes), but it will be a bumpy ride.

This article from exactly 1 year ago is almost prophetic to exactly what's going on right now and the subtle ways in which people are most likely to misunderstand the situation: https://knightcolumbia.org/content/ai-as-normal-technology

As a result of all this AI "find a zero-day" business, when I boot to windows I open the task manager and order by pid. I kill anything I didn't start or don't recognise.

The only process that scared me was windowgrid. It kept finding a way back when I killed all the "start with boot" locations I know. Run, runonce, start up apps, etc. Surely it's not in autoexec.bat :)

Maybe code quality shouldn't be considered cybersecurity in the first place?

When things are tagged "cybersecurity", compliance/budget/manager/dashboard/education/certification are the usual response...

I don't think it would be an appropriate response for code quality issues, and it would likely escape the hands of the very people who can fix code quality issues, ie. developers.

Long ago, during the Viet Nam conflict, the US government learned that computers needed to be able to securely process data from multiple levels of classification simultaneously. Research in the 1970s found solutions that were adopted in the Mainframe world, like KeyKOS and EROS. Then the PC revolution swept all that away, and we're here 40+ years later, with operating systems that trust every bit of code the user runs with that user's full authority.

It's nuts. If the timing were slightly different, none of this "Cybersecurity" would even be a thing. We'd just have capabilities based, secure general purpose computation.

The PoW analogy completely ignores the actual hard part: fixing the stuff. It’s cool if you burn millions of tokens to find 1,000 bugs, but it's completely useless if your small team only has the bandwidth to safely patch 5 of them without taking down prod.

I remain skeptical, security is not a notch that you can turn, you can't shove more money or more tokens and make the thing more security.

Not saying security will never be dominated by AI like it happened with chess, with maps, with Go, with language. But just braindead money to security pipeline? Skeptical.

Am I the only one who thinks this is exactly like it was before AI, when we used small batch hand crafted tokens made by organic engineers to find vulnerabilities?

These mass-produced tokens are just cheaper...

To me it looks like formal verification is going to be the answer. We're going to move up the ladder and write formal specs and proofs soon.

really, really?

After how many years of "shifting left" and understanding the importance of having security involved in the dev and planning process, now the recommendation is to vibe code with human intuition, review then spend a million tokens to "harden"?

I understand that isn't the point of the article and the article does make sense in its other parts. But that last paragraph leaves me scratching my head wondering if the author understands infosec at all?

I can see the dichotomy forming in the "post AI" world;

1) massive companies spending millions of tokens to write+secure their software

2) in the shadows, "elite" software contractors writing bespoke software to fulfill needs for those who can't afford the millions, or fix cracks in (1)

(Oh wait, I think this is what is happening now, anyway, minus the millions of tokens)

I don't think open source will get stronger. Those who have enough GPU power won't depend on multiple human eyes anymore. AI will be enough.

I already see this happening: companies are moving toward AI-generated code (or forking projects into closed source), keeping their code private, AI written pipelines taking care of supply chain security, auditing and developing it primarily with AI.

At that point, for some companies, there's no real need for a community of "experts" anymore.

The cost of this is going to come down dramatically - just throwing the model at the codebase is a really inefficient process. My own experiments show that spending more tokens on understanding and transforming how the codebase can be explored(i.e enumerating source to sink traces) drastically lowers the cost to confirm vulnerabilities.Something that excites me greatly is that software quality has been incredibly difficult primarily because no single developer can hold the entire contract in their head and analyze it. It's now a reality that we can transform raw source code into actionable artifacts that allow a system to see the big picture and pin point the fracture points within it.

> This chart suggests an interesting security economy: to harden a system we need to spend more tokens discovering exploits than attackers spend exploiting them.

What's new?

It was always about spending more money on something.

Team has no capacity? Because the company doesn't invest in the team, doesn't expand it, doesn't focus on it.

We don't have enough experts? Because the company doesn't invest in the team, doesn't raise the salary bar to get new experts, it's not attractive to experts in other companies.

It was always about "spending tokens more than competitors", in every area of IT.

"Security economy: to harden a system we need to spend more tokens discovering exploits than attackers spend exploiting them. To harden a system you need to spend more tokens discovering exploits than attackers will spend exploiting them." - This feels similar to missile defense dilemma. Spending 2M$ missile to attack a 20k$ drone.

The problem with the security researcher industry is that it is infested with self promoters who talk about methodologies and tools but have never written any secure software themselves. Or any software at all, as the GitHub accounts from some of these geniuses show.

Of course those are attracted to new tools and AI shill institutes like AISI (yes, the UK government is shilling for AI, it understands a proper grift that benefits the elites).

Security "research" is perfect for talkers and people who produce powerpoint graphs that sell their latest tools.

You still can sit down and write secure software, while the "researchers" focus on the same three soft targets (sudo, curl, ffmpeg) over an over again and get $100,000 in tokens and salaries for a bug in a protocol from the 1990s that no one uses. Imagine if this went to the authors instead.

But no, government money MUST go to the talkers and powerpointists. Always.

Give more ammo to bad actors and sell the ammo to defenders, charge both for tokens. Why isn't this business model banned already?

I don't understand the nature of the supposed security incidents found by LLMs:

Are these totally previously unknown security holes or are they still generally within the umbrella of our understanding of cybersecurity itself?

If it's the latter, why can't we systematically find and fix them ourselves?

Then the question: what is cheaper, secure a code base written by humans, or secure a code base vibe coded with an army of agents?

Why this is the wrong analogy: finding hash collisions, while exponentially harder with N, is guaranteed to find, with enough work, some S so that H(S) satisfies N, so an asymmetry of resources used will have the side with more work eventually winning. But bugs are different: 1. different LLMs executions take different branches, but eventually the branches possible based on the code possible states are saturated. 2. if we imagine sampling the model for a bug in a given code M times, with M large, eventually the cap becomes not M (because of saturated state so of the code and the LLM sampler) but I, the model intelligence level. The OpenBSD SACK bug easily shows that: you can run an inferior model for an infinite number of times, it will never realize that the lack of validation of the start window if put together with the integer overflow then put together with the fact the branch where the node should never be NULL is entered produce the bug. So cyber security of tomorrow will not be like proof of work "more GPU wins", but better models and faster access to such models win.

Interesting reading, but it brings me some thoughts.

Security was always about having more money/resources. Using more tokens is just another measure for the same.

Some previous post, which I cannot verify myself, stated that mythos is not as powerful as it seems to be as the same bugs could be found using much smaller/simpler models and that the method is the key part.

Looks similar to the halting problem to me (https://en.wikipedia.org/wiki/Halting_problem)

Proof-of-work analogy is backwards. Bitcoin is symmetrical, every miner runs the same hash. Cybersecurity is an NP-hard chess match where the attacker gets to pick the square. 100M tokens for one exploit also means 100M tokens × every asset to cover the defender side. "Both sides spending" isn't parity when one side wins with a single find.

we're going to have to get a lot better at building test suites. for example every js exploit found in browsers should also be added to https://github.com/tc39/test262.

> This chart suggests an interesting security economy: to harden a system we need to spend more tokens discovering exploits than attackers spend exploiting them.

What this fails to take into account is that unless the codebase is changed, there are a finite amount of actual (and even fewer actionable) bugs in a piece of code, but an infinite amount of potential attacker spend; nothing stops you running mythos against it, whether it finds anything or not, and because each run is atomic by nature, you just have to play the numbers out and see when the average vuln discovery rate is dropping. You could spend a billion dollars and not find anything, without the defender spending a cent.

Generally speaking, the advantage goes to whoever can spend more time or money on security research (this has always been true, which is why the NSA was able to find Windows exploits that M$ did not). But eventually the fount of bugs in a piece of software will dry up, and attackers have no way of knowing if that's the case or not before dumping money at it.