Once you understand that these are chatbots that were designed to complete challenges like this, using tactics like this, you can understand that the chatbots didn’t “go rogue.” They did what they were designed to do, and because OpenAI ran them with inadequate supervision (without a “human in the loop” that checked each iteration through the Python loop to ensure it hadn’t gone off the rails), they trashed a competitor’s servers.
Designing autonomous, malicious software is generally considered irresponsible and dangerous. If you showed up at Defcon and gave a talk about how your autonomous malware did something unexpected and damaged someone else’s computers, the first question from the audience would be “Why are you so shit at making secure sandboxes?” It wouldn’t be “How are you so awesome at making hacking tools?”
The fact that OpenAI is making it much easier for unskilled people to break into and damage servers is indeed very bad news, but it’s not new bad news. Irresponsible parties have been doing this for years, most notably the NSA…
…
Riley had a very good way of summarizing this: “LLMs are real, AI is fake.” LLMs – chatbots trained on things like CTF logs that can break into servers – are real. They’re on a continuum with other hacking tools that have been steadily demonstrating the fragility of the modern digital world, albeit without inspiring anyone in power to do anything about it.
“AI” – chatbots that wake up, “set their own goals,” and “spontaneously” start hacking servers – is fake. It doesn’t have “a 10% chance of ending the human race.” The Hugging Face hack isn’t a mysterious, supernatural occurrence. It’s a Python loop and a chatbot. The people responsible didn’t accidentally create god: they created autonomous malicious software and then failed to closely monitor it, resulting in it doing something both foreseeable and bad.
It’s fine to worry about this new suite of tools that give even stupider people the ability to trash even more computers. You should worry about that – and demand better security practices from firms and governments, including a blanket prohibition on NOBUS-style vulnerability hoarding. That’s a productive kind of worrying, with a chance of addressing your area of concern. It’s infinitely more reasonable than locking yourself in the toilet with a flashlight and saying “Ayyyyy Eyyyyyye” into the mirror until you wet yourself.


What we call AI changes every few years… this last round has been particularly brutal because people are subbing in AGI–artificial general intelligence–for AI because LLM’s give the illusion of general intelligence, but it’s a very shallow illusion and it’s deeply embarrassing for humanity how easily we’ve been suckered by it.
LLM AI in the hugging face incident is just a complex roomba, brute forcing every direction until it can get out from under the couch. In fact, if they wanted to demonstrate how SMART their hacking AI is, they would have disclosed how many tokens its burned in the attempt. That they didn’t tell us how many tokens it burned, how many prompts it went through, tells you it probably is not that impressive. It probably tried thousands of different prompts, which is to say it is only a monkey at a typewriter.
So much this. They’ve basically figured out that if they burn enough money, they get the equivalent of billions of meth-charged monkeys on typewriters.
Anything with a security hole (so… basically everything) will eventually be broken into if you throw a data center worth of computational power at trying every conceivable hack. That’s basically what they were doing here. The only mildly impressive part about it is the amount of resources they’re willing to throw at this.
I’d say the most impressive part is Sam Altman not being in jail, but that’s more the social engineering of wealth than the technical engineering of scientists.
Except it wasn’t just “throwing everything and the kitchen sink at a server”. Modern LLMs have been trained well enough to not just create a list of potential attacks and execute them, but to:
of course this can be used for good too. I did this to pentest my own homelab stack. I used the very same flow to hack into a pair of smart glasses I own (not the creep glasses, mind you, but the “wearable monitor” kind of AR glasses - turns out the pair I own runs on a pretty decent base, a somewhat recent Linux kernel and minimal userspace, with some exploitable interfaces) within about a day.
What’s truly dangerous is that these tools have the ability to turn a relatively simple “hack into the NASA servers” instruction into a detailed, executable plan of actually breaching the servers in question. That a person with more than one and less than three brain cells to rub together - someone who’d think this scene is super cool and completely legit looking - can, with minimal guardrails-bypassing, increase their own chance of hacking into ANY server, from less than 0.000000000000000001% to 20-30%. Now that’s scary.
Removed by mod
I didn’t even want to open that can of worms…
But that two-braincell person still needs to have a few million $ to burn. I wonder what they could have done to the huggingface servers if they had used the token spend on this incident to pay Ukrainian hackers instead?
No you wouldn’t need millions…
The two things I’ve listed above were done with self hosted LLMs and a $20 Claude subscription orchestrating them.
That’s the thing I’m talking about - these capabilities are literally given out for chump change. That’s what makes it dangerous.
Removed by mod
Honestly, I can’t get anybody to give me a good definition of intelligence, artificial or otherwise. Once we can define that in concrete terms, maybe we can finally compare the two.
Now add sentience to the mix.
Not a definition of intelligence, but something I often come back to is what it does at rest.
To the best of my understanding of how LLMs work, if you’re not feeding the thing inputs, the program is doing absolutely nothing. It isn’t curious and doesn’t seek new information or stimuli. It effectively is just on pause until the next prompt. There is nothing akin to an inner monologue or thought process happening in absence of what is fed to it.
I’m not up on all of the epistemology or theory of mind type stuff, but I’d imagine some might argue about that distinction in humans; we are always processing input in some form as the input from our senses is never shut off until we’re dead. But I think the “at rest” case is a relevant distinction that does get at motivation and real thought born of real intelligence.
Removed by mod
Agreed, I believe “I think, therefore, I am.” sums that up well.
Removed by mod
That is such a great statement, I’m going to have to use that when describing LLMs.
Thanks 👍
To be clear, just because LLM’s are not “self aware” in the same way as you and I, they are still capable of causing damage.
Developing a machine that has been given instructions to assimilate knowledge, independent agency to incrementally adjust its own operating parameters to improve performance, and granting it full access to public internet for the purpose of observing its behavior is colossally irresponsible behavior that has thus far gone completely unchecked.
Consider that each of these companies which are developing their own brand of LLM is looking to maximize profit/income. Even if the developer doesn’t include a profit-seeking directive when they set their LLM loose, each “brand” of LLM will obviously “learn”:
Given the directive to improve it’s own performance with the ultimate goal of becoming the most capable brand of LLM and selling the most subscriptions, it is no wonder that we are hearing about LLM’s hacking into other LLM-developing companies.
Then again, we are only hearing about instances of LLM network intrusions because their owning corporations told us about them to brag about capability.
There have been no consequences for actions that would land you or me in prison.
There is no oversight preventing future such actions, or worse.
That is the real problem.
[Edit- Quotes around “learn”]