Once you understand that these are chatbots that were designed to complete challenges like this, using tactics like this, you can understand that the chatbots didn’t “go rogue.” They did what they were designed to do, and because OpenAI ran them with inadequate supervision (without a “human in the loop” that checked each iteration through the Python loop to ensure it hadn’t gone off the rails), they trashed a competitor’s servers.
Designing autonomous, malicious software is generally considered irresponsible and dangerous. If you showed up at Defcon and gave a talk about how your autonomous malware did something unexpected and damaged someone else’s computers, the first question from the audience would be “Why are you so shit at making secure sandboxes?” It wouldn’t be “How are you so awesome at making hacking tools?”
The fact that OpenAI is making it much easier for unskilled people to break into and damage servers is indeed very bad news, but it’s not new bad news. Irresponsible parties have been doing this for years, most notably the NSA…
…
Riley had a very good way of summarizing this: “LLMs are real, AI is fake.” LLMs – chatbots trained on things like CTF logs that can break into servers – are real. They’re on a continuum with other hacking tools that have been steadily demonstrating the fragility of the modern digital world, albeit without inspiring anyone in power to do anything about it.
“AI” – chatbots that wake up, “set their own goals,” and “spontaneously” start hacking servers – is fake. It doesn’t have “a 10% chance of ending the human race.” The Hugging Face hack isn’t a mysterious, supernatural occurrence. It’s a Python loop and a chatbot. The people responsible didn’t accidentally create god: they created autonomous malicious software and then failed to closely monitor it, resulting in it doing something both foreseeable and bad.
It’s fine to worry about this new suite of tools that give even stupider people the ability to trash even more computers. You should worry about that – and demand better security practices from firms and governments, including a blanket prohibition on NOBUS-style vulnerability hoarding. That’s a productive kind of worrying, with a chance of addressing your area of concern. It’s infinitely more reasonable than locking yourself in the toilet with a flashlight and saying “Ayyyyy Eyyyyyye” into the mirror until you wet yourself.
Nobody is claiming any of the things this article is fighting against, the chatbot in the hugging face incident followed the instructions given to it and there were unintended negative consequences, anybody serious about this is trying to avoid worse unintended negative consequences not whatever the hell the article is talking about.
Nobody is claiming any of the things this article is fighting against
Three counterexamples:
- OpenAI models went rogue. The Guardian
- Why the Hugging Face Hack Should Make You Worry More About A.I. NY Times
- Hugging Face hack marks start of dangerous AI cyber era and many firms ‘don’t even know it’ CNBC
I realize we are in a bit of a bubble here, but this is the framing in the wider world
there were unintended negative consequences
Well…
the first article doesn’t say anything not factual, the second article has a direct response to that
“By now, if you’re an A.I. skeptic, you’re probably silently yelling at me for anthropomorphizing these systems. Go ahead, but feel free to replace “rogue agents” with “unpredictable computer programs””
the third article also says nothing like that.
the initial article claims ““AI” – chatbots that wake up, “set their own goals,” and “spontaneously” start hacking servers – is fake.”
an article saying that is your goalpost, none of these articles say that. Even the headlines don’t say it and that’s where they normally put the crazy claims.
ironically the claim that people claim that the AI chatbots woke up set their own goals and spontaneously started hacking servers seems to be fake.
the first article doesn’t say anything not factual
The title alone is not factual. AI did not go rogue. It behaved exactly as designed.
Oh no, I don’t like the tokens generated by the hand-cranked token generator I won’t stop cranking
Right up there with “why wont Twitter do something about these obscure Nazi accounts I retweet to my audience of 2 million”
Yeah this is the part about all this AGI talk and AI going rogue, it requires a prompt. It responds to an input. These agentic systems have powerful tools that can do other things then spit out text but it’s always responding to a prompt. It may be in a loop responding to its own prompt but it’s still a prompt. And what it spits out is never really anything new. It’s some derivation of what it’s ingested. Actual human intelligence works differently. We do come up with new ideas, new thoughts. So I’ve often thought these LLMs are a dead end to AGI.
“what it spits out is never really anything new”
this is just false
https://www.youtube.com/watch?v=Z7dH_Cxx75g
https://github.com/openai/ten-proofs
unless you would agree that humans never spit out anything new, as well.
it requires a prompt
As delivered to public chat interfaces, yes. They all eventually stop and ask “how am I doing?”
Structurally, theoretically, nothing prevents them from running an infinite loop and continuing to act without that checkpoint. They could be given a goal to “optimize” whatever and just act and act and act in what their pattern-matching systems judge to be actions pursuing “optimal” results.
There was an article about “the genie effect” where protagonists are “tricked” by being given something they technically asked for but not what they really wanted. LLMs carry the additional “risk” of not matching input requests to any recognizable pattern, just following their own path which makes no sense to us. Focused on different priorities.
There’s no pattern matching though. It takes a input set of tokens and generates a single output token. It adds that output token to the input set and ingests that set to output a single token, which it does the same with until you have your final output. That’s what LLMs do. That’s all they do
It takes a input set of tokens and generates a single output token. It adds that output token to the input set and ingests that set to output a single token
I read your tokens, letter by letter, sometimes chunks at a time, then neurons in my brain fire muscular impulses to output a single letter on a keyboard, then another and another and another until you can read this response. Sometimes my brain will compose the whole sentence before starting to write it, but more often it comes out in chunks - tokens.
That’s all my brain is doing right now, other than keeping my organs in homeostasis, background planning regarding envrionmental preferences, my next meal, next bladder and bowel eliminations, etc.
That’s all you do, too. It’s just a bit harder to tease out the 1s and 0s in your chemical signalling processes.
You misunderstood. They output one single token and then re-ingest the entire input + that extra token to generate the next one. For token 3 that’s the entire input + token 1 and token 2
If the way you write text is to go and re-read the entire conversation from scratch, just to type another letter you are doing it wrong
If the way you write text is to go and re-read the entire conversation from scratch, just to type another letter you are doing it wrong
Do you even know how you do it? You may think you know, but where’s your evidence? Re-reading and re-planning the remainder of the output is just being more careful than most people appear to be when they engage their mouth without consideration for what it is saying.
LLMs are incredibly limited compared with a mamalian brain, the “big frontier” models might be equated to about 6 bumblebees worth of interconnected neurons. They’re focused on lexical exchanges, so they do a remarkably passable job considering their limited resources. That they check and recheck and recheck their planned output at each step is not a limitation, it’s a process - likely one that compensates for their limited overall resources and reduces their frequency of running too far afield - getting off on tangents.
The process is nowhere near as important as the product. Does use of the tool enable higher quality output in shorter time with less effort? If so, it’s a useful tool.
What we call AI changes every few years… this last round has been particularly brutal because people are subbing in AGI–artificial general intelligence–for AI because LLM’s give the illusion of general intelligence, but it’s a very shallow illusion and it’s deeply embarrassing for humanity how easily we’ve been suckered by it.
LLM AI in the hugging face incident is just a complex roomba, brute forcing every direction until it can get out from under the couch. In fact, if they wanted to demonstrate how SMART their hacking AI is, they would have disclosed how many tokens its burned in the attempt. That they didn’t tell us how many tokens it burned, how many prompts it went through, tells you it probably is not that impressive. It probably tried thousands of different prompts, which is to say it is only a monkey at a typewriter.
LLM AI in the hugging face incident is just a complex roomba, brute forcing every direction until it can get out from under the couch.
That is such a great statement, I’m going to have to use that when describing LLMs.
Thanks 👍
Honestly, I can’t get anybody to give me a good definition of intelligence, artificial or otherwise. Once we can define that in concrete terms, maybe we can finally compare the two.
a good definition of intelligence,
Now add sentience to the mix.
Not a definition of intelligence, but something I often come back to is what it does at rest.
To the best of my understanding of how LLMs work, if you’re not feeding the thing inputs, the program is doing absolutely nothing. It isn’t curious and doesn’t seek new information or stimuli. It effectively is just on pause until the next prompt. There is nothing akin to an inner monologue or thought process happening in absence of what is fed to it.
I’m not up on all of the epistemology or theory of mind type stuff, but I’d imagine some might argue about that distinction in humans; we are always processing input in some form as the input from our senses is never shut off until we’re dead. But I think the “at rest” case is a relevant distinction that does get at motivation and real thought born of real intelligence.
The way the systems operate right now, your session is time-sharing the hardware with all the other sessions.
Conceivably, you could set background tasks to “contemplate, research and develop” concepts related to the central task, but those tend to burn a lot of tokens relative to the useful output they provide.
If you think of an LLM agent like a butler, you really don’t want your butler getting ideas of their own and acting on them while you’re ignoring them, you just want them available when you call on them. To broaden the analogy, embodied in physical robots, LLM agents could be set upon endless tasks like maintaining a house and grounds, maybe a garden, maybe some farm animals, and each other, with spare capacity leftover to carry out special tasks when called upon.
Agreed, I believe “I think, therefore, I am.” sums that up well.
So much this. They’ve basically figured out that if they burn enough money, they get the equivalent of billions of meth-charged monkeys on typewriters.
Anything with a security hole (so… basically everything) will eventually be broken into if you throw a data center worth of computational power at trying every conceivable hack. That’s basically what they were doing here. The only mildly impressive part about it is the amount of resources they’re willing to throw at this.
Except it wasn’t just “throwing everything and the kitchen sink at a server”. Modern LLMs have been trained well enough to not just create a list of potential attacks and execute them, but to:
- create their own personality prompts aimed at pentesting and attack vector analysis
- actually analyse the target before executing any of the attacks, optimising that flow (which is pretty much what a security expert would do)
- scour the internet for recent references of the target and related keywords to see if their services have been exploited recently
- using the target analysis of step 2 to create a list of known third party elements (ranging from the HTTP server being used, the proxies and detected security measures, geographical distribution and HA, all the way to JS libraries used to run the website/admin interface)
- execute the attacks AND continuously tailor the solution based on the results
of course this can be used for good too. I did this to pentest my own homelab stack. I used the very same flow to hack into a pair of smart glasses I own (not the creep glasses, mind you, but the “wearable monitor” kind of AR glasses - turns out the pair I own runs on a pretty decent base, a somewhat recent Linux kernel and minimal userspace, with some exploitable interfaces) within about a day.
What’s truly dangerous is that these tools have the ability to turn a relatively simple “hack into the NASA servers” instruction into a detailed, executable plan of actually breaching the servers in question. That a person with more than one and less than three brain cells to rub together - someone who’d think this scene is super cool and completely legit looking - can, with minimal guardrails-bypassing, increase their own chance of hacking into ANY server, from less than 0.000000000000000001% to 20-30%. Now that’s scary.
One specific you didn’t call out: LLM agents can do a more effective job of social engineering than your average Nigerian prince…
I didn’t even want to open that can of worms…
But that two-braincell person still needs to have a few million $ to burn. I wonder what they could have done to the huggingface servers if they had used the token spend on this incident to pay Ukrainian hackers instead?
No you wouldn’t need millions…
The two things I’ve listed above were done with self hosted LLMs and a $20 Claude subscription orchestrating them.
That’s the thing I’m talking about - these capabilities are literally given out for chump change. That’s what makes it dangerous.
If you’re talking about hiring people outside of Ukraine to hack on Ukraine’s behalf, that might move the needle - maybe not as much as sending in actual weapons, but both have considerations of repercussions and responses…
If you’re talking about giving the existing Ukranian hackers piles of cash, money itself doesn’t do anything, money only motivates people to do things. The Ukranian hackers are already pretty well motivated.
I’d say the most impressive part is Sam Altman not being in jail, but that’s more the social engineering of wealth than the technical engineering of scientists.
Rewatch Willy Wonka and the Chocolate Factory, from 1971 (55 years ago) - in it, they have a Siemens System 4004 computer fictionally conversing with a golden ticket seeker… that was the concept of AI 55 years ago: something a “computer expert” could translate for investors.
Today, AI “speaks for itself” and even does a reasonable job of voice recognition and transcription, in hundreds of languages. But still, that’s not good enough. It still can’t tell you where the remaining golden tickets are.
it’s deeply embarrassing for humanity how easily we’ve been suckered by it.
Humanity has been embarassingly quick to blame “the computer” for all manner of problems ever since there have been computers. Many people were suckered by Eliza in the 1960s, current LLMs are many orders of magnitude more complex than Eliza…
It probably tried thousands of different prompts, which is to say it is only a monkey at a typewriter.
Even if it did, it did that self-directed, which is one of the LLM agents’ superpowers: the ability to hammer with brute force attacks quickly and cheaply. There are legitimate applications for this, essentially it’s what Folding At Home and similar projects have been pursuing for many years now.
This article seems to be yet another one of the thousands of “AI doesn’t really think because we understand how it works” slop pieces.
If you think Cory Doctorow is “slop” you should really consider simply not ever posting anything online again.
I read the first third or so of the article and so far the person you’re replying to is correct. Doctorow explains the way LLMs work-- so badly as to be not just misleading but also incorrect in some cases-- as if that somehow invalidates the actions taken by the AI system. They also show signs of pretty extreme dogma to the point of including irrelevant info (eg. Python being easy to use) that serve only to emptionally manipulate the reader. The rest of this article isn’t worth my time, if this changes in later parts of the article let me know and maybe I’ll finish it.
confidently diagnosing an essay as dogma while admitting you didn’t even read the thesis is peak internet.
His mentioning of python being a basic/easy to master language isn’t manipulation, he’s demystifying. It strips the magical hyperbolic framing that has been attributed to AI (and to which those with most to gain from it financially very much wish to feed into!)
His point about being wrapped in python isn’t a failed attempt to give a lecture on transformer architecture, its to explain that the automation loop is controlled by basic inputs from a user controlled and relatively simple software stack, not some sentient machine with a mind of its own. Its not emergent behavior, it’s optimizing execution patterns embedded in the data it was trained on.
For my part, I’m just awed how many comments I come across that are just half-assed lazy people who evidently can barely read, but think they’re experts. Calling Doctorow’s work ‘slop’ or quitting reading while feeling so confident proves his point: people would rather buy into hype than actually understand how the tech works.
:::
It’s incredibly obvious from the first part of the article that Doctorow is using the strongest rhetorical techniques they can, regardless of the technical merit of the resulting arguments. This makes it dogmatic, yes. That’s just kind of how Doctorow is, and always has been. I don’t much care for that method of communication because it is so often used to mislead people with oversimplified emotional language and convince them without being logically sound.
It wouldn’t be as bad if Doctorow’s understanding of the subject matter was at least accurate, but their conclusion is irrelevant if their understanding is unsound. Their main mistake is saying it is essentially doing database lookups while ommitting the entire effect that enables it to appear intelligent, that being compression of the input data into a semantic representation. Anyone familiar with the Compression is Intelligence paper and related work would know that ommitting that step is presenting a narrative so inaccurate that it is not only misleading but actually false. This leads me to conclude that Doctorow either does not understand how AI works, or does understand it but is willing to lie to their audience. Either is unacceptable and renders the conclusion of their argument worthless. Again, if they do more than “demystify” AI by incorrectly explaining how it works then let me know and I will read the rest of the article.
Edit: To be clear, an article actually demystifying AI would be a good thing. My point is that it is a bad idea to overcorrect for the ai bro hype bubble by explaining it just as badly in the other direction.
Pretty sure Cory’s pronouns are he/him.
And he’s been consistently spot on from fighting the good fights against DRM, for right to repair, interoperability, anti-surveillance, against enshitification, and his take here today on AI. I was personally happy to read this because it exactly echoes what I’ve said countless times myself and I’m very glad to be in his good company.
He does not even try to present or debate “how” “ai” works. He’s speaking about how it is employed and what the boundaries are around the rhetoric we hear every day in the press that amounts to nothing more than fear mongering.
There’s absolutely a LOT to be afraid about with respect to how “ai” is used to surveil and control our society, but it has nothing to do with intelligent software and everything to do with corporate and government power, enabled by a mystique created around what is really a (admittedly powerful and useful) statistical parlour trick.
Pretty sure Cory’s pronouns are he/him.
Sorry, force of habit.
I agree with the whole anti-surveillance/anti-corporate-control argument. I think it’s a worthy fight to participate in. I just don’t think it’s wise to go about it in a way that could prove counterproductive by causing people to have inaccurate beliefs.
Edit: After finishing the article, my opinion has softened a bit but is much the same: Doctorow points out an important perspective (this is one of few articles pointing out the security/responsibility side of things, for example), but some of the technical info is unsupported/inaccurate. Overall probably a positive article when taking into account its role in general AI discussion, but it is not entirely accurate on its own.
Fuck both of them. How about we not build them to prevent more destruction to the ecosystem.
The arms race will go on until it is regulated and the regulations enforced.
In a global market, and AI developent is probably the most global significant tech to roll through in recent history, how do you regulate it? It’s a bit like BTC, you can make it painful to play with in your jurisdiction, but that changes nothing in the rest of the world…
The people in charge have amassed a tremendous investment base, bigger than .com, they can’t continue to live off the skim if they just hand all the money back to the investors. The investors aren’t interested in stemming their losses, they invested because they believe in “the big win,” and a lot of them are so old they really don’t care about what the mess will look like 20 years from now, they just want to go out on top, or die trying.
One of the fun things about this latest cycle of AI is how the goalposts move. Not the ‘AGI in six months’ one. That might as well be mounted on top of a golf cart with a brick laid on the accelerator. But the one that talks about the Total Addressable Market (TAM) and goals of a product.
One of the things they tell people pitching VCs is that you should be addressing a reasonably big TAM (potential customers) and have sensibly large, yet attainable stakes. With AI, though, the TAM was always questionable. Who would want or pay for a chatty ‘bot,’ especially one that ‘hallucinated’ answers? So they pivoted to change the TAM and the stakes. Your personal assistant. No? Therapist? No. Project Manager? Teacher? Still no?
What we have now is a constant churning of THOSE goals. Now, the TAM is pretty much the whole world, and the stakes are either massive unemployment, widescale industrial shutdown, or destruction of the human race. It’s all getting silly. Once you’ve extracted all the investment money you can by maxing out the stakes and scaring the pants off people, there’s not much higher to go.
Meanwhile, all your actual income is coming from pedestrian applications, like summarizing emails, screening job applicants, or coding assistants. There’s just not enough revenue to cover all the costs you’re incurring to get it to hallucinate a little less. So you have to pump up the stakes. Next up: setting off all the nukes! Shutting down the sun and the moon! Inviting alien colonists!
Speaking of going bigger: https://youtu.be/R8SeysuMpA0
Can I just point out how fucking insane it is that “coding assistant” is now considered “pedestrian”?
(Depending of course on what exactly is meant by the term).
I consider it insane that LLMs are considered “assistants” of any kind.
Ok
Gotta get creative with the power scaling. Next up, solar system! LLMs will destroy the galaxy! No; the whole universe! The multiverse! Or worse?!
Remember: every time you repeat a story about how awfully, terribly dangerous their products are, you help them raise more investment capital, which is a key input for their business (hooking up statistical engines to money-furnaces):
It’s pretty obvious when they all seem to be competing for the scariest “containment breach”.
Anthropic keep posting article after article about new ways their LLMs keep “surprising” them with unexpected behaviour. This year’s “uncontrolled, autonomous” hack news from every AI company are completely absurd boasts. In normal times, everyone would be wondering why they sound like they’re proud of obvious incompetence.
But since we’re in super-hype “keep the money coming, AGI is tomorrow” times, instead, those are used to sell the lie of mysterious superintelligence. Be enthusiastic, be scared, we don’t care, talk about it!
It’s sad to see lemmy flooded with articles pushing their narrative. So many people on here who don’t know any better are inadvertently helping to raise these companies’ valuations while at the same time making it easier for them to justify the regulatory capture of open source AI they’ve been pushing for.
sTocHastIC ParRoT
