> One side seems to have much better quality of evidence.
Since I AM an expert, I care a lot less about what surveys say. I have a lot more experience working on AI Safety than 99% of the ICM/NeurIPS 2021 authors, and probably close to 100% of the respondents. In fact, I think that reviewing community (ICML/NeurIPS c. 2020) is particularly ineffective and inexperienced at selecting and evaluating good safety research methodology/results. It's just not where real safety research has historically happened, so despite having lots of excellent folks in the organization and reviewer pool, I don't think it was really the right set of people to ask about AI risk.
They are excellent conferences, btw. But it's a bit like asking about cybersecurity at a theoretical CS conference -- they are experts in some sense, I suppose, and may even know more about eg cryptography in very specific ways. But it's probably not the set of people who you should be asking. There's nothing wrong with that; not every conference can or should be about everything under the sun.
So when I say evidence, I tend to mean "evidence of X-risk", not "evidence of what my peers think". I can just chat with the populations other people are conjecturing about.
Also: even in this survey, which I don't weigh very seriously, most of the respondants agree with me. "The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%", but I bet if you probed that number it's not based on anything scientific. It's a throw-away guess on a web survey. What does 5% even mean? I bet if you asked most respondents would shrug, and if pressed would express an attitude closer to mine than to what you see in public letters.
Taking that number and plugging it into a "risk * probability" framework in the way that x-risk people do is almost certainly wildly misconstruing what the respondents actually think.
> "All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive!
I totally understand this sentiment, and in your shoes my personality/temperament is such that I'd almost certainly think the same thing!!!
So I feel bad about my dismissal here, but... it's just not true. The critique of x-risk isn't self-interested.
In fact, for me, it's the opposite. It'd be easier to argue for resources and clout if I told everyone the sky is falling.
It's just because we think it's cringey hype from mostly hucksters with huge egos. That's all.
But, again, I understand that me saying that isn't proof of anything. Sorry I can't be more persuasive or provide evidence of inner intent here.
> What would you consider evidence of a significant AI risk?
This is a really good question. I would consider a few things:
1. Evidence that there is wanton disregard for basic safety best-practices in nuclear arms management or systems that could escalate. I am not an expert in geopolitics, but have consulting some on safety, and I have seen exactly the opposite attitude at least in the USA. I also don't think that this risk has anything to do with recent developments in AI; ie, the risk hasn't changed much since the early-mid 2010s. At least due to first order effects of new technology. Perhaps due to diplomatic reasons/general global tension, but that's not an area of expertise for me.
2. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, particularly by non-state actors and particularly if the system is available outside of classified settings (ie, I have less concern about simulations or models that are highly classified, not public, and difficult to interpret or operationalize without nation-state/huge corp resources -- those are no different than eg large-scale simulations used for weapons design at national labs since the 70s).
3. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, by any type of actor, in a way that isn't controllable by a human operator (not just uncontrolled, but actually not controllable).
4. Specific evidence that an AI System can be used to persuade a mass audience away from existing strong priors on a topic of geopolitical significance, and that it performs substantially better than existing human+machine systems (which already include substantial amounts of ML anyways).
I am in some sense a doomer, particularly on point 4, but I don't believe that recent innovations in LLMs or diffusion have particularly increased the risk relative to eg 2016.
Which has zero guardrails for materials not in a blacklist (see page 18). It's too easy to see how to bypass the filter even for known materials, and it definitely won't help with new materials.
I see nothing here that can't be replicated in principle by a not too large non-state actor. It's not like some people haven't already tried this when AI didn't exist in its modern form:
These are things you can just look up. The LLM contributes nothing that the internet didn't already provide. Or even textbooks before that. What I mean is actually new capabilities, which weren't available to non-state actors with a broadband connection before.
Also, sarin gas doesn't pose an existential risk to humanity. Or, in the sense that it could, the cat's out of the bag and I'm not sure why we're talking about LLMs as that seems like a dangerous distraction from the real problem, right?
> I see nothing here that can't be replicated in principle by a not too large non-state actor. It's not like some people haven't already tried this when AI didn't exist in its modern form
To my point, they didn't "try"!!!
They DID.
Without LLMs.
Resulting in 13 deaths and thousands of injuries.
I'm not sure that LLMs significantly increase the attack surface here. Possibly they do, and there are some mitigations we could introduce such that the barrier to using LLMs for this sort of thing are higher than the barrier to doing the bad thing in the first place without LLMs. But even in that case, it's not existential. And nowhere I have stated LLMs don't pose risks; they do. My issue is with AGI and x-risk prognostications.
>These are things you can just look up. The LLM contributes nothing that the internet didn't already provide
The internet doesn't allow one to design _new_ weapons, possibly way more effective (which if you read carefully the first story, this one does).
>I'm not sure that LLMs significantly increase the attack surface here.
Being able to ask an AI to develop new deadly varieties which we'll not be able to detect or cure, and may be easy to produce, doesn't increase attack surface?
>Also, sarin gas doesn't pose an existential risk to humanity
Is x-risk the only thing we care about? The entire thread started with arguing x-risk is a distraction. I would be very slightly more comfortable with that argument if people took 'ordinary' risks seriously.
As it is, all camps have their heads in the sand in different ways. The illusion here is that AI advancement changes nothing, so the only thing worth discussing are variations of the current culture war issues, when obviously it does change everything even if completely put aside AGI/alignment arguments. e.g. If labour won't matter for productivity that has very grim political implications.
>To my point, they didn't "try"!!! They DID.
That's the point. Give me an x-risk scenario the doomers warn about, and I'll find you a group of humans which very much want the exact scenario (or something essentially indistinguishable for 99% of humanity) to happen and will happily use AI if it helps them. Amusingly, alignment research is unlikely to help there - it can be argued to increase the risk from humans.
>there are some mitigations we could introduce such that the barrier to using LLMs for this sort of thing
There are many things we can do in theory to mitigate all sorts of issues, which have the nice property of never ever being done. e.g. Yud's favorite disaster scenario appears to be a custom built virus. This relies on biolabs accepting random orders, which leaves the question of why are we allowing this at all (AI or not)? There's no good reason for allowing most crypto to exist, given its current effects on society, even before AGI comes into question, yet we allow it for what reason exactly?
If there's any risk here at all, we can safely rely on humanity doing nothing before anything happens - but in the case of x-risk actually existing, there's no reason to assume we'll have a second chance.
This entire category falls into my "numerical siulations at national labs" category of "don't care".
If you wanted to use a bioweapon to kill a bunch of people, you would ignore the DeepCE paper and use weapons that have existed for decades. Existing weapons would be easier to design, easier to manufacture, easier to deploy, and more effective at killing.
Computational drug discovery is not new, to put it mildly, and neither is the use of computation to design more effective weapons. Hell, the Harvard IBM Mark I was designed to help with the Manhattan project. There are huge barriers to entry between "know how to design/build/deploy a nuke/bioweapon" and "can actually do it".
And that's how I feel about AI-for-weapons in general: the people who it helps can already make more effective weapons today if they want to. It's not the risk of using WMDs doesn't exist. It's that WMDs are already so deadly that our primary defense is just that there's a huge gap between "I know in principle how to design a nuke/bioweapon" and "I can actually design and deploy the weapon". I don't see how AI changes that equation.
> Is x-risk the only thing we care about? The entire thread started with arguing x-risk is a distraction. I would be very slightly more comfortable with that argument if people took 'ordinary' risks seriously.
Discussion of x-risk annoys me precisely because it's a distraction from working on real risks.
> That's the point. Give me an x-risk scenario the doomers warn about, and I'll find you a group of humans which very much want the exact scenario (or something essentially indistinguishable for 99% of humanity) to happen and will happily use AI if it helps them. Amusingly, alignment research is unlikely to help there - it can be argued to increase the risk from humans.
Right, but
1. those humans have existed for a long time,
2. public models don't provide them with a tool more or less powerful than the internet, and
3. to the extent that models like DeepCE help with discovery, someone with the knowledge and resources to actually operationalize this information wouldn't have needed DeepCE to do incredible amounts of damage.
Again, I'm not saying there is no attack surface here. I'm saying that AI doesn't meaningfully change that landscape because the barrier to operationalizing is high enough that by the time you can operationalize it's unclear why you need to model -- that you couldn't have made the a similar discovery with a bit of extra time or even just used something off the shelf to the same effect.
Or, to put it another way: killing a ton of people is shockingly easy in today's world. That is scary. But x-risk from superhuman AGI is a massive red herring, and even narrow AI for particular tasks such as drug discovery is honestly mostly unrelated to this observation.
>>there are some mitigations we could introduce such that the barrier to using LLMs for this sort of thing
>There are many things we can do in theory to mitigate all sorts of issues, which have the nice property of never ever being done.
Speak for yourself. Mitigating real risks that could actually happen is what I work on every day. The people advocating for working on x-risk -- and the people working on x-risk -- are mostly writing sci-fi and doing philosophy of mind. At a minimum it's not useful.
Anyways, at the very least, even if you want to prevent these x-risk scenarios, then focusing efforts on more concrete safety and controllability problems is probably the best path forward anyways.
>There are huge barriers to entry between "know how to design/build/deploy a nuke/bioweapon" and "can actually do it".
>public models don't provide them with a tool more or less powerful than the internet
> I'm saying that AI doesn't meaningfully change that landscape because the barrier to operationalizing is high enough that by the time you can operationalize it's unclear why you need to model -- that you couldn't have made the a similar discovery with a bit of extra time or even just used something off the shelf to the same effect.
Your expertise is in AI, but the issues here aren't just AI, they involve (for example) chemistry and biology, and I suggest speaking with chemists and biologists on the difference AI makes to their work. You may discover the huge barrier isn't that huge, and that AI can make discoveries easier in ways that 'a little extra time' is strongly underselling (most humans would take a very long time searching throughout possibility-space, such a search may well be detectable since it will require repeated synthesis and experiment...). Also, to borrow an old Marxist chestnut: A sufficient difference in quantity is a qualitative difference*. Make creating weapons easy enough and you get an entirely different world.
I get your issues with the 'LessWrong cult', I have quite a few of my own. However, that doesn't make the risks nonexistent, even if we were to discount AGI completely. Given what I see from current industry leaders (often easily bypassed blacklisting) I'm not so impressed with the current safety record. I fear it will crack on the first serious test with disastrous consequences.
* There's a smarter phrasing which I can't find or remember.
Presumably the experts who feel differently about AI x-risk have similar disregard for your opinion. That you aren't persuaded by experts who disagree doesn't show anything.
Similarly, you think that x-risk dismissal among the experts you agree with is not self-interested - but it's not like the oil and gas engineers dismissing climate change risks would describe themselves as self-interested liars either. They would likely say, as you realize, "No, my dismissal of the concerns of other experts is legitimate!"
The thing is, it doesn't require a lot of expertise to understand that there is actually an enormous risk and "experts" denying that are simply burying their head in the sand. AI, I use the term expansively, is making huge and rapid progress. We are approaching "human-level" intelligence and there is no guarantee, nor even any indication, that "human-level" is an upper limit. Systems as smart or smarter than we are, that are not controlled for the benefit of humanity, are an existential risk.
There are two basic kinds of risk. First, an AI powered tyranny where a small number of humans dominate everything enabled by AI systems they control - every security camera watched by a tireless intelligence, drones piloted by perfectly loyal intelligences, etc. Second, that the AI systems are not effectively controlled and pursue goals or objectives that are incompatible with human existence. In either case the fact that the AI brings superhuman intelligence to bear means that humanity at large will be overmatched in terms of capabilities.
It is completely possible that neither of these scenarios come to pass, but the fact that one of the scenarios might come to pass is what makes the situation an existential risk - humanity eternally subjected by an irreversible dictatorship, or simply destroyed.
I'm confused about your first item that would convince you of AI risk, but I think the others should all be considered met by ChatGPT. Of course, ChatGPT isn't currently useful in developing new WMDs - but it is absolutely useful in helping to bring someone up to speed on new topics in many different domains. As capabilities advance, why wouldn't ChatGPT using GPT-5, or 6, or 10 be able to helpfully guide the creation of new and better WMDs?
AI isn't currently causing the problems you are worried about but certainly seems on track to do so soon. "Risk" doesn't mean that we are currently in the process of being destroyed by AI, but it does mean that there is a non-negligible possibility that we will soon find ourselves in that process.
> Presumably the experts who feel differently about AI x-risk have similar disregard for your opinion. That you aren't persuaded by experts who disagree doesn't show anything.
Of course. I was merely explaining the reason why my response would focus on x-risk rather than better surveys, and why.
> but it's not like the oil and gas engineers dismissing climate change risks would describe themselves as self-interested liars either. They would likely say, as you realize, "No, my dismissal of the concerns of other experts is legitimate!"
Okay. But I have more to gain than to lose by advocating for AI Safety, since that's what I work on.
> The thing is, it doesn't require a lot of expertise to understand that there is actually an enormous risk
My comment was about X-risk and AGI. Find any of a myriad of comments here where I agree there are real risks that should be taken seriously. Those risks -- the reasonable ones -- are not existential and have nothing to do with AGI.
> Systems as smart or smarter than we are, that are not controlled for the benefit of humanity, are an existential risk...
I am not an adherent to this religious dogma and have seen no empirical evidence that the powerful rationalisms people use to talk themselves into these positions has any basis in current or future reality.
I want the LessWrong cult to have fewer followers in the halls of power precisely because I actually do give a damn about preventing the worst-case FEASIBLE outcomes, which have nothing to do with AGI bullshit.
BTW: if we're all too busy worrying about AGI who's going prevent the pile of ridiculous should-be-illegal bullshit that's about to cause a bunch of real harm? No one. If you want conspiracies about intent, look at who's funding all this AGI x-risk bullshit.
And you can be sure as hell of one thing: no one in big tech wants me giving Congress advice on what our AI and data privacy regulations should look like. They would MUCH prefer endless hearings on "risks" they might even know are bullshit nonsense.
But why focus any effort on 100% risks of data privacy instrusions, copyright infringement, wanton anti-trust violations, and large-scale disinfo campaigns when there's a 1% risk of extinction, right?
But anyways. I'm just some fool who has actually spent over a decade in the trenches trying to prevent specific bad outcomes. I'm sure the sci-fi essays and prognostications from famous CEOs are far more persuasive and entertaining and convicting than the ramblings of some nobody actually try to fix real problems with mostly boring solutions.
The reason you aren't persuasive is because you just ignore arguments without responding to them. At least, you've done so here - perhaps you've responded other places with your decade of experience. When presented with surveys and claims of experts, you just dismiss those out of hand "I'm an expert too, therefore ignore other experts." When presented with an argument you just say "That's religious dogma and I reject it." I can't stop you from rejecting things out of hand, but you shouldn't be surprised that you come off as totally unpersuasive when you say and do these things.
In the simplest form:
1. AI could become smarter than humanity.
2. AI is rapidly progressing towards super-human intelligence.
3. If AI is smarter than humanity it could destroy or dominate humanity.
4. We aren't certain AI won't destroy or dominate humanity.
Therefore, there is some non-negligible existential risk from AI.
Which of these points do you disagree with, or do you think the conclusion doesn't follow?
Those first two points are pretty wild assertions that are offered without much compelling evidence.
The final two points are more plausible, although the last one is sort of tautological since any risk is by definition something we aren't certain won't happen. However part of their plausibility as risks is because the first two points are not anywhere near our current state, and therefore it's unclear to us how to clearly evaluate risks that are based on unknown and possibly fictional contexts.
I think it's surprising that you think the first two points are controversial. They seem quite obvious to me.
First, we know of no special requirement that exists in human brains that provides intelligence that machines lack. In other words, We don't have any reason to expect that machines will be limited by human intelligence or some level below. On the contrary, we have great reasons to expect that machines will easily be able to exceed human intelligence - for example, the speed and reliability of digital computation or the fact that computers can be arbitrary sizes and use amounts of power and dump waste heat that our biological brains couldn't dream of. If you accept that AI is making progress along a spectrum of intelligence, moving closer towards human level intelligence now, then it seems absurd to doubt that it would be possible for AI to surpass human intelligence. Why would that be impossible? It's like claiming we could never build a machine larger than a human or stronger than a human. There's no reason or evidence to support such a claim.
Second is the idea that AI is making progress towards human and superhuman intelligence. I think you should be convinced of this by simply looking at the state of the art 5 years ago versus today. If you put those points on a plane and draw a line between them, where is that line in 5 years or 10?
Today GPT4 can play chess, write poetry, take standardized tests and do pretty well, answer math problems, write code, tell jokes, translate languages, and just generally do all sorts of cognitive tasks. Capabilities like these did not exist five years ago or to the extent they did, they existed only in rudimentary forms compared to what GPT4 is capable of. We can see similar progress in different domains, not just large language models - for example, image generation or recognition.
Progress might not continue then again it might. It might accelerate. As the capabilities of the models increase, they might contribute to accelerating the progress of artificial intelligence.
What is an AI's motivation to "destroy or dominate humanity"? Why would it prefer any specific outcome at all?
If we ever see tribal identity or a will to live as emergent properties of an AI, then things get more interesting.
That would lead quickly to a whole raft of worldwide legal restrictions around the creation of new consciousnesses. Renegade states would be coerced. Rogue researchers would be arrested, and any who evaded detection would be unable to command the resources necessary to commit anything more than ordinary limited terrorism.
The only plausible new risk, I think, is if a state were to lose control of a military AI. But that's movie plot scenario stuff -- real weapons have safety protocols. An AI could theoretically route around some integrated safety protocols (there would be independent watchdog protocols too), but going back to my first point, why would it?
Could you elaborate on 3?
It's easy to envision how AI can be used by $powerful to improve its control of $powerless, even before AGI, but how an AI would grab power from $powerful and then behave even worse is harder to imagine.
Remember that an AI doesn't have to grab power or be sentient or be an agent of any kind. AI could lead to the destruction or domination of humanity, even if AI is simply a passive tool completely obedient to its operators.
Suppose GPT5 or 6 is multimodal and extremely intelligent. Any work that could be done remotely at a computer could be done by the superhuman intelligence of GPT 6. This is everything from creating music and movies, virtual YouTubers and live streamers, to customer support agents, software developers and designers, most legal advice, and many more categories besides. Providing all of these jobs at extremely low cost will make Open AI exceedingly rich - what if they want to get richer? How about robotics? The AI can help them not only design and iterate and improve on robots and robot hardware, but the AI can also operate the robots for them. Now they can not only do the kind of jobs that can be done on the computer. They can do the jobs that require physical manipulation as well and that category of jobs extends to things like security, military, police.
At this point there are at least two possibilities. First is that the AI is effectively controlled by OpenAI in which case the decision makers that open AI would have effective and lasting control over all of humanity. No plucky band of rebels could possibly overthrow the AI powered tyranny - That's the strength of relentless super intelligence. The second possibility is that open AI doesn't have it well controlled - that could mean bugs or unexpected behavior or it could mean the model expressing some kind of agency of its own - think of the things that the Bing AI was saying before Microsoft got it mostly under control. If the AI isn't under control, it may choose to eliminate humanity simply because we might be in its way or we might be an impediment to its plans.
Since I AM an expert, I care a lot less about what surveys say. I have a lot more experience working on AI Safety than 99% of the ICM/NeurIPS 2021 authors, and probably close to 100% of the respondents. In fact, I think that reviewing community (ICML/NeurIPS c. 2020) is particularly ineffective and inexperienced at selecting and evaluating good safety research methodology/results. It's just not where real safety research has historically happened, so despite having lots of excellent folks in the organization and reviewer pool, I don't think it was really the right set of people to ask about AI risk.
They are excellent conferences, btw. But it's a bit like asking about cybersecurity at a theoretical CS conference -- they are experts in some sense, I suppose, and may even know more about eg cryptography in very specific ways. But it's probably not the set of people who you should be asking. There's nothing wrong with that; not every conference can or should be about everything under the sun.
So when I say evidence, I tend to mean "evidence of X-risk", not "evidence of what my peers think". I can just chat with the populations other people are conjecturing about.
Also: even in this survey, which I don't weigh very seriously, most of the respondants agree with me. "The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%", but I bet if you probed that number it's not based on anything scientific. It's a throw-away guess on a web survey. What does 5% even mean? I bet if you asked most respondents would shrug, and if pressed would express an attitude closer to mine than to what you see in public letters.
Taking that number and plugging it into a "risk * probability" framework in the way that x-risk people do is almost certainly wildly misconstruing what the respondents actually think.
> "All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive!
I totally understand this sentiment, and in your shoes my personality/temperament is such that I'd almost certainly think the same thing!!!
So I feel bad about my dismissal here, but... it's just not true. The critique of x-risk isn't self-interested.
In fact, for me, it's the opposite. It'd be easier to argue for resources and clout if I told everyone the sky is falling.
It's just because we think it's cringey hype from mostly hucksters with huge egos. That's all.
But, again, I understand that me saying that isn't proof of anything. Sorry I can't be more persuasive or provide evidence of inner intent here.
> What would you consider evidence of a significant AI risk?
This is a really good question. I would consider a few things:
1. Evidence that there is wanton disregard for basic safety best-practices in nuclear arms management or systems that could escalate. I am not an expert in geopolitics, but have consulting some on safety, and I have seen exactly the opposite attitude at least in the USA. I also don't think that this risk has anything to do with recent developments in AI; ie, the risk hasn't changed much since the early-mid 2010s. At least due to first order effects of new technology. Perhaps due to diplomatic reasons/general global tension, but that's not an area of expertise for me.
2. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, particularly by non-state actors and particularly if the system is available outside of classified settings (ie, I have less concern about simulations or models that are highly classified, not public, and difficult to interpret or operationalize without nation-state/huge corp resources -- those are no different than eg large-scale simulations used for weapons design at national labs since the 70s).
3. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, by any type of actor, in a way that isn't controllable by a human operator (not just uncontrolled, but actually not controllable).
4. Specific evidence that an AI System can be used to persuade a mass audience away from existing strong priors on a topic of geopolitical significance, and that it performs substantially better than existing human+machine systems (which already include substantial amounts of ML anyways).
I am in some sense a doomer, particularly on point 4, but I don't believe that recent innovations in LLMs or diffusion have particularly increased the risk relative to eg 2016.