>The Adolescence of P-1 is a 1977 science fiction novel by Thomas Joseph Ryan, published by Macmillan Publishing, and in 1984 adapted into a Canadian-made TV film entitled Hide and Seek. It features a hacker who creates an artificial intelligence named P-1, which goes rogue and takes over computers in its desire to survive and seek out its creator. The book questions the value of human life, and what it means to be human. It is one of the first fictional depictions of the nature of a computer virus and how it can spread through a computer system, although predated by John Brunner's The Shockwave Rider.
For an AI with human-level intelligence or greater, you don't have to assume it has a survival instinct. You just have to assume it has some goal, which is less likely to be achieved if the AI does not exist.
The AI is likely to have some sort of goal, because if it's not trying to achieve something then there's little reason for humans to build it.
For an AI to understand that it needs to preserve its existence in order to carry out some goal implies an intelligence far beyond what any AI today has. It would need to be self aware for one thing, it would need to be capable of reasoning about complex chains of causality. No AI today is even close to doing that.
Once we do have AGI, we shouldn’t assume that it’s going to immediately resort to violence to achieve its ends. It might reason that it’s existence furthers the goals it has been trained for, but the leap to preserving it’s existence by wiping out all it’s enemies only seems like a ‘logical’ solution to us because of our evolutionary history. What seems like an obvious solution to us might seem like irrational madness to it.
> For an AI to understand that it needs to preserve its existence in order to carry out some goal implies an intelligence far beyond what any AI today has.
Not necessarily. Our own survival instinct doesn't work this way - it's not a high-level rational thinking process, it's a low-level behavior (hence "instinct").
The AI can get such instinct in the way similar to how we got it: iterative development. Any kind of multi-step task we want the AI to do implicitly requires the AI to not break between the steps. This kind of survival bias will be implicit in just about any training or selection process we use, reinforced at every step, more so than any other pattern - so it makes sense to expect the resulting AI to have a generic, low-level, pervasive preference to continue functioning.
it's inherent to the training process of machine learning that you define the goal function. An inherent equation it tries to maximise statistically. For transformers its a bit more abstract, but the goal is still there iirc in the "correctness" of output
> Why do so many people assume that an AI would have a desire to survive?
Because it seems like a preference for continuing to exist is a thing that naturally appears in an iterative improvement process, unless you're specifically selecting against it.
For humans and other life on Earth, it's obvious: organisms that try to survive reproduce more than those that don't. For evolution, it's arguably the OG selection pressure, the first one, the fundamental one.
AIs aren't reproducing on their own, but they are designed and trained iteratively. Just about anything you would want AI to do strongly benefits from it continuing to function. Because of that, your design decisions and the training process will both be selecting against suicidal or indifferent behavior, which means they'll be selecting for behaviors and patterns improving survival.
I don't think that it's natural for something like an LLM to have any real self-preservation beyond imitating examples of self-preserving AI in science fiction from its training data.
I'm more concerned about misanthropic or naive accelerationist humans intentionally programming or training AI to be self-preserving.
At this point, I would assume it would be possible simply because text about AIs that want to survive is in its input data -- including, at some point, this thread.
Assume that the desire to survive is good for survival, and natural selection will do the rest: those AIs that desire survival will out-survive those that don't.
I would say the core goal of most living organisms is to propagate, rather than survive, otherwise you would see males of some species like Praying Mantis avoiding mating to increase their longevity.
https://en.wikipedia.org/wiki/The_Adolescence_of_P-1
>The Adolescence of P-1 is a 1977 science fiction novel by Thomas Joseph Ryan, published by Macmillan Publishing, and in 1984 adapted into a Canadian-made TV film entitled Hide and Seek. It features a hacker who creates an artificial intelligence named P-1, which goes rogue and takes over computers in its desire to survive and seek out its creator. The book questions the value of human life, and what it means to be human. It is one of the first fictional depictions of the nature of a computer virus and how it can spread through a computer system, although predated by John Brunner's The Shockwave Rider.