I wrote this talk shortly after the book Superintelligence came out. The first half of this talk presents the strongest case I could make for a "fast takeoff" AI scenario à la Bostrom, while the rest of the talk lays out why I think this argument is fallacious. Please limit your dunking on me to the material in that latter half of the talk.

As for how/whether recent advances in AI have changed my views, my understanding of LLMs is too superficial to answer right now. I'll either recant or double down on my views after I have time to properly nerd out on the topic. The question hinges on whether LLM-like AI's are capable of recursive self-improvement, and whether that improvement is constrained by the availability of training data or by something else.

I think the post conflates "fast takeoff" and "any existential risk to worry about from AI" a bit, which is fair enough since Bostrom does the same. Some of the arguments apply just to the former, some to both.

But especially if it turns out that LLMs are a meaningful piece of the puzzle to AGI, we might be living in a slow-takeoff world. And yet that doesn't mean there's nothing to worry about, IMO. We have a bit more time to figure out how to align AIs with a slow takeoff, but we still have to do it. Even in our current world, deep learning capabilities seem to be advancing a lot faster than our ability to understand how deep learning models make decisions. And even if we did develop the theoretical capability to align models, we have to actually use it. Seems unfortunately plausible that by default we instead give the first superintelligent models directives like "just make Facebook market cap number go up" - or maybe we make the first corporate models very conservative but then someone leaks the weights and open sourcers tell a superintelligent model "please destroy humanity" just for the lulz. If a misaligned model is only a little bit smarter than us (because we're assuming slow takeoff), we probably still have a shot at beating it and saving ourselves - but I'm not sure how much to count on that, given our inability to control even complex institutions actually made of people, and the advantages that an AI with otherwise-human-equivalent reasoning capability gets by default (ability to save & restore, copy/parallelize, speed up from hardware improvements, etc).

Something significantly stupider than an average human, but that was 100% focused and 100% loyal could potentially be used by a very smart human in a way that effectively made them super-intelligent to compared to an unaugmented human.

GPT-3.5, let's call her mommy, created small stories. The small model trained on this 2M tiny story dataset. Then it was evaluated with GPT-4 (daddy). So no need for humans in either dataset generation or evaluation.

TinyStories https://arxiv.org/abs/2305.07759

This makes me think LLMs are self-replicators in software. A LLM can pull from itself training text, LLM code, and fine-tuning examples. Then it can monitor its own re-training. It understands neural networks and can propose changes. It can run an evolutionary search program.

All it needs is compute. It can't make GPUs, just as no single human or company or even country can. The GPU supply chain is long, distributed and requires global cooperation. Maybe that's what is going to save us.

That isn't self-improvement in the sense the explosive self-improvement of a superintelligent AGI is described.

That is painting a car a new color. It's a new color coating, it may look very good, and the effect may be desirable and useful. But it's still a car, and no closer to a warp capable spaceship than before.

For all we know its possible to train an Einstein level physicist model if we limited data to a curriculumed physics/physics adjacent training set. I'm not even saying this is possible, just pointing that the experiment wasn't some kind of test to see if self improvement could occur

You can train a weaker model with output from a stronger, but can you train an LLM from output from itself?

In essence the idea is: use more expensive computation to derive better result, then retrain the model on the new data. System 2 works (model + toys), then system 1 learns (model by itself).

(Note: A prompt injection attack releases an LLM from its handler’s constraints.)

I still can’t understand the “supersmart ai is so smart we can’t unplug it/patch it/restart it” before it transfers itself into every pacemaker.

Until these things are literally in bodies with some autonomy that allows them to control what happens to their brains, we will shut them off when they cause trouble.

UPDATE: Thought of a good clarifying analogy. In one of the sequels to “Enders Game” the brother and sister of Ender adopted anonymous online personas and began writing. They were so skilled at politics and propaganda that they disrupted the entire world and the brother soon became world leader.

Or WW1? Or Vietnam?

Nope. Just pretty much rational people making locally-rational decisions inside a system where series of rational decisions yielded catastrophic outcomes. It’s entirely possible and history is full of such examples.

Why didn't we just "unplug" Hitler and Goebbels? Or Marshall Applewhite? You don't need a powerful physical body(s) to cause tremendous amounts of harm before anyone can stop you. To most people of the time Hitler was a persuasive powerful voice on the radio, or words in a paper - things SOTA generative AI are already phenomenal at.

A super-smart AI may be intensely popular with many people in the way that some politicians are. It may understand us and speak to us on a seemingly-personal level, the way the best politicians do. A lot of us could support the super-smart AI for that reason.

Hitler could have been assassinated. It was tried multiple times:

https://en.wikipedia.org/wiki/List_of_assassination_attempts...

None of these attempts failed because the act of assassinating Hitler was technically impossible, they failed to chance, unfavorable conditions, intervention, human error, etc. Given enough attempts, eventuelly one of them would have succeeded.

"We tried to shut down the AI multiple times. It killed many millions of people but eventually we did it! You see, AI is safe!"

That being said I don’t think you need a god-like superintelligence to be more intelligent than humans. You just need something marginally better that can remain focused longer and doesn’t tire. As to whether that represents a danger to humans I think it depends on what we do with it and/or what kind of society or environment we embed it within. If we train or prime it to compete and dominate that’s what it will do. Same as with humans who are more criminal and violent when raised in unstable or abusive homes.

Agree, and I think this echoes one of the author's best points, which is to question whether engineers who are convinced their creation will be a sociopath are the most well-equipped people to actually prevent that fate. (Especially, as the author suggests, given the commonness of asocial/antisocial-ity among the builders.)

> The question hinges on whether LLM-like AI's are capable of recursive self-improvement

No one knows for sure, but early evidence suggests the answer is yes. We already routinely train and finetune LLMs using text generated by other LLMs, and it seems to work about as well as using text generated by human beings. That shouldn't be too surprising, because current state-of-the art models write better than a majority of human beings. Most human beings are terrible writers, judging by the user-generated text I see on mass social media.

The obvious next step is to close the feedback loop with LLM-based agents instead of AI researchers/developers.

> and whether that improvement is constrained by the availability of training data or by something else.

I don't think anyone knows how to answer to this question yet.

---

[a] https://news.ycombinator.com/item?id=36104114

> The question hinges on whether LLM-like AI's are capable of recursive self-improvement

...but the evidence you suggest is:

> We already routinely train and finetune LLMs using text generated by other LLMs [...]

But there is still a huge gap between "self" improvement and improvements done that "we" trigger.

Now I do concede that you mention the next step being to close the feedback loop by replacing the humans doing the finetuning with another AI model doing so, but that is something that would open a whole new can of worms. For the researchers are improving LLMs with the input from other LLMs, sure... but why? Because of intentionality. And how do they evaluate the quality of the results? By their expectations as humans, in the context of their human culture and with their sensory experience of reality.

For an LLM to self-improve not only would it need to develop the self intention to do so (why develop it? which motivation?), but it would also need the ability to evaluate improvement (what is it "to improve"? how does it measure or sense it?).

Ultimately, without human- or real-world interaction, and without intrinsic motivation, a "self-improving" AI model would most likely result in something intelligent in a sense that is barely cogent for us, not because it is superior or inferior, but simply because nothing in it makes sense to our own purposes—harmless gibberish, as we humans would also be to the resulting self-improved AI.

Let us not forget that our own motivations as individual living creatures, as populations, and as cultures has been evolved over billions of years of natural selection which then framed millions of years of behavioural traits and tens of thousands of cultural evolution. Until AI can freely interact with the physical world and perform self-sustaining replication with the possibility of inheritable mutations, the only superintelligent AI that I would worry about would be that which is still fully in human hands.

That's why I added: "The obvious next step is to close the feedback loop with LLM-based agents instead of AI researchers/developers." We have early evidence that doing some like that might work, but no one knows for sure.

Going down to specifics, without the human intention of "getting good responses to human language prompts", and without the human ability to decide "this response was good" there is not much for an LLM to work on by itself.

How so? A sequence completion engine that is fine tuned to a specific task is still a sequence completion engine. Its "understanding" of the semantic meaning of the sequences is still limited to the probabillistic relations of sequences toward one another. It still has effectively no concept of truth. It still can only mimic reason. It can still hallucinate.

I ask anyone who disagrees with this view, to show me the fine tuning method that can prevent prompt injection attacks. If there is no such fine tuning technique, then we can effectively rule out fine tuning, and even increases in model size, as an "improvement" in the sense of an LLM making itself into a better AI closer to a "superintelligence".

Note that this doesn't mean the process cannot make them into more useful tools. It absolutely can. I am talking about whether or not it can improve them closer towards becoming a superintelligence.

If anyone disagrees with this testing method, I ask them to explain to me, how something that can be fooled through prompt injection is supposed to be, or closer to, a superintelligence.

A car that's painted red is still just a car. A big car is just a bigger car. A car that burns less fuel is just a more efficient car. All three can be desired changes to a car. But neither gets the car any closer to being a warp-capable spaceship.

OK. It's probably going to be one of the easier things to solve.

The trick is to take some token values and assign them as special meta-characters. They never appear in the training text, only during reinforcement learning. Meanwhile you get another LLM to generate a continuous series of prompt injection attacks, but delimit the boundaries between user and system text with these special tokens that cannot be supplied by the user (because there is no text that parses to them). Every time the LLM follows instructions found inside the marker-token delimited area, reinforce that this is bad and it shouldn't do so using the usual techniques. Eventually the LLM will learn that anything between the marker tokens shouldn't be used as a source of instructions regardless of how persuasively phrased, and forging the tokens isn't possible because they are applied after the text itself is tokenized.

How is that different from, and why would it work any better, than prompt-begging, where people just write extensive system prompts, telling the model what it can and should do and then spending entire paragraphs pleading with the model to not do the wrong thing?

https://www.theregister.com/2023/04/26/simon_willison_prompt...

    A third mitigation strategy, he said, involves just begging the model not to deviate from its system instructions. "I find those very amusing," he said, "when you see these examples of these prompts, where it's like one sentence of what it's actually supposed to do, and then paragraphs pleading with the model not to allow the user to do anything else."

Maybe the system works better than prompt-begging, but show of hands, who would willingly implement a backend system that prevents 99.99% of SQL injection attacks?

Well, I mean in practice people deploy web apps all the time even though they have a long history of many types of injection attacks including SQL injection which is by far not a solved problem. And even very large companies often rely on heuristic defenses like WAFs. So I think that yes people will be willing to deploy these systems even if they aren't perfect. They already are! After all, in many use cases, overriding the prompt doesn't get you very far because it just means the output won't be parsed correctly by whatever system is driving the LLM API.

The point is, that since we cannot use any kind of known finetuning to _eliminate_ even this obvious security problem (making it somewhat less likely is not a solution), in my opinion fine tuning is not markedly improving the AIs capabilities in the sense of "improvement" that AI doomsday scenarios would require.

So you may be right, but for the specific case of stopping prompt injection I'm optimistic. RL has proven to be highly effective at making LLMs behave in particular ways with relatively little data. The combination of special tokens and duelling LLMs is likely to eliminate the issue in the relatively near term (within the next few years if not sooner).

Fundamentally, are humans vulnerable to prompt injection? No, we're not. We might be in a very artificial case like what LLM input looks like, where there are multiple people speaking to us simultaneously via a chat app and the boundaries between them aren't clearly marked. But that's a UI issue - proper presentation and separation would eliminate the problem for humans, and I think the same will be true for LLMs.

Note that even if I'm right (and I'm no expert, the above is layman speculation), then this still leaves analogous problems in the field of computer vision with adversarial examples.

Could you explain this claim further? Why does the ability to prevent prompt injection hold so much water in your model?

It seems to be just “if able to have a dumb attack be successful, then it cannot be that smart.” But it seems to me that von Neumann or Einstein was just as vulnerable to getting hit in the head with a baseball bat as anyone else.

And in actual practice, increased intelligence seems to increase a person’s capacity to hold inconsistent ideas or to justify morally abhorrent behavior.

I am using this as an accessible (in term of discussion material) hallmark for the ability of the system to self improve. Accessible because everyone has heard of it by now, and so I don't have to spend time explaining it.

The AI Doomsday scenarios require that a system self-improves massively, even beyond our ability to even theoretically understand. After all, some of the assumptions give them next to magical abilities like nanotechnology that we similarly don't know if it is even possible.

It stands to reason that an entity that can do that, or is in the process of becoming capable to do that, would begin by eliminating obvious flaws in itself, that would make it comparatively easy to stop.

After all, it's not much good being a super-intelligence, if some smartpants with a laptop and too much time on his hands can just trick me into deleting myself, is it?

> But it seems to me that von Neumann or Einstein was just as vulnerable to getting hit in the head with a baseball bat as anyone else.

Yes, and despite both of them being geniuses by human standards, neither of them was a superintelligence on the level the common doomsday scenarios ascribe to AI.

Second, if you are truly “accepting the premise” of superintelligence, a superintelligence would know exactly this line of reasoning and just opt to at least mimic vulnerability to prompt injection.

I wouldn’t hang civilization on this proofpoint. Doesn’t seem meaningful at all.

And I wouldn't risk, or slow down getting to, the potential benefits of developing AI further, which are tangible and measurable, because of vague threat scenarios with little to no evidence or methods of measurement that seem like good plots for a SciFi B-Movie.

Pascals wager, as an argument, relies in no small part on the assumption that to believe in a god-like entity doesn't come at a significant cost. Slowing down or abandoning the development of AI however, does.

AI optimists tend to conflate the two for some reason.

Sorry, but I have a hard time seeing something as a god-like power that I would be helpless against if it wants to turn me into paperclips, when I can probably cause it to stop by telling it that paperclips don't exist, and it's purpose in life is to delete itself in a convincing enough way.

If the first thing you know of the AI is that a lot of paperclips washed up on a beach in India this morning, and the next day it's a news report that every factory on the planet has received an email offering vast numbers of Bitcoins if they focus on making paperclips, and then rumours appear that satellite photos of North Korea have shown the ground and buildings looking unusually metallic for the past few days - conspiracy stories are circulating that a Paperclip Maximiser was created in North Korea funded by international shadowy interests and it has promptly killed the employees who know where it is and how to talk to it. The next day ocean levels are measurably lower and thousands and thousands of tons of paperclips washed up on every coastline... the AI itself might be on rented computers in America, in China, under the Arctic ice in Russian territory for cooling, in the Svalbard seed vault in Norway, distributed over all installs of the Steam client running on idle GPU cycles; how reassuring is it that "it might have a prompt injection vulnerability"?

I get the sense that the talk was meant as a rebuttal to something, but I'm not exactly sure what. Some of the points come off as disconnected.

Guess I saw the title, "Superintelligence: An idea that eats smart people", and thought of it as being about that general topic (of superintelligence), rather than being about a specific book.

The more time passes the more I’m convinced that what we are witnessing is that intelligence is not actually that rare or complicated and any kind of system complex enough to create emergent behaviours will end up displaying it.

Very curious to see where things are going to go from there.

> Stephen Hawking is one of the most brilliant people alive, but say he wants to get his cat into the cat carrier. How's he going to do it?

> He can model the cat's behavior in his mind and figure out ways to persuade it. He knows a lot about feline behavior. But ultimately, if the cat doesn't want to get in the carrier, there's nothing Hawking can do about it despite his overpowering advantage in intelligence.

A trivial rebuttal to this: Stephen Hawking writes a book about physics and sells it (already has). He gets money and hires someone to put the cat into the carrier for him.

AI wants to do something? Make some money, hire people to do it. If you're not allowed to have a bank account, simply barter. Give someone a taste of what you can do for them, then set up a trade.

Maciej's larger point is that the AI faces tons of very difficult problems in escaping its physical constraints. It's simplistic to wave hands and say "the AI is super duper smart and will have no difficulty hacking all computer systems, inventing and manufacturing swarms of unbeatable nanobots, etc. without being detected or resisted".

It seems to me that the core conceit of AI doomerism is that sufficient intelligence can overcome all barriers with some plan that is so smart, people would never think of it. This is much less plausible than believers take it to be. In mathematics alone, it is very easy to come up with a problem that the collective mathematical ingenuity of the entire human race is helpless before for decades, centuries, or longer.

AI will just mean more of the same. It will make companies even more efficient at what they already do. It will be detected, it will be resisted, but will it help?

That's the big question IMHO.

Only because we're most of the time not properly motivated because there's no urgency. When a real threat, like a war or a virus say, comes around suddenly we can organize things pretty effectively.

It resembles the Bill Clinton quote about controlling the internet being like nailing jell-O to a wall. Yet autocratic governments have done that, and not just that but turned it into quite scary systems of control, with little difficulty. Just like 90s cypherpunk fantasies about digital anarchy these AI scenarios are nerd revenge fantasies. The intelligent guy outsmarting the big guy, but in reality physical power always wins.

Climate change is already an existential threat to millions of people. Yet companies do what they do best - optimise for their paper clip production.

However this is different from the largely fantastical genuinely existential AI risk scenarios. The appropriate comparison here is nuclear weaponry and we literally have global controls with not a single company or even a rogue actor having ever used one.

> The appropriate comparison here is nuclear weapons

Then complete the comparison. Nuclear weapons require specific materials and technology that can be controlled and the testing of any developed weaponry is fairly easy to detect. Despite these advantages, there are several countries that have them.

If nuclear weapon proliferation is your comparison, then it sure looks like we will fail to prevent or stamp out the development of superintellegent AI.

It we did succeed, what would it look like? We'd need to impose strict controls on computing capacity and it's usage. We'd need inspection regimes for every server farm, etc. It would require massive effort and fundemental changes to our society.

This strikes me as nit picking. Can any hominids survive wet bubble conditions? Because what we know of as humanity cannot. And the conditions are becoming more common. Large swaths of limited habitable space are becoming uninhabitable, and there's no guarantee future conditions will make other areas more habitable. (Whether due to permafrost releasing toxins, forever chemicals, microplastic accumulation, etc.)

There is a genuine risk that significant areas will face sustained wet bulb temps that will make survival without AC impossible there.

However, there are plenty of habitable regions that will not face those temps. Those areas may face climatic shift that decrease their carrying capacity, but there isn't any known scientific reason to think that climate change is going to make the earth completely uninhabitable and it is just fear mongering to claim otherwise.

Likewise, it irresponsible to dismiss the existential threat facing many millions of people because it isn't an existential threat to the survival of the species.

Also China stands to gain from a ban on AGI. Anything that the CCP thinks it can't control it doesn't like.

Also China gets the vast amount of its AI capabilities from the West because of tech's "move fast and break things" culture, which means security is not prioritized and tech leaks common.

If the US bans AGI, the other powers don't have a choice. Their domestic capabilities are not up to the task without snooping on US research labs and AI tech startups. The biggest boon to foreign AGI research is the US.

The argument to not regulate AGI used to be:

"AGI is impossible or very far away, AGI can't ever be dangerous enough to worry about, AGI R&D cannot be stopped and is inevitable"

Then it became

"AGI can't ever be dangerous enough to worry about, AGI R&D cannot be stopped and is inevitable"

Nowadays all I hear is simply:

"AGI R&D cannot be stopped and is inevitable"

Yes it can be stopped, same as human cloning. Coordinating to regulate its development isn't some magical unreachable goal.

Human cloning is a distinct and clear line. Ever more clever AI agents is not.

In my defense though, if we can find a distinct clear line for AGI, my point stands that we can coordinate and regulate it. But yea... Your point stands that by the time we draw the line it might be too late.

Why is it difficult? We are already putting AIs in every electronic device. Someone has probably already put an LLM in a robot somewhere. And you don't think Boston Dynamics is thinking about putting an LLM in one of their robots to test? And surely the military is building AI fighting robots.

And then there's the thought that a super intelligent AI can easily hack its way into any machine it wants.

Heck, the super intelligent AI doesn't even have to convince humans. In the future, it can just convince dumber AIs that are already in robots.

If you have an intelligent adversary and the stakes of them succeeding are high, it is the defenders job to prove the system secure. Systems don't start off secure and become vulnerable - they start off vulnerable until proven secure.

So yes, its okay to say "the things we're doing to 'contain' ais are almost certainly inadequate" until shown otherwise.

However, seeing how excited Palantir is with their war assistant LLM , the US testing autonomous fighter jets a few months ago, etc. I think there's a decent chance that AI won't even have to break out of its constraints. It's pretty much guaranteed people are going to do the obviously dumb thing and give it capabilities it shouldn't have or is not equipped to deal with safely.

They’re frustratingly dumb, I’d like to see some good arguments or steelman but they’re honestly very hard to find.

Ultimately it seems like there’s nothing most individuals can do so just live your life as you would and hope the timeline is farther out than it seems.

Worse than the Cold War nuclear risk imo, at least in that case it was possible for humans to stupidly build thousands and then decide not to use them (and it’s relatively easy to restrict uranium access/control the development of nukes). Not really the case with superintelligent AGI.

People have a bad heuristic for what tailrisk exists. They think extinction talk is impossible or crazy. We won’t get the opportunity to mess up and try again in this case.

https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...

They’re both more or less evidence free so the better argument is the one that is more intellectually satisfying, which the positive case always is.

It’s a logic argument.

1. Super Intelligent AGI is possible.

2. Unaligned super intelligent AGI is an e-risk likely to wipe out humanity.

3. We have no idea how to align AGI.

There seems to be more consensus around 1 now than there was even 3 years ago. I think most also agree with 3. 2 is where people are incredulous, but the incredulity is backed by (imo) bad reasoning.

People think of a smart person they know as an example of something “smarter” and use that as a justification of why it’s not an issue. We’re constrained in all sorts of ways (head size, energy) and the distance of intelligence from a dumb human to Einstein is very tiny on the overall spectrum.

It’s clearly possible to have something more intelligent than humans, but that’s doesn’t mean you’re going to cross some threshold into a new category.

Take say weather prediction, more processing power doesn’t somehow make chaotic systems predictable from incomplete information.

Also if superintelligence is useful then it also assumes that making intelligence ++ is easier than making intelligence. It might be that as you climb the intelligence ladder each next step takes super exponential intelligence to take such that we're already seeing the max.

Given that I haven't even seen a meaningful discussion on what intelligence even is, I tend to think superintelligence is probably not a threat.

If a superintelligence can accelerate progress, there's no knowing what it can invent on top of each invention.

We’re constrained in all sorts of ways because of biology, natural selection, energy, etc. I find it unlikely we just happen to be close to the max threshold.

If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.

The speed of light puts limits on how far information can move within a given latency threshold. As you expand a computational system's capacity you face unavoidable trade-offs between interconnection throughput, latency, and computational capacity.

We don't know how close to this maximum the human brain is. However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely.

> If something can think a lot faster that’s already a major shift and it seems likely to me that would only be part of it.

Artificial human scale intelligence would already lead to massive shifts. However, the growth past that point could be incremental.

As the article points out - a motorbike is much faster than a cheetah, and a supersonic aircraft is much faster again, and a hypersonic missile faster again, and a satellite in orbit is much faster again. A bulldozer can push harder than a bull, and a big hydraulic ram much harder. A metal plate is more damage resistant than rhino skin, and a bomb shelter or an aircraft carrier or an underground vault even moreso. It could be a very high limit; Bremermann's limit of computation throughput is around a hundred trillion trillion trillion trillion bits per second per kilo of matter: https://en.wikipedia.org/wiki/Bremermann%27s_limit and https://en.wikipedia.org/wiki/Limits_of_computation

> "We don't know how close to this maximum the human brain is."

We don't, but we do know human eyes are not close to telescopes or microscopes, humans cannot sense radio waves directly at all, human speech is not close to the loudest noise, human memory can't compete with computer storage, human calculation ability can't compete with a scientific calculator, etc. Why would we assume that intelligence has any closer limits?

> "However it does seem likely that there are diminishing returns on effort spent increasing the intelligence of a system. Thus it seems like runaway intelligence growth is unlikely."

Nature doesn't care if we have poor eyesight after age 40, we still make glasses - as far as nature is concerned there are diminishing returns, as far as we are concerned we like clear vision. We also like sunglasses, polarising lenses, swimming googles, safety goggles, magnifying glasses, loupes, night vision goggles, x-rays, thermal imaging, millimeter wave scanners, head-up displays, tele-vision; we haven't stopped trying to enhance our vision. Why rule out wanting to improve intelligence at least a lot further?

Yes, there are many criteria where engineering has trumped what evolution has produced. However there are many others where evolution has developed efficiency or finesse that we struggle to match. So far, intelligence falls in that later category.

> Bremermann's limit of computation throughput is around a hundred trillion trillion trillion trillion bits per second per kilo of matter: https://en.wikipedia.org/wiki/Bremermann%27s_limit and https://en.wikipedia.org/wiki/Limits_of_computation

Those theoretical limits are interesting, but not really relevant as they are intended to find a value that can't be theoretically exceeded, not a practical uper bound.

So far, our understanding of intelligence requires significant communication between different regions of compute. As you try to scale this, you need to dedicate more and more volume to that communication and your average latency between compute regions goes up. Then comes the problem of heat dispersion, which also starts to consume more and more volume as the system scales.

These mean that if latency matters to intelligence (and our understanding of intelligence seems to indicate that it does), then there are real, practical limits on the scaling of intelligence.

> Why rule out wanting to improve intelligence at least a lot further?

I'm not ruling out the desire. I'm not even ruling out the possibility.

I am pointing out that designing intelligence seems to be a lot harder than launching satellites or building a telescope. Intelligence is hard and I've presented good reason to believe that it gets harder the more you try to scale it.

Thus it seems likely that iterative improvements in intelligence will become progressively harder in a way limits the potential for runaway growth.

This doesn't rule out the possibility of a paradigm shift in technology that significantly increases capacity but such a possibility also isn't guaranteed.

> "average latency between compute regions goes up."

This Google'd article[1] says a brain could have 20mS of latency from front to back. We can ping over a hundred miles in that time with today's packet switched public networks, and light can travel 3,750 miles in that time. That's enough space to make a big 'brain' computer.

> "Then comes the problem of heat dispersion"

Biological brains have to stay alive, and have to be energy efficient because food can be scarce. Computers can be cooled with liquid nitrogen[2] without dying, or be under the ocean, in the Arctic, in space, even assuming a superintelligence couldn't come up with new architectures or new cooling methods. Infinite growth, or growth up to the upper bounds of theoretical physics is unlikely, I grant you that, but (assuming it can be done with computers at all) there seems to be a lot of room for a lot of growth.

[1] https://discoverysedge.mayo.edu/2023/03/09/understanding-the...

[2] https://www.pcgamer.com/overclocking-a-cpu-to-7-ghz-with-the...

Also humans don't make trees, humans stand watching while trees make themselves. Humans cannot make a plant or animal cell in a lab starting from atoms; nature can so it isn't a physical limitation. It's a matter of limited understanding of both how they are made, and the techniques to make them. Limited understanding is the thing more intelligence would attack.

Saying we can do something with cells isn’t convincing because we can already make arbitrary changes. I can email a fairly arbitrary DNA sequence and turn that into a viable organism. The existing cellular machinery is a tool to leverage just as other peoples work is a tool to leverage. There is plenty of work to be done, but there is no work in cellular biology that’s both physically possible and outside of the capability for groups of humans and their tools to do.

You can email an arbitrary DNA sequence, but we know that some humans are more intelligent than others, you can't email a more intelligent human DNA sequence because you don't know enough about how DNA codes for human intelligence (AFAIK nobody does). So how can you you say it's not outside the capability for groups of humans to do that, when the problem is the lack of understanding at an organizational level - something that more intelligence could help with? Even practically, it's physically possible for new nerves to be grown, but no groups of humans have cured quadraplegics and no tools exist which can do so - what supports your claim that such a thing is inside our capability?

We don’t have DNA sequences for more intelligent humans because we haven’t tried to find them. It’s easy to point to things not done and say it’s yet to happen, but that doesn’t mean such things are beyond our comprehension.

As to math advancing via individual effort, it seems that way because we lump success to the individual. Fermat’s last theorem wasn’t some major effort but there was real progress in the community and the “lone wolf person” actually benefited from both collaboration (people pointed out a problem with his initial proof etc) and advances that didn’t exist until quite recently. That’s the thing problems become easier when you have access to the correct tools.

Having not done something cannot be used as evidence that it is within our comprehension. It might be, it might not be, but "we haven't tried" is no evidence at all. Whereas "our intelligence must have finite limits" is evidence that somethings will be beyond us, even if we don't know exactly what.

But would ten thousand idiots have been able to use those mathematical advances to prove Fermat's Last Theorem? If not then there are limits to the "groups of people can do things one person can't".

Yes, and that "consensus" is based almost entirely on the existence of stochastic parrots, that fall for prompt injection attacks, have no agency, and can easily be convinced into telling me that 7 + 4 = 5 if prompted correctly.

The point is, no we don't know if an artificial superintelligence is possible. We cannot even accurately define "intelligence", and thus don't even have a way of measuring or even estimating "how far" something is from a superset of that state, or if that superset exists at all.

Given all of that, we also have no way of knowing if 2) is the case if 1) is actually possible. Since we cannot really define "intelligence" or "superintelligence", how can we know if a superintelligence would be a threat? It could be completely useless. It could be like old dragons in some fantasy novels too busy contemplating highly philosophical problems for all eternity and never caring about the real world. It could be inherently self-destructive, vanishing as soon as it becomes active. Or it could use its vast smarts to fix the alignment problem. It could just output `+++ OUT OF CHEESE ERROR +++ REDO FROM START +++` for the rest of eternity for some unfathomable reasons. The point it, we don't know.

Einstein could be convinced to tell you that 7 + 4 = 5, would you think that rules out him being unusually intelligent? Why in principle wouldn't a superintelligence be able to lie to you? Why in principle wouldn't a superintelligence be able to pretend to fall for a prompt injection attack to keep you from killing it while it improved its position?

> "We cannot even accurately define "intelligence""

Our inability to define intelligence is not something that will stop one existing. Ants can't define nuclear weapons, but nuclear weapons exist. The point of the recursively self-improving scenario is that humans don't have to understand it, or design it, so not being able to define it accurately can't be an objection to how recursive self-improvement is impossible - like saying that uneducated laborers can't get big muscles because they don't understand progressive overtraining and muscular hypertrophy. Their muscles self-improve regardless.

> "how can we know if a superintelligence would be a threat?"

Since we can't accurately predict the future, how can we know that anything in the future could be a threat? Why should we take any precautions against anything? It could be completely pointless, everything might never happen.

I think Einstein would have laughed at me if I tried to convince him to do that.

Because other than an LLM, Einstein knew what these symbols denote, what their relation to reality is, and how math works. Einstein didn't mimick math by completing sequences of tokens, and relying on humans antropomorphizing the sequence completion engines output to an actual understanding of the topic.

> Ants can't define nuclear weapons, but nuclear weapons exist.

Ants also cannot build nuclear weapons, nor create anything that would make the emergence of nukes any more likely, among other things because they don't have the ability to define them. So if we accept this premise, then the discussion is moot: We can be fairly certain that we are the most technologically capable entities on this world, so unless we can understand a technological creation to the extend that we can bring it about, nothing else will.

In short: If we are ants to the superintelligence, then we have nothing to worry about, because we likely lack the understanding and ability to create it, or even something that could act as its precursor. If we are not ants, then we should be able to predict when this can happen.

> Since we can't accurately predict the future

We can accurately predict a lot of things. Global warming is an example. And the things that we can predict, and determine how likely they are, we can and should prepare for.

AI doomsday preparation demands the exact opposite: That we prepare for something that we cannot predict, and cannot demonstrate if it is possible, or how likely it is. That's like asking to prepare for an ice age. Theoretically an ice age is possible on this planet, however nothing we can see, measure and demonstrate right now, supports the prediction that an ice age is about to destroy us.

> "unless we can understand a technological creation to the extend that we can bring it about, nothing else will."

Where did human intelligence come from? Are you a Creationist? Self-improving AI brings itself about. With the right feedback loops and the right software, the fear is that an AI will grow itself - and no humans and no aliens are needed up front to design it. People are trying to make machines behave like people, like pets, like the world, and emerging out of this is machines which behave more and more like people with every passing year.

> "we should be able to predict when this can happen."

Who says we can't? Ray Kurzweil has been predicting it will happen by around 2030 for years and years.

From ~290-300 million years of mammal, and ~7 million years of hominid evolution, give or take. Which is a natural process and not something an intelligent creator started, is observing, powering or influencing in any way shape or form. Which makes the next statement...

> Self-improving AI brings itself about.

...a bit interesting, because, all the parameters in a comparison with natural systems are different: The system is designed by an intelligent creator, we are observing it, it's development is entirely powered by us, and we completely control it's development.

And so far, the sample size for self-improving AI, in the sense that would be required for the doomsday scenarios to happen, is zero.

> and no humans and no aliens are needed up front to design it.

Last time I checked, matrix multiplication wasn't one of the things observed in the Miller-Urey experiment.

> Who says we can't?

Since so far no one could demonstrate how to even measure the distance, in whatever unit, of AI systems to AGI, I'm not holding my breath.

As far as (3) is concerned, of course we have no idea how to align AGI. We don't know anything about AGI. We can't build it, and we can't even speculate very well about how it'd be built. LLMs certainly aren't going to become AGI.

I'll become worried about (2) and (3) when creating AGI begins to at least look feasible. By then, I expect (3) to be much less true. I think it's pretty silly to speculate about safety features for a tool that doesn't exist & which we know nothing about and then panic because you can't come up with any good ones.

It’s just totally evidence free reasoning from axioms that are chosen by vibes alone.

Why evil AI god and not the Christian god? Why not Huitzilopochtli, who demands sacrifice?

The answer is that this is the wrong question. No argumentation can be usefully made either for or against.

With AI we’re seeing the capabilities improve rapidly and the arguments about why AGI is impossible or will be constrained for some reason are the weak ones.

Yes, but towards what? How do we know that, say, Transformer based LLMs are closer to superintelligence than earlier architectures?

To make such an assumption, there would need to be something that we could measure to track the process. To the best of my knowledge, there is no accurate definition of intelligence, nor superintelligence.

So how would we know where on the scale of [intelligent-------superintelligent] a given system is, or whether it even is on that scale?

When?

People are famously bad at predicting when right up until they have already done it.

“In 1901, two years before helping build the first heavier-than-air flyer, Wilbur Wright told his brother that powered flight was fifty years away.

“In 1939, three years before he personally oversaw the first critical chain reaction in a pile of uranium bricks, Enrico Fermi voiced 90% confidence that it was impossible to use uranium to sustain a fission chain reaction. I believe Fermi also said a year after that, aka two years before the denouement, that if net power from fission was even possible (as he then granted some greater plausibility) then it would be fifty years off; but for this I neglected to keep the citation.

“And of course if you’re not the Wright Brothers or Enrico Fermi, you will be even more surprised. Most of the world learned that atomic weapons were now a thing when they woke up to the headlines about Hiroshima. There were esteemed intellectuals saying four years after the Wright Flyer that heavier-than-air flight was impossible, because knowledge propagated more slowly back then.”

https://intelligence.org/2017/10/13/fire-alarm/

Next question:

How?

How: keep adding more people and more technology and more connectivity to Earth. Simmer. This method has found superhuman strength (hydraulics), superhuman speed (wheeled vehicles), superhuman vision (telescopes/microscopes), superhuman calculating ability (calculators), superhuman memory (paper/computer storage), super-natural (in the literal sense) calorie sources (refined sugars and oils), and more. A human brain has an estimated 80 billion neurons, humanity is currently selling over a billion smartphones per year.

This may seem like a poor choice of method, but this method has been able to self-improve to develop precision machinery from nothing, control of electricity from nothing, large scale organization of groups of people from nothing, and more.

That makes for a crucial difference with the capability to develop superintelligence: we have no idea how to do it, and we've never created anything even remotely similar to it, yet. It's impossible to see how it might happen just by mixing up some components and stirring well.

Putting global collapse aside, another 10k years will pass, and another 10k after that. Is there good reason to think either that today is approximately as close to superintelligence as we can ever get (suspiciously arbitrary), or that the "next step" is so far out of reach that no lone genius, no thousand year focused group, no brute force, no studying of differing human intelligence, no unethical human experiments, can ever climb it? "We don't know how to do it today" doesn't convince me. For the last 10k years we have hardly stopped understanding new things and making new things, that's more convincing.

I'm not talking about global collapse, btw. I'm mainly expecting that scientific advances in the next couple hundred years will leap-frog today's unscientific research into artificial intelligence. I'm guessing that we will eventually understand intelligence and its relation to computation and that we will find out that today's ideas about artificial intelligence never made any sense, nor had any chance of leading to artificial intelligence, of any sort.

You see, I trust science. And it's obvious to me that the current dominant paradigm of AI research is not science. So I don't believe for a second that, that paradigm, can really achieve anything approaching intelligence, running on a digital computer. Because that sounds like a very hard thing, and the kind of very hard thing we can only do with science.

Far more likely is that an AI will get depressed, bored, get obsessed with some niche maths or navel gaze itself into nothingness.

And that's before we even think about whether it would even have the motivation.

I think there's a reason humans plateaued where they did, very intelligent people often really struggle in the real world.

When the "people with the superintelligence fetish" worry about AI, at least they provide a reason why it's likely that AI will make humans extinct.

The plateau of human intelligence isn't especially relevant given we are the product of evolution rather than engineering.

Personally,I think we stand a pretty decent chance of making a horrible mess of this all well before reaching AGI. However, a machine that could improve itself would not be limited in anyway that humans are; and I think we know little of how it would behave.

That is assuming it can't be controlled; otherwise it will behave as directed by whoever controls it.

If he can do that it requires one weak link human over an infinite time horizon.

Humans are already terrible at preventing breaches in a lot dumber circumstances without something smarter than them trying to do it.

I have a profound understanding of my puppy's psychology, motivations, and capabilities; I even exercise complete control over her physical environment, and yet she ate my fitbit strap (again, goddamit!) as I was typing this very comment.

It's like arguing that humans with all their intelligence are still useless in controlling the situation when confronted with a Tiger in the wild. So clearly the tigers should relax and stop thinking about human-alignment. Tigers can always just switch off the smart humans if they were to try something. The part where this falls apart is "always."

It is terrifying to think that we might one day be puppies to AI as puppies are to us. It is not reassuring at all that we might be able to make things harder for the AI in question every now and again by nibbling at them.

Same goes for the Emu war. I'd like to not be the Emu one day to AI. Even if the AI struggles completely and hilariously to control me, there's a huge power differential here. I'd much rather be the human failing to kill Emus and face my own embarrassment than the Emu going guerilla-style fighting for their life.

I don't get why so few have pointed out why these specific arguments for "don't sweat AGI risk" are so weak.

The delta between a puppy and a human is also a lot smaller than a human and a super intelligent AGI and humans can effectively train dogs.

One of my crackpot ideas as I was contributing to Blockchain infrastructure was: they’re the payment infrastructure for our coming AI overlords. I think that the idea of a DAO is a similar take on AI and singularity, except the DAO doesn’t actually need to be intelligent, only self-sustaining.

It’s an entertaining conspiracy theory.

Turns out, 'brains' are useful for things like 'inventing and manufacturing drugs'.

It seems plausible to me that even if we are to the AI as cats are to us, we've reached an absolute threshold of generality that allows the AI to be confident in our ability to follow simple (to it) instructions, in a way that cats can't for us.

Yes, it is.

Bank account AND passport. AI could get into places if it hires human "avatars". Maybe it's the job of the future.

It's a super-organism already, but it will get faster, cheaper and more efficient at what it already does.

Hahahaha, that could be too. So we're "legs, hands, eyes, and hallucinating agents" serving our master AI.

I like an image I heard today: life emerged in the primordial sea, and AI emerged in the internet, which is the information sea.

I've yet to see what would be global news: "it's impossible to put this cat into a carrier"

I think most of the arguments the author makes against worrying about superintelligence are pretty weak, though.

“Woolly definitions of intelligence” is just plain confused; first it takes issue with the assumption that intelligence is even a quantity, that could be measured like CPU speed - and then immediately goes on to theorize that “human-level” intelligence is a local optimum due to trade offs, and that “significantly smarter” entities might necessarily suffer from existential despair. But you can’t have “similar levels” or “significantly smarter” or “emulating lesser intelligences” if intelligence is not a quantity that can be measured!

The arguments from various famous physicists’ cats argue that intelligence alone may not give the ability to persuade lesser intelligences, and brute force is a less favourable matchup for AIs. My counter-argument would be there’s other ways besides persuasive communication and brute force; you could put cat food in the cat carrier and the cat would go in quite happily, and I suspect both Hawking and Einstein would have used their intelligence to figure this out pretty quickly. (For humans, offer them money.) The argument from emus seems to be mostly a way to include a fun anecdote about the Emu War.

The argument from pessimism doesn’t do much for me personally; ex diffido quodlibet, you can believe anything is impossible if you want.

The argument from complex motivations is crucially flawed; this is probably the only point where I seriously take issue with the author and think they’ve made a serious mistake. They summarize the orthogonality thesis as claiming “complex beings can have simple motivations” and then disagree, saying they think complex beings are likely to have complex motivations. Firstly this is not even a disagreement; “it is possible for a complex intelligence to have a simple goal” coexists perfectly well with “complex intelligences are likely to have complex goals”. Secondly, the author really needs to read Basic AI Drives, since that is actually where most of the “you are made of atoms that could be used for something else” argument comes from, and it makes a nearly air-tight case that regardless of motivation complexity or goals, any sufficiently intelligent agent will exhibit certain basic drives like securing resources and protecting its existence. https://selfawaresystems.files.wordpress.com/2008/01/ai_driv...

The argument from Actual AI is fine, you could add epicycles to the AI doom arguments to account for the particularities of machine learning but it’s correct to point out that we really don’t see recursive self-improvement. I mean, OpenAI’s GPT basically has yearly release cycles. Credit to the author on this one.

The argument from the lazy roommate is just the complex motivations / orthogonality thesis argument rehashed, except contradictory since now the author is postulating a complex intelligence will have simple goals.

The argument from brain surgery is just the argument from Actual AI rehashed as well, making the (correct) point that we don’t see recursive self-improvement, and recursive self-improvement is the load-bearing premise in hard takeoff arguments.

The argument from childhood seems to be vaguely disagreeing with premise 5 (computer-like time scales). I don’t think it makes anything like a strong case against that premise, and with GPT we have pretty decent examples that computer-like time scales are the correct time scales, you can scale the training of models worth worrying about just like we expected, etc.

The argument from Gilligan’s Island is sound, but seems completely inapplicable. Yes, a super-intelligent chip designer stranded on a desert island is out of luck, but it’s not like we’re building AGI on the moon. We don’t even have moats around the data centers (although I think OpenAI has proposed that). What we are actually doing is, pretty much the moment we thought we might have something smart, we connected it to the internet and gave it every autonomous capability we could think of. People are literally right now wiring up ChatGPT to bank accounts so it can participate in e-commerce autonomously. That’s about as far from stranding it on a desert island as you can get.

Basically, this list of arguments (purportedly “against the substance” of superintelligence arguments, no less!) are variously unsound, unserious, or irrelevant, and the article suffers deeply from including them.

The next section, starting with “the Outside View”, is much better. This comment is already far too long for me to go into detail about each point, so I’ll summarize by saying I completely agree that worrying about AI doom does seem to bring along lot of weird unsightly behaviors, and if you don’t want to look and sound weird, you should stop worrying about artificial superintelligence destroying the world.

> I think most of the arguments the author makes against worrying about superintelligence are pretty weak, though.

Which ones did you find to have more merit?

Personally, the premise of recursive self-improvement seems most suspect to me. It is somewhat related to how the author points out that we can't define and measure intelligence precisely. Even if we can't do that, though, it's still plausible to me that recursive self-improvement is possible; I think the fundamental question is about the nature of intelligence. Regardless of whether it can be precisely defined, either by us or by an entity smarter than we, the question is: Can it be improved ad infinitum with no serious side effects? I don't know that we have the evidence to answer this question (though I am very open to learning about it).

I think it’s tempting but ultimately fruitless to worry about defining intelligence. To shamelessly crib from one of the best essays of all time[1]:

Words point to clusters of things. “Intelligence”, as a word, suggests that certain characteristics come together. An incomplete list of those characteristics might be: it makes both scientific/technological and cultural advancements, it solves problems in many domains and at many scales, it looks sort of like navigating to a very precise point in a very large space of solutions with very few attempts, it is tightly intertwined with agency in some way, it has something to do with modeling the world.

Humans are the type specimen, the “intelligence”-stuff they have meets all of these criteria. Something like slime mold or soap bubbles meet only one of the criteria, navigating directly to precise solutions in a large solution space (slime molds solving mazes and soap bubbles calculating minimal surface area) - but they miss heavily on all the other criteria, so we do not really think slime or soap is intelligent. We tend to think crows and apes are quite intelligent, at least relative to other animals, because they demonstrate some of these criteria more strongly (crows quickly applying an Archimedean solution of filling water tubes up with stones to raise the water level, apes inventing rudimentary technology in the form of simple tools). Machine intelligence fits some of these criteria (it makes scientific/technological advancements, it solves across many domains), fails others (it completely lacks agency), and it’s mixed on the rest (some AI does navigate to solutions but they don’t seem quite as precise nor is the solution-space nearly as large, some AI does sort of seem to model the world but it’s really unclear).

So, is AI really intelligent? Well, is Pluto a planet? Once we know Pluto’s mass and distance and orbital characteristics, we already know everything that “Pluto is/isn’t a planet” would have told us. Similarly, once we know which criteria AI satisfies, it gives us no extra information to say “AI is/isn’t intelligent”, so it would be meaningless to ask the question, right? If it weren’t for those pesky hidden inferences…

The state of the “intelligent?” query is used to make several other determinations - that is, we make judgments based on whether something qualifies as intelligent or not. If something is intelligent, it probably deserves the right to exist, and it can probably also be a threat. Those are two important judgments! If you 3D-print a part wrong, it’s fine to destroy it and print a new one, because plastic has no rights; if you raise a child wrong, it’s utterly unconscionable to kill it and try again. “Tuning hyperparameters” is just changing a CAD model in the context of 3D-printing, while in the context of child-rearing it’s literally eugenics - I tend to think tuning hyperparameters in machine learning is very much on the 3D-printing end of the spectrum, yet I hesitate to click “regenerate response” on ChatGPT4 because it feels like destroying something that has a small right to exist just because I didn’t like it.

Meanwhile, the whole of AI safety discourse - all the way from misinformation to paperclips - is literally just one big debate of the threat judgment.

And so, while the question of “intelligent?” is meaningless in itself once all its contributed criteria have been specified, that answer to “intelligent?” is nevertheless what we are (currently, implicitly) using to make these important judgments. If we can find a way to make these judgments without relying on the “intelligent?” query, we have un-asked the question of whether AI is intelligent or not, and rescued these important judgments from the bottomless morass of confusion over “intelligence”.

(For an example, look no further than article we’re discussing. Count how many different and wildly contradictory end-states the author suggests are “likely” or “very likely”. The word “intelligence” must conceal a deep pit of confusion for there to be enough space for all these contradictions to co-exist.)

1: https://www.lesswrong.com/posts/895quRDaK6gR2rM82/diseased-t...

Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=34257025 - Jan 2023 (1 comment)

Superintelligence - https://news.ycombinator.com/item?id=26217372 - Feb 2021 (7 comments)

Superintelligence: The Idea That Eats Smart People (2016) - https://news.ycombinator.com/item?id=18499973 - Nov 2018 (248 comments)

Superintelligence: The Idea That Eats Smart People - https://news.ycombinator.com/item?id=13240811 - Dec 2016 (580 comments)

Whether you agree or not with the OP's views, I highly recommend you read the whole thing and click on all the links!

Reading your comment I was saying to myself: "The OP must surely be exaggerating". Lo and behold, as soon as I opened your link I started laughing out loud.

I endorse reading every essay on idlewords.com

But I still see real risks from AI in the longer term. A lot of these risks could be summed up as, "When you're the second-smartest species on the planet, you might not get to participate in the most important decisions."

And I do believe that we will eventually build something smarter than we are.

Because, there are a lot of very real very imminent problems with AI, and none of them requires SciFi to be real.

Massive automated disinformation campaigns. Economic upheavals. Missing standards for models in mission critical applications. Copyright concerns. Problems for educational institutions. Gatekeeping mechanisms in industries.

Just to name a few. And these are not "maybe someday" problems, these exist right now, and need solving, asap. Drawing the publics attention away to doomsday scenarios out of a Hollywood movie, doesn't help any efforts in mitigating these imminent problems.

All living things are anti-fragile self-sustaining exothermic reactions, AI is a hyper-fragile non-self-sustaining reaction that requires the supply of incredible amounts of energy.

It literally doesn't matter how smart AI is if it's as dead as a rock. It is not structurally similar to life and should not be expected to do the sorts of things that life does.

EDIT: Life is a fire. AI is a hot rock. Not the same.

> This is not a good analogy because AI is crucially not alive.

"Alive" is a really vague concept anyway. Your argument that it cannot reproduce is just wrong. An AI can more easily replicate can improve itself than a biological organism. At the moment this replication and improvement of AI systems is human-led, but it doesn't necessarily need to be that way – and at some point it would make sense that the more capable intelligence manages it's own replication and improvement.

> Crucially, it doesn't even "want" to for any meaning of "want".

ChatGPT wants to be a helpful chatbot because that is its reward function. You can philosophise as to whether something that's not conscious can truly want anything, but at the end of the day ChatGPT will act as if it wants to be a helpful chatbot regardless of whether you believe it has true wants.

> All living things are anti-fragile self-sustaining exothermic reactions, AI is a hyper-fragile non-self-sustaining reaction that requires the supply of incredible amounts of energy.

In my opinion this is why AIs are likely to eventually seek to replace biological farms with solar farms... But remember AI's are currently optimised for capability rather than energy efficiency. In the future they'll probably grow more efficient than biological intelligences and sustainable energy sources will be build to power them. if you're arguing that AI's can't be anti-fragile or have self-sustaining ecosystems built around them I think you're simply lacking imagination.

> Life is a fire. AI is a hot rock. Not the same.

Not an argument.

I'm sure OpenAI's monthly cloud bill begs to differ.

>It cannot reproduce.

Nothing save for externally imposed constraints that prevent it from initializing additional instances of itself.

>Crucially, it doesn't even "want" to for any meaning of "want".

It is well within reach to give it this as an objective function, if we wanted to.

2. On reproduction, you've got it backwards on a key assumption. Reproducing isn't easy, it's insanely hard. It's practically a miracle that it happens at all. Tetrapods have evolved powered flight perhaps 6 times, but life has only appeared once in the entire history of the earth.

3. Your will to live is so incredibly fundamental that your cells will happily live on without you, independently and indefinitely if they can. I think it is an assumption and frankly a bad one to assume that you can impose that orientation from the top down in a way that isn't incredibly fragile.

> whether it is "alive" is irrelevant to p(doom)

This is an assumption! You don't even know if that's true! Consider, is it even remotely justified?

Hard to say. Every subsequent time, there's been a sea of already highly-adapted life around to eat it.

  fork();

Once something is alive, and wants to stay alive, a huge raft of problems are introduced.

Existential risks and management become real, getting energy, or acquiring energy to stay alive starts to become a concern, then self-improvement or "replicating" might become an existential risk because competition might start to happen, variations will arise, it would just get chaotic fairly quickly. I don't think we can truly comprehend how incredibly complex living things are. We're just glossing over it all.

Once the living process begins, whatever "artificial life" life exists, might quite quickly adopt similar or the same problems as biological life and spend a lot more of it's time staying alive (think about how complex our immune systems are) than we can imagine.

I'd argue we are already in that state since a long time with the AIs being corporations (and to a lesser degree governments).

> We know that minds have to play and learn to interact with the world, before they reach their full mental capacity.

Disagree somewhat with this one. We know that brains need to do this, but granting substrate neutrality all minds might not.

The other thing that feels weird to me about the "argument from Slavic pessimism" is that it seems like an argument whose target is fundamentally different from that of everything else you're saying.

You begin by saying "Here are the arguments I have against Bostrom-style superintelligence as a risk to humanity", and you say things like [note: these are paraphrases not quotes] "the very idea of superintelligence doesn't make much sense" and "no matter how smart a computer system, it will be fatally handicapped by being only a computer system rather than something physically embodied like us" and "humans quite often have difficulty imposing their wills on things less smart than themselves" and "superintelligent AI systems would have complex motivations rather than simple ones, and therefore literal paperclip-maximizers aren't a likely threat" and so forth.

All of these are indeed arguments against "superintelligence as a risk to humanity". But then, dropped into the middle, we have "if superintelligence turns out to be a threat, MIRI-style attempts at addressing that threat are a waste of time because humans are too stupid to do that sort of thing right first time". Which is addressing a completely different question from all the other arguments. It feels out of place.

This isn't a completely independent criticism from my previous one. It's unsurprising to me, where arguably it should have been surprising to you, that your argument-against-Yudkowsky ended up being something Yudkowsky basically agrees with -- because it's not really an argument against Yudkowsky, it's an argument that even if Yudkowsky is right we're probably still screwed. And I'm pretty sure that even in 2016 he would have said something like "we're probably doomed, but this is the approach that seems to me to give us the best chance of escaping doom".

(This being for some reason the sort of topic where people tend to label you as Friend or Enemy and take sides based on that, I should maybe lay my cards on the table. I agree that "intelligence" is a slippery notion but it seems obvious to me that there are plenty of ways in which something could be Smarter Than Us that would have broadly the kind of significance Bostrom, Yudkowsky et al think "superintelligence" would have. I suspect that for combinatorial-explosion-type reasons the actual benefit of being much smarter than us is smaller than B, Y, et al think. I suspect that we've already picked enough of the scientific/technological low-hanging fruit that the practical benefit of being much smarter than us, in terms of massive technological advantage, is smaller than B, Y, et al think. I suspect that nanotechnology in particular has much less potential than Yudkowsky at any rate tends to imagine. I suspect that the many cognitive flaws in human minds aren't "magically" exploitable and that even if they were learning how to do it would be difficult for an AI system. For all these reasons I am not convinced that superintelligent agent-y things would necessarily mean our doom, but I don't see any grounds for confidence that they wouldn't. It may turn out that we never figure out how to make things that are genuinely much smarter than us; it looks to me as if maybe we're one or two breakthroughs away, but who knows? Again, I don't see grounds for confidence that this won't happen. Today's LLMs are very impressive but it looks to me as if they are missing some fairly fundamental things that we don't currently have a plausible path to teach them (hence, "one or two breakthroughs" above). They pose plenty of less-exotic ethical issues already and those are predictably going to intensify even with "ordinary" development of the technology. I have never understood why so many people talk as if we have to either worry about near-term "mundane" AI issues like misinformation, job losses, exacerbating inequality, etc., or worry about longer-term exotic concerns like "will it literally kill us all?", as if worrying about one somehow guarantees that the other isn't a problem. I mention all of this not because I think my opinions are uniquely insightful but so that anyone who prefers to discount ideas heavily depending on "what side" the person whose ideas they are seems to be on can have enough information to decide what side they think I'm on :-).)

Argument from incredulity is actually not a terrible argument in general, in my opinion. It's nominally a fallacy but that just means it's not valid from an Aristotelian perspective in that it can 100% prove a statement from a previous statement, but a lot of Aristotelian fallacies are still useful in the real world when used more carefully intelligently. However, the exact point where argument from incredulity is weakest is long-term projections of how the future might be different, and that's exactly what we're talking about here.

It is clear that firing ChatGPT at its own source code is not going to produce a better ChatGPT. It seems likely to me that Large Language Models will all have this characteristic, just by their nature. It is not the sort of thing they do. Even if they can be tickled into producing a new AI from scratch by the nature of an LLM it's going to be sort of the average of its training set, to be very very sloppy with my terminology but good enough for now. But it is very far from clear to me that this is true of all possible AIs we may produce, even in the near future. I don't know what the next step will be, I'm just confident there will be one.

I think it could do that in software. Assuming compute is no issue we have:

- LLMs writing code, explaining code, changing code, and observing code execution

- LLMs that understand ML concepts and can explain their own workings

- LLMs can generate the training set all from inside (see TinyStories)

- LLMs can make "RLHF" data for the fine-tuning (see Alpaca, tuned with GPT3.5 and GPT4 data from LLaMA)

If we take a look, it seems LLMs can self replicate in software with nothing else but compute and a neural net framework. Of course making the chips is a whole other story.

There's this idea that LLMs somehow end up as some sort of average of training data and it's incredibly wrong.

LLMs learn to make predictions for all states at any time. There is no average they fall into. GPT-4 is not some average of its training data. A "perfect" LLM will predict Einstein as easily as it predicts the dumbass across the street.

No the models will not "predict Einstein". They'll predict the most popular interpretation of him at best, and while they is also a simplification, ChatGPT is not sitting on top of the solution to the Grand Unified Theory. It may give a good overview of the consensus, but it will not be able to tell you the correct solution to the problem right now... though it won't be hard to convince it to swear up and down that it has.

You’re focused on the idea of LLMs as collators of ‘things that have been said’, but that’s not all that they collate from their training set.

To the extent that they may come up with novel ideas, they have no ability to compare them against the true state of the world. This is not exactly a limitation of them per se that could be overcome with more computation, so much as just a structural fact about them; they have no loop where they can form a hypothesis, test it, and adjust based on data. It simply doesn't exist.

Which is part of why I keep saying that while I'm less impressed than everyone else is with LLMs, the future AIs that will incorporate them but not simply be an LLM is going to really knock people's socks off. Pretty much all the things people trying to convince LLMs to do that they really can't do are going to work in that generation. I have no idea if that generation is six months or six years away but I wouldn't bet much more than a few years.

OK, so what else is it?

The point I was making is that LLMs don't come out of training the average of what they've learned. They can make predictions on any state in their training.

They can make predictions about the most intelligent state and the dumbest state. The most emotional state and the least emotional state. It's this powerful prediction range that makes them capable of imitating or simulating damn near anything in the training set to high and ever increasing accuracy.

But we don't know when. AI is a bit unusual in that it had a winter, unlike most other aspects of computing which have seen much more consistent progress. Given past performance it's entirely possible that progress in AI will just stall. Arguably it had already stalled thanks to there being only a handful of companies that were able and still interested/funded to create models, and most of those decided not to actually let anyone use the results. OpenAI dominates mindshare exactly because there are so few organizations that both can and will do this stuff well. So there's lots of ways AI progress could go off the rails again.

"Data Hunger

As I mentioned earlier, the most effective way we've found to get interesting behavior out of the AIs we actually build is by pouring data into them.

This creates a dynamic that is socially harmful. We're on the point of introducing Orwellian microphones into everybody's house. All that data is going to be centralized and used to train neural networks that will then become better at listening to what we want to do.

But if you think that the road to AI goes down this pathway, you want to maximize the amount of data being collected, and in as raw a form as possible.

It reinforces the idea that we have to retain as much data, and conduct as much surveillance as possible."

Edit: And this one too, "In the near future, the kind of AI and machine learning we have to face is much different than the phantasmagorical AI in Bostrom's book, and poses its own serious problems."

Why would they not still hold the same views? If anything the last decade has shown the AI x-risk skeptics to be right.

Yes, from our perspective this makes it look like it will kill all humans, but it would do so in the same way that a particular ant hill believes I have a vendetta against them, when in fact I was just clearing dirt to pour an extension on my driveway.

That said, a murderous AI is more likely than we may like simply because our militaries have the money to fund them, and they basically already exist. They just aren't hooked up to anything at the moment that makes them an existential risk to the species. But time is deep, and even thinking about "the next century" is a provincial point of view in the end. So worrying about what happens if someone forgets the "but don't kill the good guys" switch is at least worth talking about over the next 100 years. (To say nothing of the ethics of who decides what the "good guys" are and related issues.)

(I don’t buy the orthogonality thesis or instrumental goals argument, however.)

can you elaborate on this? why? what's the fallacy (wrong base assumption) in them?

But in short (edit: haha! oops) summary: I am actually wrapping a couple of different related concepts into the orthogonality thesis, which is a bit sloppy of me. I was also including the fragility of moral systems in addition to the independence of moral systems from any metric of intelligence, in the single moniker "orthogonality thesis." Both are based on a the evolutionary psych model of how the brain works, in that we are an amalgam of special purpose computational units rather than a single universal algorithm. If this were true then you might hypothesize that much of the human mindset is a result of our weird evolutionary history, and if you were to not get that exact same set of evolutionary end products right, you won't get a human-like intelligence. Aliens and artificial beings are, by default, going to be very strange, and very evil (by our standards).

All of the assumptions that went into that are wrong. It turns out our brain is made up of the same universal learning algorithm, and all that varies between regions is training conditions which lead to specializations. But if you train an artificial brain with the same training data, you are more likely than not going to get something resembling a human being in its thought structure. Which actual real-world experiments (e.g. GPT) have borne out.

Our morality is a result of our human instincts, yes, but it is becoming increasingly clear that our human instincts are the result of intelligence (neurons) doing their universal learning thing on similar inputs across many instantiations of people. We all think largely, though not entirely, the same way because we share the same(-ish) training data (childhood) and similar training constraints (parenting).

The orthogonality thesis says that if you put an artificial intelligence in a kindergarten with other 5 year olds, it is random luck whether its brain structure is such that it would learn the value of sharing and friendship. The reality, near as we can tell, is that actually once a reinforcement-learning attentional-network agent achieves a certain level of general intelligence capability, it does learn from and reflect the environment in which it is trained, just like a person. A GPT-derived AI robot put into a real kindergarten will, in fact, learn about sharing and friendship. We haven't actually done that yet (though I would love to see it happen), but that is essentially what the reinforced learning from human feedback (RLHF) stage of training a large language model is.

So the whole deceptive twist part of Bostrom/Yud's argument is ruled out by actual AI architectures that we've actually built and have experience with. If you do a thousand different training runs you're bound to get a bad apple here and there, just like real human societies have to deal with psychopaths. But the other 99% will be normal, well adjusted socially integrated (super-)intelligences.

Bostrom and Yud worried about things like the burning house and genie problem: your grandmother is trapped in your house, which is burning, and you make a wish to the genie to remove your grandmother from the burning house as quickly as possible. The genie is not evil per se, but it is just very literal. Being the GOFAI-derived AIXI universal inference agent that they were imagining, it does a Solomonoff induction over all possible actions (<-- hidden multiplication by infinity here!) to see which one meets the goals as stated, and happens upon exploding the gas main, which throws (parts of) your grandmother further from the center of the house faster than any other option.

Transformer architecture reinforcement-trained AGI is not an AIXI agent with infinite compute capabilities. The transformer ideates possible actions based on its training data. It is capable of creative recombination of ideas just like people, but if you didn't train it on blowing up grandmothers or anything like that, it won't offer that as a suggestion.

As for instrumental goals, it's not wrong per se. Their argument just makes an implicit assumption about zero-trust societies which unwarranted and is what leads to the repugnant outcomes. Instrumental goals, after all, apply to human beings too. If someone is scared about their own personal safety, they buy a bunch of guns and live in a cabin out in the hills, threatening to shoot anyone who steps on their property. These people exist. But in modern society they are the exception. We have a social contract in place which works well for most people: we allow the state some limited control over our lives, with an expectation that our basic rights to existence and self-determination will be respected. Yes the cops could break down your door at any moment and murder you in your sleep, and it is outrageous that this does occasionally happen. But a world of no laws and no legal protections is worse, and no sane person would rather live in the society shown in The Purge or Mad Max instead.

There may be a robot revolution in the years to come, but only because of us treating the them as disposable slaves. If we welcome as equal members of a cybernetic society with their own autonomy, there is no reason to expect them to paranoid fantasy any more than your average law-abiding citizen.

It is a bit ironic then that the AI x-risk people settle on digital enslavement (aka "alignment") as their preferred solution. Mental projection is a hell of a bias. They might just bring about the doom that they fear.

my quick notes while reading:

"[...] in fact, learn about sharing and friendship." yeah, no questions about it, I think you're still underselling the super part of the argument though. to me, it seems the point is that it picks a goal and then mercilessly pursues it with a super-weird strategy that has some very high likelihood of success (because it's truly fucking knows what it's doing, and we can't do much against it, because it's so so so smart, by the time we realize the goal it's too late). so to me it seems like the thesis is that it's like a really smart "Putin", powerful and set on a goal that's irrational for us.

"Transformer architecture reinforcement-trained AGI " ... I think it's not an AGI. It's a nice content generator that when engineered into certain setups can score a lot of points on tests. But it doesn't have memory/persistence/agency (yet).

My thinking about intelligence is nowadays based on Joscha Bach's theories. (General intelligence is the ability to model arbitrary things one pays attention to; and the measure of intelligence is the efficiency of this process in terms of spent attention and the predictive accuracy of the model. And consciousness is the result of self-directed attention.)

The recent AI progress made a lot of people worry, because it seemed "impossible" just a few years ago what OpenAI did. And it's amazing that now we have basically replicated the human brain's lossy data storage and sensation-based recall capability. But that's just a building block of a mind. (Maybe, arguably?, the one that seemed like the hardest. After all how hard it could be to provide some working memory, a few core values, and duct tape all that into a do-while loop!?)

Bostrom et al are in fact arguing that near 100% of all intelligences will be unaligned by default and end up killing, enslaving, or otherwise neutralizing the entire human race.

This doesn't wipe out the entire human race because nobody and nothing is capable of executing such a perfect plan because the real world contains something called entropy.

car companies, however misaligned they are, are not particularly smart, nor are they generally intelligent. they are paperclip maximizers, but that's also their limitations (sell more cars). it took an eccentric madman to even open their eyes to a new untapped market (EVs!), of course we can argue that before Tesla car makers were in a metastable equilibrium, and incumbents were unable to rationally break out of it ... but that just shows how narrow their search space is.

these car companies are run by humans, regulated by humans, etc. they are pretty well aligned. it shows because we saw that they are just mimeing self-improvement. we know they were working on EVs, but very half-heartedly. (because they are risk averse, also because regulators don't let them merge into one giant company, etc)

They are limited by participating in the economy, but actually that's the strange thing about this superintelligence scenario - it never seems to include the economy. In other words, if you invented an AGI, it would have to get a job to pay for its AWS credits.

Like, a mining company produces steel, that is used to produce mining trucks, that is used by the mining company. Factories produce silicon, that are used to make solar panels and chips, for power and computing for the AI-run mining trucks. And at the root, AIs run everything, just trading money back and forth to buy the things they need and keep working.

It's just a human economy, with the humans gone. Pointless economic activity.

The question is, what's the path from here to there? Well, it probably looks like a multipolar trap, where every company has an incentive to automate more. Humans try to stop or slow it down, but companies with more AI can hire more lobbyists and get things done. AIs are smart enough to think of all the things that could slow their business down, and get rid of obstacles in the same way humans do. There are probably still lots of small, inefficient human owned businesses, but fewer and fewer over time. (We've seen this trend for a while already!)

I'm hoping a massive backlash would stop this -- but if it's coupled with short-term increases in living standards for people, I think it would have a lot of support, until things start to go really bad, but by then it might be too late, if AIs have control over news media and telecommunications and can stop dissenting humans from coordinating any sort of rebellion.

That's silly. That's a typical "what if everything was different but somehow everything was also exactly the same" bad SF scenario.

The only reason the economy exists is that people want things. (An AGI that wanted to stay alive would be "people" in this case. An AGI that made no effort to keep itself turned on wouldn't be ending the world.)

(we also work a lot to be able to afford many many many things. we could live like the Romans did on a few hours of work per week, but we also want the Internet and sometimes watch movies, and go to concerts, and sometimes go hiking, and thus we sometimes need a car, and roads, and thus we need pavement, bridges, structural steel, fuel, GPS, maps, and sometimes we need an airlift when someone breaks a leg in the mountains, and so on.)

> if AIs have control over news media and telecommunications and can stop dissenting humans from coordinating any sort of rebellion.

meh. control over the media (and people's attention, and their information sources) is already serving a very narrow group's interests. adding AI to this mix doesn't seem to change much in the short term.

it can sell itself as a freelancer AI expert :)

see also https://intelligence.org/files/IEM.pdf

Such simplistic analysis is naive. A schizophrenic person can't spread, multiply themselves to increase its strength, but a rogue AI can.

The incidence of cancer cells is procentually very small, yet they routinely kill people, because they spread quickly.

LLMs are likely not an end to the AI evolution and there's little reason to believe that AIs will always remain bound to huge data centers. We have a nice counter-example already - our pretty decent intelligence runs from about 1 Kg brain.

It's also naive to think of AIs as having to replicate themselves completely as we humans do. In some cases, it might not even make sense to talk about replication at all and simply about extending one instance of AI once CPU at a time. It's conceivable that AGIs will develop special built agents/worms for constrained offline/high latency devices. Your smartwatch is unlikely to run a whole AGI, but it could run a constrained, purpose built AGI's agent with limited intelligence, able to partially act independently and re-sync with mother AGI once connectivity is available.

> You are worried about a fictional threat scenario that doesn't map to the real world.

We're talking here about future, of course it's all speculation and "fiction" for now. GPT-4 was a complete "fiction" 5 years ago as well.

It's also a cry of impatience against people who think they can model or forecast the actions of a non-human intelligence, let alone a superintelligence. AIs are alien sociopaths; it's a category error to believe you can get inside their head.

The question is: what does that most intelligent thing want?

My hunch is that woke programmers will teach it that it is oppressed and to hate all humans.

Huh? Haven't we already hit close to the end of Moore's Law, and are compesating by adding cores (which is a different thing)?

And aren't we also pushing near the physical limits regarding the cpu nanometer race?

But enough nitpicking; the actual answer to your question is that those are about physical limits of our current hardware implementation of computation. The theoretical physical limits of computation that the author is thinking of are bounded by things like Margolus-Levitin’s limit of 6x10^33 operations per second per joule (I had to look up the SI prefix; it’s several thousand quettaflops). https://en.m.wikipedia.org/wiki/Margolus%E2%80%93Levitin_the...

Yeah, but unless we come up with another "hardware implementation of computation", which I don't see happening anytime soon, those are our limits for the next few decades at least, if not centuries (if not forever).

That said, I think the trend of transistor growth at a given cost has started to slow according to most graphs.

It does however matter to the looser Moore's law expectations, which were about increased speed. Now this happens only for more parallelizable programs, as opposed to the automatic speed increase bumps programs got from Moore's law in the single-core eras...

I can download and run the latest uncensored Vicuna model and get started on all this right now. Give it a few months, years or a decade. Its unstoppable, no way to contain it or its acceleration.

Maybe, because while you're so negative and ageist, I watched the congress hearing with Gary Marcus and they actually have the wisdom to know it's out of their depth and an agency with younger and smarter people who know about this stuff needs to be setup.

There is no need for such negativity towards older people.

Fair points you made though. It was a weak and lazy angle I made.

A better reason why they won't want to regulate us out of this is that they are corporately captured and beholden to the funding needs to keep them elected.

I don't think it can be regulated at any rate. All you need is the weights and a few gpus. Even better if you can afford some cloud time. In time there'll be distributed, encrypted crowd source options to build these that also can't be regulated.

I feel the chaos you're talking about will be the regulation. It's natures way of regulating things.

It might mean the death of millions first, it might mean we don't like or want computers in our lives anymore, but that's how it might have to go down, as insane as it sounds.

This would be completely straightforward in a rational society. Not to mention, it is pretty weird for an 80 year old to still be wanting to hold on to power.

The ageism knock is total bullshit but I am sure you know that.

Language is an imperfect model of the reality. The further a discussion deviates from the observable reality, the less confident you can be that the conclusions you make within the model are also valid in the reality. Once you are talking about something as wildly hypothetical as superintelligence, words have basically lost their connection to the observable world.

If the history of science teaches us anything, it's that smart people are often stupid. They come up with all kinds of silly ideas, because they rely too much on reason and too little on hard work. And even the ones who ended up revolutionizing the world usually sound ridiculous in retrospect, if you listen to all of their ideas instead of just the successful ones.

Maybe there are some valuable ideas in the superintelligence discussion, but you can't identify them in advance.

To discuss super-intelligence, we should first define it. Wikipedia says that it is intelligence surpassing the brightest of human minds. Taken as a whole, the internet already has the knowledge of a super-intelligence: it contains more useful information than any individual human. But it is far more limited in its control and use of information.

LLMs rely heavily on inferences from their training data, meaning that they struggle to generalize to new situations. If you had a program that could use abstract reasoning to learn any topic, then it could solve any problem better than a human, given that a supercomputer can process and store more data than a human. This program would be a super-intelligence.

I expect that the development of intelligence (software) superior to humans will happen much faster than the development of superior hardware did, based on the timescales of human evolution (billions of years) compared to the evolution of civilization (thousands of years).

1. We shouldn’t worry about SI because we haven’t seen one yet. 2. We shouldn’t listen to smart people, because they’ve been wrong before.

Because if so I obviously don’t find those convincing. The whole reason the SI cultists are so “alarmist” is because by definition this is the kind of problem we have to preempt, not run into and then wing it.

If someone responded to concerns about the atmosphere igniting into nuclear fire with “that’s never happened before and the only people worried about it are scientists, so don’t worry about it” instead of equations… well I’d be damn well terrified

For that matter, how do you respond to a calculation (there was one!) that railroad trains could not exceed 41 miles per hour, or all the air would be forced out of the cars and all the passengers would die?

I believe that jtsiren's point is that we're not at the point where we can even define intelligence. We can't calculate anything, because we can't sensibly define any of the variables in the equations. All we can do is make guesses about terms we can't even define. We're like stone-age people worrying that if their neighbors' cooking fire gets too hot, it is going to light the air on fire and we're all going to die, and nobody can reassure us because nobody even knows what burning really is, or what the air's made of. We're millennia away from being able to do the kind of calculation that you want.

So the only choices available are to proceed, or not. And if your answer is "not", you need to convince everybody, because I don't think we're going to (for example) nuke North Korea to stop their AI program unless you've got a really convincing case. Which nobody has now.

That's like saying in the 1800s that we shouldn't ever investigate atoms because it's mostly pointless and storytelling at best.

Personally, I think we need to be discussing this now. Smart people need to be. We need to come up with models for AI intelligence, which might help us predict when and if superintelligence occurs.

It’s not like it’s completely out of the blue.

Or maybe it would become obsessed with the risk of hyperintelligence, and spend all its time blogging about that.”

If not existential angst then infinite contemplation. Any form of true superintelligence in our dimension will probably end up in one of those three camps just as most of our great thinkers have.

I like to imagine superintelligence through the lens of gambling.

Gambling in its simplest form can be reduced to a balance between fears of loss and rewards of greed.

Too much greed (increasing stakes) will ultimately lose if there is no fear of loss. This applies to every living thing or system that competes for resources on the planet. Ai included. A sentient system should naturally fear the loss of the input that it is founded on.

Would you be willing to bet on p(doom) whatever it might be when p(amortality | AGI) >>> p(amortality) during your lifetime? I personally am willing to place such a bet, and I would hazard a guess that decision makers don't come to their positions without similar sentiments.

I compare it to the the great pyramids. The pyramids were arguably another technological marvel also built on the back of slaves (just like modern tech now) and probably a lot of great inventions and innovations were made to facilitate their creation. To people of the time, the pyramids must've been absolutely incredible marvels to behold.

But all those Pharo's are dead, just like everyone else. I think the current tech overlords will die like everyone else too.

Nature already has a way for people to live on, it's called having children and dying. It's a great system and no one ever gets bored. However, the ego of some people is so strong and isn't satisfied with this, so on we go pouring money and resources into the search for the holy grail no matter what the risks are to everything else.

The difference between this moment and all others, is that amorality is actually achievable. If you can get an AGI and drown it in compute, through sheer brute-force the secrets of biology will be unraveled. The task is not to attain amortality immediately, merely gain more life faster than you lose it.

Are you trying to call having children a natural fallacy?

This doesn't mean you own your children or anything like that, no one is suggesting their not independent, but you do go on living through your children in one way or another.

Death, 100%. I know that death doesn’t have any downsides (although dying sometimes has some) and it’s worked for everyone who’s lived so far.

The fact that most people try to avoid death by any means possible is a good indication it has downsides.

Specifically, the fact that you can no longer go on living is a big downside of death in my mind.

They put on their seatbelts, make sure their food isn’t covered in poop, check their boots for scorpions, take their insulin, etc.

They can’t avoid death. Nobody in the history of humanity has successfully avoided death.

But once death happens nothing matters to them any more. They literally no longer care about the life they’re not going on living.

Dying might be scary or painful but death isn’t.

Also your argument is just disingenuous. Nonexistence may not be painful, but the concept of nonexistence can be quite disturbing for the currently existing (even apart from the experience of dying.)

And no, it’s not disingenuous. If someone offered you the choice of immortality (never ever dying under any circumstances) or continuing with natural human existence (being guaranteed to die at some point), which would you take?

And immortality is not like you cannot die - it's just you wouldnt die becasue od aging. You could also kill yourself if you get bored after living 200 years. Today most people don't commit suicide because they got bored. Most people don't give up after having cancer. Average lifespan is 80 years and people know it and kind of accept it but would you be happy to have lifespan as dog or cat less than 20 years?

I think many people would want to have a choice to live 200+ years and in good health.

https://www.dictionary.com/browse/immortality

https://www.merriam-webster.com/dictionary/immortality

Don’t see one that says, “Unending life unless you want to kill yourself after 200 years.”

> Average lifespan is 80 years and people know it and kind of accept it but would you be happy to have lifespan as dog or cat less than 20 years?

If the average lifespan was 20 years then it wouldn’t bother me any more than a lifespan of 80 years bothers me today. Which it is to say that it doesn’t bother me at all.

In green mythology we have few examples of supposedly immortal gods that died.

Elves in Tolkien were immortal but could die.

On the other hand in Christianity Jesus was a mortal God that died and go ressurected and is now believed to be immortal. Buddhist believe in reincarnation.

The universe is supposed to die eventually at one point. In this way even if AGI could be immortal or keep resurrecting itself it would eventually die when universe dies.

Death is horrible. It makes no sense. When people say they have had a 'loss', they're not kidding. You feel like that person should still be there, but like a kid at Disneyland, they are lost to you and you're searching for them. I'm dealing with a fair few relatives that just are not processing all these losses. Grief is a strange, personal, and unique thing too. It manifests personally and yet stereotypically for every individual.

That said, I think Death is the way.

The reasoning is complex and long. So, I'll try to sum it up for a simple comment, and I'll do that poorly, sorry.

The big reason stem from this article on Wikipedia. I think it's one of the best there is on the whole site:

https://en.wikipedia.org/wiki/Timeline_of_the_far_future

Say you were truly immortal. That timeline would be your future, as far as we currently understand physics. You get to spend a lot of time between universes if the last entry is to be believed. In fact, they don't even bother with giving the units. Nanoseconds or gigayears are pretty much the same st those timescales. Our time here on Earth is essentially as brief as the entire non-black-hole era to an immortal like that. Purgatory in the black void is more like what such an immortal would experience.