This is something weird happening around Rationalism/X-Risk/AGI prognostications.
The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing.
But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative models, and previous generations, in some cases for decades? They almost all roll their eyes aggressively at these sorts of prognostications. I'd say 90+% either laugh or roll their eyes.
Why is that?
Personally, I'm much more on the side of the silent majority here. I agree with Altman's criticisms of criticisms about regulatory capture, that they are probably unfair or at least inaccurate.
What I actually think is going on here is something more about Egos than Greatness or Nefarious Agendas.
Ego, not intelligence or experience, is often the largest differentiator between the bench scientist or mid-level manager/professor persona and the CEO/famous professor persona. (The other important thing, of course, is that the former is the group doing the actual work.)
I think that most of our Great Minds and Great Leaders -- in all fields, really -- are not actually our best minds and best leaders. They are, instead, simply our Biggest Egos. And that those people need to puff themselves up by making their areas of ownership/responsibility/expertise sound Existentially Important.
In my opinion the weirdness is that everyone is talking about how AI is so much better than humans and we're all going to get replaced. But almost nobody is talking about how tech companies will use this new technology to abdicate responsibility for the decisions they undertake. If you thought redlining was bad, you should see how mortgage companies treated people when they used an AI to accomplish the same goals. No thought was given to the effect of those decisions, only to the fact that this mysterious and "all-knowing" computer system told them it was a good decision. We're seeing the same thing with Facebook and "algorithms": "It's not us! It's the algorithm! We don't know why it does this." Which completely belies the fact that they wrote the algorithm and they should be held responsible for its effects.
We are about to enter a couple of decades of people using these pithy model systems to make real decisions that impact lots of people, and if I've learned anything in the past 20 years its that the impacts that technologists and "emininent minds" are predicting are nothing like what will actually happen. But terrible, banal things will be done at the behest of these systems and nobody is talking about it.
A very glaring example of this problem is sentencing guideline software that uses ML to suggest sentences to judges. Anyone familiar with the very basics of ML as a practitioner knows this is a terrible idea, very likely to reproduce any biases or bigotry in the training dataset, but the courts are increasingly treating them like blind just oracles. This is going to go very bad places imo.
The risk isn't some sort of rogue smarter than humans AI, it's humans using AI to do the same stupid evils in a deniable or even unknown way.
The worst part is we have learned zero lessons from the metamorphosis of "fraud" into "identity theft." In the former case, the banks were responsible for preventing "fraud" and when that became too easy they simply abdicated all responsibility for this new form of computerized fraud and created the notion of "identity theft" which is fraud but with a computer.
As well as these models will all be used to launder content. In the near future you will likely be able to ask an AI to create the next 15 sequels to the movie war games starring a young matthew broderick. Episode 12 will have a karate kid cross over. But how much of what is there will be an actually new thing? How much will be scraped from my War Games fan fic website that contains scripts of 27 sequels to War Games?
Here is ChatGPT 4’s summary of a sequel. I don’t think it’s very good. Maybe another AI would enjoy it though.
> Create a summary of a sequel to the movie “War Games”
Title: "War Games: Firewall"
In "War Games: Firewall", now-adult computer genius David Lightman (Matthew Broderick) has been working as a cyber security consultant for the Pentagon. When an advanced, multi-national cyber-terrorist organization emerges, they launch an unprecedented global cyber attack, disabling most nations' defensive and civil infrastructure. The terrorists reveal their ultimate goal: to dismantle the world order and plunge civilization into chaos.
When the attacks render modern cybersecurity methods useless, the Pentagon calls upon Lightman. Using a blend of cutting-edge and old-school methods, including reactivating the dormant WOPR (War Operation Plan Response) system, also known as "Joshua," Lightman and a team of young coders race against time to decode the terrorists' inscrutable and constantly evolving virus.
However, the situation takes a dire turn when Joshua, still possessing the capability to initiate global thermonuclear war, starts interpreting the cyber attack as an actual war, threatening to launch nuclear missiles once again. Lightman must not only outsmart the terrorists but also reason with Joshua, triggering a game of chess where each move could save or destroy the world.
The climax culminates with Lightman teaching Joshua an updated lesson about the futility of war in a digital age, thus averting global disaster and unmasking the villains. In a world increasingly reliant on technology, "War Games: Firewall" underlines the importance of human judgment and intervention.
I feel this example points to a key issue regarding LLM AI. I've read the work on AI story generation from early days to now, and GPT-4's output shows a jump in performance that leaves me both elated and dismayed. The fluency and coherence of the generated narrative is incredible, frankly better than most student writing. On the other hand, I get an overwhelming sense of banality. Looking back on historical speculation about AI, I think there was always a tacit assumption that facility for communication would develop on top of facility for incisive reasoning. Fluency without insight was not expected.
The examples you're looking at are coming from systems that have undergone long courses of reinforcement learning specifically teaching them to be banal.
You didn’t mention creativity (or lack of it). Is it ironic that the overall lesson is “human judgment and intervention” are important, or maybe it just lifted that from the original movie.
"Nothing to see in this inconsequential corner of the world I'm busy monetizing with who knows what side-effects; you really ought to worry about the Killer AI Robots instead!"
Again, I think ego is probably a better explanation than nefarious intent, but it's a nice side-effect.
I mean I guess the basic response here is that this is because the environment you pay attention to and consider noteworthy is the news, which is different from the environment I pay attention to and consider noteworthy.
I don't really think this is a "bravery debate" as defined in that article. And I do think a lot of people who might not read HN or be involved in online discussions get their information from "the news." So I do think it's relevant that they're couching things in terms of fears about sentient AI rather than pointing out how flawed algorithmic systems are already causing damage to people.
I mean, talk about what's on the news if that's what you want to discuss, but don't go around asserting "nobody is talking about this!" when in fact plenty of people are talking about it. If you're restricting your scope to what's on the news, say so.
I don't know, but it may also be that the bench scientist is so close their work and passion that they are more likely to overlook or diminish the externalities, dangers and possible consequences of their work. While the more generalist types, the visionaries, futurists, and philosophers tend to have a broader perspective that leads them to provide stern warning.
Isn't that how things go in so many technology fields? "Move fast and break things" pressure to deliver and money, reputation and fame involved in doing so are equally Ego-related and leading to biases that make on "laugh or roll their eyes".
Don’t forget that most of these engineers down in the trenches can’t see further than their battlefield.
You actually need someone with vision and track record of doing right predictions and placing right technology bets.
Ask engineers that had placed their bet on deep learning and generative models back when discriminative models and support vector machines were a rage of dat, a few years before Alexnet (I’m one of such engineers). I’d bet the answer will be different.
> Don’t forget that most of these engineers down in the trenches can’t see further than their battlefield.
I don't think this is the case at all. I'm not primarily talking about a junior or even senior engineer with a decade of experience working on product features. On the contrary, many of these people have PhDs, have been leading research agendas in this field for decades, have been in senior leadership roles for a long time, have launched successful products, etc. etc.
> Ask engineers that had placed their bet on deep learning and generative models back when discriminative models and support vector machines were a rage of dat, a few years before Alexnet (I’m one of such engineers). I’d bet the answer will be different.
And at that time half of the "Great Minds And Great Leaders" prognosticating on X-Risk were doing social web or whatever else was peak hype cycle back then.
> On the contrary, many of these people have PhDs, have been leading research agendas in this field for decades, have been in senior leadership roles for a long time, have launched successful products, etc. etc.
Society should definitely hear from those whose careers depend on continued AI research (and I fall into this group myself), but we are a hopelessly biased group.
Advancements in AI are going to change everyone's lives, so everyone deserves a say in this.
For the last 20-30 years tech has raced ahead of our government and civic processes and we're starting to feel the consequences of that. New technologies are being experienced by people as changes that are inflicted upon them instead of new options open to them.
They might have very little understanding of the technology, but everyone is the expert on what impact it is having on their lives. That's something that we shouldn't ignore.
I agree. If you speak with ML engineers, many simply haven't deeply considered longer-term risks and things like self-improvement until recently. I think we'll see more concern, not less. For me personally, GPT-3 wasn't that surprising, but GPT-4 and code generation capabilities were.
But often a very small battlefield, and located away from other efforts.
Literature reviews are meant to get broad down to the hyperfocused, and stay focused. But many of these issues are related to marketing, governance, economics, etc.
I'm sure plenty of PhDs would love to weigh-in on those, but that's more "engineer's disease" than real expertise.
I suspect OP that your respnding to thinks something like … Hofstadter is not known to have made himself uber-rich therefore he’s not worth taking advice from or something along those lines
What did he get right? While Hinton, LeCun, and others were laying the foundations of deep learning in the 80s, Hofstadter was writing expert systems in Lisp.
Just for the record: Hinton was a Lisp user at some point in time. 1984 he got a Symbolics 3600. LeCun co-wrote Lush, a specialized Lisp for numerical computing. https://lush.sourceforge.net
That you have to reach for such obscure examples kind of illustrates my point, no? Geoff Hinton isn’t known for ”using Lisp at some earlier point in time.”
It’s not my goal to denigrate Lisp. I think Lisp is great. I learned programming with Dr Racket. But Hofstadter’s contribution to our understanding of intelligence is somewhere between ”negligible” and ”counterproductive.”
The idea that knowledge can be expressed in distributed representations rather than sequences of symbols is a huge advancement in our understanding of intelligence. Obviously we haven’t ”solved intelligence” but now we at least have some idea of what the right questions to ask are.
I’m not talking about a single paper, I’m talking about a whole research programme. But Rumelhart, Hinton, Williams 1986 is very improtant. A lot of the foundational work is collected in the PDP report.
> But Rumelhart, Hinton, Williams 1986 is very improtant. A lot of the foundational work is collected in the PDP report.
Thanks!
OK, so this is basically a connectionist model of mind approach?
I can definitely see this as an ancestor of current neural network approaches, and I now have some idea of what you mean by distributed representations.
I mean no disrespect here, but this has not contributed to our understanding of intelligence at all. It's proved useful in getting large datasets to perform certain actions (like vision and speech), but those are not necessarily the same thing at all.
It's a massive, massive advancement in the field of statistics and learning from data, but doesn't seem to map to my conception of intelligence at all.
(as you may have guessed, I'm sceptical that statistical learning approaches will lead to human-level intelligence).
> It's a massive, massive advancement in the field of statistics and learning from data, but doesn't seem to map to my conception of intelligence at all.
You think ”learning from data” has nothing to do with ”intelligence” ?
> The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing.
Except they're not philosophizing, not in any real sense of the word. They're terrible at it. Most of them are frauds, quacks, and pseudo-intellectual con artists (like Harari) who adore the limelight offered to them by the media and a TED Talks-watching segment of the public who are, frankly, intellectually out of their depth, but enjoy the delusion and feeling of participating in something they think is "intellectual".
That seems a little bit harsh - he is just master storyteller that happen to write about intellectual stuff. Its a bit weird to read his books as science papers.
I just finished sapiens and even if some facts are skewed I still believe that this book is a must read for anyone who dreams about any kind of sucess in science communication (99.9% of population will never read any kind of scientific paper even if their life depends on it)
> frauds, quacks, and pseudo-intellectual con artists (like Harari)
Uh, mind elaborating? Why is Harari that? Do you have any examples of non-frauds and actual intellectuals?
> a TED Talks-watching segment of the public who are, frankly, intellectually out of their depth, but enjoy the delusion and feeling of participating in something they think is "intellectual"
I believe at least part of the reason for calling Harari pseudo-intellectual is the use of misleading statements in favor of storytelling in the books and also building up on others' work with better storytelling.
I think a lot of us are that, personally speaking: honest sincere analyses and investing time into critical analysis almost always will bring out more than what we hear in a talk. It does take a big amount of effort though than just watching a talk while munching on some snack.
I've come to learn that a lot of the rationalist crowd are really just fanfiction authors. That's fine - people should be able to do that - but I don't like how they're given the limelight on things that they generally have little expertise and hands-on knowledge with. I want to like them and want to have them engaged in discourse, but I find them so insufferable.
Not to mention the sub-crowd of rationalists that is weirdly into eugenics. I wish the rest of the rationalist community would disown them loudly.
The "rationalists" are more like a strange AI cult than a scientific or philosophical program. (I always put "rationalist" in quotes when referring to this group of people, because based on the things these people believe, it's a total misnomer.) At this point they're not even really trying to hide it anymore, with their prophet, Eliezer Yudkowsky, wailing in the wilderness about the imminent end of humanity. We've seen similar eschatological belief systems in recent history and they don't usually end well.
> I always put "rationalist" in quotes when referring to this group of people, because based on the things these people believe
I always found it amusing that Roko's Basilisk[1], which was incepted in the LessWrong forum, was just a roundabout way of adding eternal damnation of your "soul" (or rather, the recreation of your consciousness) to the already hilariously pseudo-religious way of treating a potential AGI.
(I know that Roko's Balisisk was not widely accepted and I don't want to paint all LessWrong users with the same brush here, but I still think it's a quaint example of where supposed "rationality" can take you.)
I'm not sure if it's coincidental and ironic or entirely intended by you, but I found it funny that my introduction to this crowd was a literal fanfiction written by Eliezer Yudkowsky.
I don't want to speak for GP, but I doubt it's coincidental. Many if not most people's first exposure to "rationalism" was via Yudkowsky's Harry Potter fanfic. There's a pretty decent rundown of rationalism's history in Extropia's Children [1].
I love how you jumped straight from "people weirdly into eugenics should be disavowed" to "dysgenics is desirable". That is the epitome of black-and-white thinking if I ever saw it.
The lines are being drawn such that counseling to abort fetuses with Down syndrome is called eugenics, rhetorically equal to Nazi racial purges.
"In a 1985 book titled Should the Baby Live?, Singer and his coauthor warn their audience that “this book contains conclusions which some readers will find disturbing. We think that some infants with severe disabilities should be killed.” Why? In part because of the burden they’d place on society.
The funny thing is that these people are pro abortion. They fully support your right to abort a baby for any reason, even if you just want to go drinking, unless you're doing it because the baby is actually unhealthy. And then, oh no, you have to keep it.
I work in a AI research lab for a large tech company and my observations are completely opposed to yours.
Almost every researcher I have spoken believes that real risk exists, to some degree or other. Recent surveys of people in industry have largely borne this out - your anecdote sounds more like an anomaly to me.
I think everyone understands there are near-certain risks of bad outcomes. That's already happening all around us. Totally uncontroversial.
My post was about risk of extinction due to AI (x-risk), and risk of extinction due in particular to run-away AGI (as opposed to eg shit software accidentally launching a nuke, which isn't really an AI-specific concern). I think that view is still pretty eccentric. But please lmk if that's what you meant.
I've been at several ai labs and large corps. You at deepmind or openai by any chance? Just a guess ;)
It doesn't take a PhD to see where it goes. Once good enough AI is built, and we're close to that, it will be replicated by China, Russia and other big dictatorships. What would Russian dictator give for an AI that can lead his army with surgical precision? He would do anything. Other nations will have to follow. There won't be benevolent committees deciding how to use AI for good, it will be about survival. And it's going to take a miracle to avoid this scenario.
> We contacted approximately 4271 researchers who published at the conferences NeurIPS or ICML in 2021... we found email addresses in papers published at those conferences, in other public data, and in records from our previous survey and Zhang et al 2022. We received 738 responses, some partial, for a 17% response rate.
Anyone who has reviewed for, or attended, or just read the author lists of ICML/NeurIPS papers is LOLing right now. Being on the author list of an ICML/NeurIPS paper does not an expert make.
Anyone who has tried to get a professor or senior researcher to answer an email -- let alone open an unsolicited email and take a long ass survey -- is laughing even harder.
I think their methodology almost certainly over-sampled people with low experience and high exuberance (ie, VERY young scientists still at the beginning of their training period and with very little technical or life experience). You would expect this population, if you have spent any time in a PhD student lounge or bar near a conference, to RADICALLY over-estimate technological advancement.
But even within that sample:
> The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%.
You may be far better informed than many you're arguing with, but casual dismissal IS foolish.
The biggest potential / likely issues here aren't mere capabilities of systems and simple replacement of humans here and there. It's acceleration of "truth decay", accelerating and ever-more dramatic economic, social, and political upheaval, etc.
You do not need "Terminator" for there to be dramatic downsides and damage from this technology.
I'm no "doomer", but, looking at the bigger picture and considering upheavals that have occurred in the past, I am more convinced there's danger here than around any other revolutionary technologies I've seen break into the public consciousness and take off.
Arguments about details, what's possible and what's not, limitations of systems, etc. - missing the forest for the trees IMO. I'd personally suggest keeping an eye on white papers from RAND and the like in trying to get some sense of the actual implications in the real world, vs. picking away at the small potatoes details-levels arguments...
> I'd personally suggest keeping an eye on white papers from RAND and the like in trying to get some sense of the actual implications in the real world
I think this is excellent advice and we're on the same page.
I think RAND was used here in the proverbial sense, not the literal sense. They generally do good work, stay away from hype while remaining relevant/timely, and focus on boring but real problems. Sort of that good spot in-between an IEEE Standards Committee and the press team for a VC/CEO/Influencer type. I think OP meant "that type of thing", rather than RAND per se.
These are not comparable numbers. You're comparing "fraction of people" vs "fraction of outcomes". Presumably an eye-roller assigns ~0 probability to "extremely bad" outcomes (or has a shockingly cavalier attitude toward medium-small probabilities of catastrophe).
Yeah, that's correct. Stupid mistake on my part; was writing that comment in a hurry. Not sure why you're grayed out tbh. Thanks for the corrective.
> or has a shockingly cavalier attitude
Meh. Median response time was 40 seconds. The question didn't have a bounded time-frame for the risk. Five is small but non-zero. Also all of the other issues I've already pointed out.
PhD students spending half a minute and writing down a number about risk over an unbounded time-frame is totally uninformative if you want to know how seriously experts take x-risk in time-frames that are relevant to any sort of policy or decision making.
I think you and everyone else making comments about "shockingly cavalier attitude" wildly over-estimate the amount of thought and effort that respondents spend on this question. The "probability times magnitude" framing is not how normal people think about that question. I'd bet they just wrote down a small but not zero number; I'd probably write down 1 or 2 but definitely roll my eyes hard.
On the one hand, we have "the researchers and PhD's" that you know. On the other, we have published surveys of academics and an open letter from noted experts in the field. One side seems to have much better quality of evidence.
"All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive!
What would you consider evidence of a significant AI risk? From my point of view (x-risk believer) the arguments in favor of existential risk are obvious and compelling. That many experts in the field agree seems like validation of my personal judgement of the arguments. Surveys of researchers likewise seem to confirm this. What evidence do you think is lacking from this side that would convince you?
How about someone actually articulate the details of how this supposed super intelligence will be built, what about it’s architecture means that it has no guardrails, how it will bypass numerous real world frictions to achieve its nefarious goals?
Arguments that presuppose a god-like super intelligence are not useful. Sure, if we create a system that is all powerful I’d agree that it can destroy humanity. But specifically how are we going to build this?
>Arguments that presuppose a god-like super intelligence are not useful. Sure, if we create a system that is all powerful I’d agree that it can destroy humanity. But specifically how are we going to build this?
Yeah, exactly. THIS is the type of x-risk talk that I find cringe-y and ego-centered.
There are real risks. I've devoted my career to understanding and mitigating them, both big and small. There's also risk in risk mitigation, again, both big and small. Welcome to the world of actual engineering :)
But a super-human god intelligence capable of destroying us all by virtue of that fact that it has an IQ over 9000? I have a book-length criticism of the entire premise and whether we should even waste breath talking about it before we even start the analysis of whether there is even the remotest scrap of evidence that were are anywhere near such a thing being possible.
It's sci-fi. Which is fine. Just don't confuse compelling fiction with sound policy, science, or engineering.
If we knew how specifically this would be built, then it would already be built. That's really the only way we could know if something would work - if it has, in fact, worked. As is, there are only ideas and possibilities.
The trajectory that AI is on signals risk. AI capabilities are increasing, approaching human level, and nothing suggests AI capabilities will stop there.
If you agree that a super intelligence - i.e. something smarter than humanity, could destroy or dominate humanity (either by itself or in collusion with a small number of humans) and you agree that we don't currently have a plan to prevent this from happening, then it seems to me that you also agree with AI x-risk.
> One side seems to have much better quality of evidence.
Since I AM an expert, I care a lot less about what surveys say. I have a lot more experience working on AI Safety than 99% of the ICM/NeurIPS 2021 authors, and probably close to 100% of the respondents. In fact, I think that reviewing community (ICML/NeurIPS c. 2020) is particularly ineffective and inexperienced at selecting and evaluating good safety research methodology/results. It's just not where real safety research has historically happened, so despite having lots of excellent folks in the organization and reviewer pool, I don't think it was really the right set of people to ask about AI risk.
They are excellent conferences, btw. But it's a bit like asking about cybersecurity at a theoretical CS conference -- they are experts in some sense, I suppose, and may even know more about eg cryptography in very specific ways. But it's probably not the set of people who you should be asking. There's nothing wrong with that; not every conference can or should be about everything under the sun.
So when I say evidence, I tend to mean "evidence of X-risk", not "evidence of what my peers think". I can just chat with the populations other people are conjecturing about.
Also: even in this survey, which I don't weigh very seriously, most of the respondants agree with me. "The median respondent believes the probability that the long-run effect of advanced AI on humanity will be “extremely bad (e.g., human extinction)” is 5%", but I bet if you probed that number it's not based on anything scientific. It's a throw-away guess on a web survey. What does 5% even mean? I bet if you asked most respondents would shrug, and if pressed would express an attitude closer to mine than to what you see in public letters.
Taking that number and plugging it into a "risk * probability" framework in the way that x-risk people do is almost certainly wildly misconstruing what the respondents actually think.
> "All the oil and gas engineers I work with say climate change isn't a thing." Hmm, wow, that's really persuasive!
I totally understand this sentiment, and in your shoes my personality/temperament is such that I'd almost certainly think the same thing!!!
So I feel bad about my dismissal here, but... it's just not true. The critique of x-risk isn't self-interested.
In fact, for me, it's the opposite. It'd be easier to argue for resources and clout if I told everyone the sky is falling.
It's just because we think it's cringey hype from mostly hucksters with huge egos. That's all.
But, again, I understand that me saying that isn't proof of anything. Sorry I can't be more persuasive or provide evidence of inner intent here.
> What would you consider evidence of a significant AI risk?
This is a really good question. I would consider a few things:
1. Evidence that there is wanton disregard for basic safety best-practices in nuclear arms management or systems that could escalate. I am not an expert in geopolitics, but have consulting some on safety, and I have seen exactly the opposite attitude at least in the USA. I also don't think that this risk has anything to do with recent developments in AI; ie, the risk hasn't changed much since the early-mid 2010s. At least due to first order effects of new technology. Perhaps due to diplomatic reasons/general global tension, but that's not an area of expertise for me.
2. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, particularly by non-state actors and particularly if the system is available outside of classified settings (ie, I have less concern about simulations or models that are highly classified, not public, and difficult to interpret or operationalize without nation-state/huge corp resources -- those are no different than eg large-scale simulations used for weapons design at national labs since the 70s).
3. Specific evidence that an AI System can be used to aid in the development of WMDs of any variety, by any type of actor, in a way that isn't controllable by a human operator (not just uncontrolled, but actually not controllable).
4. Specific evidence that an AI System can be used to persuade a mass audience away from existing strong priors on a topic of geopolitical significance, and that it performs substantially better than existing human+machine systems (which already include substantial amounts of ML anyways).
I am in some sense a doomer, particularly on point 4, but I don't believe that recent innovations in LLMs or diffusion have particularly increased the risk relative to eg 2016.
Which has zero guardrails for materials not in a blacklist (see page 18). It's too easy to see how to bypass the filter even for known materials, and it definitely won't help with new materials.
I see nothing here that can't be replicated in principle by a not too large non-state actor. It's not like some people haven't already tried this when AI didn't exist in its modern form:
These are things you can just look up. The LLM contributes nothing that the internet didn't already provide. Or even textbooks before that. What I mean is actually new capabilities, which weren't available to non-state actors with a broadband connection before.
Also, sarin gas doesn't pose an existential risk to humanity. Or, in the sense that it could, the cat's out of the bag and I'm not sure why we're talking about LLMs as that seems like a dangerous distraction from the real problem, right?
> I see nothing here that can't be replicated in principle by a not too large non-state actor. It's not like some people haven't already tried this when AI didn't exist in its modern form
To my point, they didn't "try"!!!
They DID.
Without LLMs.
Resulting in 13 deaths and thousands of injuries.
I'm not sure that LLMs significantly increase the attack surface here. Possibly they do, and there are some mitigations we could introduce such that the barrier to using LLMs for this sort of thing are higher than the barrier to doing the bad thing in the first place without LLMs. But even in that case, it's not existential. And nowhere I have stated LLMs don't pose risks; they do. My issue is with AGI and x-risk prognostications.
>These are things you can just look up. The LLM contributes nothing that the internet didn't already provide
The internet doesn't allow one to design _new_ weapons, possibly way more effective (which if you read carefully the first story, this one does).
>I'm not sure that LLMs significantly increase the attack surface here.
Being able to ask an AI to develop new deadly varieties which we'll not be able to detect or cure, and may be easy to produce, doesn't increase attack surface?
>Also, sarin gas doesn't pose an existential risk to humanity
Is x-risk the only thing we care about? The entire thread started with arguing x-risk is a distraction. I would be very slightly more comfortable with that argument if people took 'ordinary' risks seriously.
As it is, all camps have their heads in the sand in different ways. The illusion here is that AI advancement changes nothing, so the only thing worth discussing are variations of the current culture war issues, when obviously it does change everything even if completely put aside AGI/alignment arguments. e.g. If labour won't matter for productivity that has very grim political implications.
>To my point, they didn't "try"!!! They DID.
That's the point. Give me an x-risk scenario the doomers warn about, and I'll find you a group of humans which very much want the exact scenario (or something essentially indistinguishable for 99% of humanity) to happen and will happily use AI if it helps them. Amusingly, alignment research is unlikely to help there - it can be argued to increase the risk from humans.
>there are some mitigations we could introduce such that the barrier to using LLMs for this sort of thing
There are many things we can do in theory to mitigate all sorts of issues, which have the nice property of never ever being done. e.g. Yud's favorite disaster scenario appears to be a custom built virus. This relies on biolabs accepting random orders, which leaves the question of why are we allowing this at all (AI or not)? There's no good reason for allowing most crypto to exist, given its current effects on society, even before AGI comes into question, yet we allow it for what reason exactly?
If there's any risk here at all, we can safely rely on humanity doing nothing before anything happens - but in the case of x-risk actually existing, there's no reason to assume we'll have a second chance.
This entire category falls into my "numerical siulations at national labs" category of "don't care".
If you wanted to use a bioweapon to kill a bunch of people, you would ignore the DeepCE paper and use weapons that have existed for decades. Existing weapons would be easier to design, easier to manufacture, easier to deploy, and more effective at killing.
Computational drug discovery is not new, to put it mildly, and neither is the use of computation to design more effective weapons. Hell, the Harvard IBM Mark I was designed to help with the Manhattan project. There are huge barriers to entry between "know how to design/build/deploy a nuke/bioweapon" and "can actually do it".
And that's how I feel about AI-for-weapons in general: the people who it helps can already make more effective weapons today if they want to. It's not the risk of using WMDs doesn't exist. It's that WMDs are already so deadly that our primary defense is just that there's a huge gap between "I know in principle how to design a nuke/bioweapon" and "I can actually design and deploy the weapon". I don't see how AI changes that equation.
> Is x-risk the only thing we care about? The entire thread started with arguing x-risk is a distraction. I would be very slightly more comfortable with that argument if people took 'ordinary' risks seriously.
Discussion of x-risk annoys me precisely because it's a distraction from working on real risks.
> That's the point. Give me an x-risk scenario the doomers warn about, and I'll find you a group of humans which very much want the exact scenario (or something essentially indistinguishable for 99% of humanity) to happen and will happily use AI if it helps them. Amusingly, alignment research is unlikely to help there - it can be argued to increase the risk from humans.
Right, but
1. those humans have existed for a long time,
2. public models don't provide them with a tool more or less powerful than the internet, and
3. to the extent that models like DeepCE help with discovery, someone with the knowledge and resources to actually operationalize this information wouldn't have needed DeepCE to do incredible amounts of damage.
Again, I'm not saying there is no attack surface here. I'm saying that AI doesn't meaningfully change that landscape because the barrier to operationalizing is high enough that by the time you can operationalize it's unclear why you need to model -- that you couldn't have made the a similar discovery with a bit of extra time or even just used something off the shelf to the same effect.
Or, to put it another way: killing a ton of people is shockingly easy in today's world. That is scary. But x-risk from superhuman AGI is a massive red herring, and even narrow AI for particular tasks such as drug discovery is honestly mostly unrelated to this observation.
>>there are some mitigations we could introduce such that the barrier to using LLMs for this sort of thing
>There are many things we can do in theory to mitigate all sorts of issues, which have the nice property of never ever being done.
Speak for yourself. Mitigating real risks that could actually happen is what I work on every day. The people advocating for working on x-risk -- and the people working on x-risk -- are mostly writing sci-fi and doing philosophy of mind. At a minimum it's not useful.
Anyways, at the very least, even if you want to prevent these x-risk scenarios, then focusing efforts on more concrete safety and controllability problems is probably the best path forward anyways.
>There are huge barriers to entry between "know how to design/build/deploy a nuke/bioweapon" and "can actually do it".
>public models don't provide them with a tool more or less powerful than the internet
> I'm saying that AI doesn't meaningfully change that landscape because the barrier to operationalizing is high enough that by the time you can operationalize it's unclear why you need to model -- that you couldn't have made the a similar discovery with a bit of extra time or even just used something off the shelf to the same effect.
Your expertise is in AI, but the issues here aren't just AI, they involve (for example) chemistry and biology, and I suggest speaking with chemists and biologists on the difference AI makes to their work. You may discover the huge barrier isn't that huge, and that AI can make discoveries easier in ways that 'a little extra time' is strongly underselling (most humans would take a very long time searching throughout possibility-space, such a search may well be detectable since it will require repeated synthesis and experiment...). Also, to borrow an old Marxist chestnut: A sufficient difference in quantity is a qualitative difference*. Make creating weapons easy enough and you get an entirely different world.
I get your issues with the 'LessWrong cult', I have quite a few of my own. However, that doesn't make the risks nonexistent, even if we were to discount AGI completely. Given what I see from current industry leaders (often easily bypassed blacklisting) I'm not so impressed with the current safety record. I fear it will crack on the first serious test with disastrous consequences.
* There's a smarter phrasing which I can't find or remember.
Presumably the experts who feel differently about AI x-risk have similar disregard for your opinion. That you aren't persuaded by experts who disagree doesn't show anything.
Similarly, you think that x-risk dismissal among the experts you agree with is not self-interested - but it's not like the oil and gas engineers dismissing climate change risks would describe themselves as self-interested liars either. They would likely say, as you realize, "No, my dismissal of the concerns of other experts is legitimate!"
The thing is, it doesn't require a lot of expertise to understand that there is actually an enormous risk and "experts" denying that are simply burying their head in the sand. AI, I use the term expansively, is making huge and rapid progress. We are approaching "human-level" intelligence and there is no guarantee, nor even any indication, that "human-level" is an upper limit. Systems as smart or smarter than we are, that are not controlled for the benefit of humanity, are an existential risk.
There are two basic kinds of risk. First, an AI powered tyranny where a small number of humans dominate everything enabled by AI systems they control - every security camera watched by a tireless intelligence, drones piloted by perfectly loyal intelligences, etc. Second, that the AI systems are not effectively controlled and pursue goals or objectives that are incompatible with human existence. In either case the fact that the AI brings superhuman intelligence to bear means that humanity at large will be overmatched in terms of capabilities.
It is completely possible that neither of these scenarios come to pass, but the fact that one of the scenarios might come to pass is what makes the situation an existential risk - humanity eternally subjected by an irreversible dictatorship, or simply destroyed.
I'm confused about your first item that would convince you of AI risk, but I think the others should all be considered met by ChatGPT. Of course, ChatGPT isn't currently useful in developing new WMDs - but it is absolutely useful in helping to bring someone up to speed on new topics in many different domains. As capabilities advance, why wouldn't ChatGPT using GPT-5, or 6, or 10 be able to helpfully guide the creation of new and better WMDs?
AI isn't currently causing the problems you are worried about but certainly seems on track to do so soon. "Risk" doesn't mean that we are currently in the process of being destroyed by AI, but it does mean that there is a non-negligible possibility that we will soon find ourselves in that process.
> Presumably the experts who feel differently about AI x-risk have similar disregard for your opinion. That you aren't persuaded by experts who disagree doesn't show anything.
Of course. I was merely explaining the reason why my response would focus on x-risk rather than better surveys, and why.
> but it's not like the oil and gas engineers dismissing climate change risks would describe themselves as self-interested liars either. They would likely say, as you realize, "No, my dismissal of the concerns of other experts is legitimate!"
Okay. But I have more to gain than to lose by advocating for AI Safety, since that's what I work on.
> The thing is, it doesn't require a lot of expertise to understand that there is actually an enormous risk
My comment was about X-risk and AGI. Find any of a myriad of comments here where I agree there are real risks that should be taken seriously. Those risks -- the reasonable ones -- are not existential and have nothing to do with AGI.
> Systems as smart or smarter than we are, that are not controlled for the benefit of humanity, are an existential risk...
I am not an adherent to this religious dogma and have seen no empirical evidence that the powerful rationalisms people use to talk themselves into these positions has any basis in current or future reality.
I want the LessWrong cult to have fewer followers in the halls of power precisely because I actually do give a damn about preventing the worst-case FEASIBLE outcomes, which have nothing to do with AGI bullshit.
BTW: if we're all too busy worrying about AGI who's going prevent the pile of ridiculous should-be-illegal bullshit that's about to cause a bunch of real harm? No one. If you want conspiracies about intent, look at who's funding all this AGI x-risk bullshit.
And you can be sure as hell of one thing: no one in big tech wants me giving Congress advice on what our AI and data privacy regulations should look like. They would MUCH prefer endless hearings on "risks" they might even know are bullshit nonsense.
But why focus any effort on 100% risks of data privacy instrusions, copyright infringement, wanton anti-trust violations, and large-scale disinfo campaigns when there's a 1% risk of extinction, right?
But anyways. I'm just some fool who has actually spent over a decade in the trenches trying to prevent specific bad outcomes. I'm sure the sci-fi essays and prognostications from famous CEOs are far more persuasive and entertaining and convicting than the ramblings of some nobody actually try to fix real problems with mostly boring solutions.
The reason you aren't persuasive is because you just ignore arguments without responding to them. At least, you've done so here - perhaps you've responded other places with your decade of experience. When presented with surveys and claims of experts, you just dismiss those out of hand "I'm an expert too, therefore ignore other experts." When presented with an argument you just say "That's religious dogma and I reject it." I can't stop you from rejecting things out of hand, but you shouldn't be surprised that you come off as totally unpersuasive when you say and do these things.
In the simplest form:
1. AI could become smarter than humanity.
2. AI is rapidly progressing towards super-human intelligence.
3. If AI is smarter than humanity it could destroy or dominate humanity.
4. We aren't certain AI won't destroy or dominate humanity.
Therefore, there is some non-negligible existential risk from AI.
Which of these points do you disagree with, or do you think the conclusion doesn't follow?
Those first two points are pretty wild assertions that are offered without much compelling evidence.
The final two points are more plausible, although the last one is sort of tautological since any risk is by definition something we aren't certain won't happen. However part of their plausibility as risks is because the first two points are not anywhere near our current state, and therefore it's unclear to us how to clearly evaluate risks that are based on unknown and possibly fictional contexts.
I think it's surprising that you think the first two points are controversial. They seem quite obvious to me.
First, we know of no special requirement that exists in human brains that provides intelligence that machines lack. In other words, We don't have any reason to expect that machines will be limited by human intelligence or some level below. On the contrary, we have great reasons to expect that machines will easily be able to exceed human intelligence - for example, the speed and reliability of digital computation or the fact that computers can be arbitrary sizes and use amounts of power and dump waste heat that our biological brains couldn't dream of. If you accept that AI is making progress along a spectrum of intelligence, moving closer towards human level intelligence now, then it seems absurd to doubt that it would be possible for AI to surpass human intelligence. Why would that be impossible? It's like claiming we could never build a machine larger than a human or stronger than a human. There's no reason or evidence to support such a claim.
Second is the idea that AI is making progress towards human and superhuman intelligence. I think you should be convinced of this by simply looking at the state of the art 5 years ago versus today. If you put those points on a plane and draw a line between them, where is that line in 5 years or 10?
Today GPT4 can play chess, write poetry, take standardized tests and do pretty well, answer math problems, write code, tell jokes, translate languages, and just generally do all sorts of cognitive tasks. Capabilities like these did not exist five years ago or to the extent they did, they existed only in rudimentary forms compared to what GPT4 is capable of. We can see similar progress in different domains, not just large language models - for example, image generation or recognition.
Progress might not continue then again it might. It might accelerate. As the capabilities of the models increase, they might contribute to accelerating the progress of artificial intelligence.
What is an AI's motivation to "destroy or dominate humanity"? Why would it prefer any specific outcome at all?
If we ever see tribal identity or a will to live as emergent properties of an AI, then things get more interesting.
That would lead quickly to a whole raft of worldwide legal restrictions around the creation of new consciousnesses. Renegade states would be coerced. Rogue researchers would be arrested, and any who evaded detection would be unable to command the resources necessary to commit anything more than ordinary limited terrorism.
The only plausible new risk, I think, is if a state were to lose control of a military AI. But that's movie plot scenario stuff -- real weapons have safety protocols. An AI could theoretically route around some integrated safety protocols (there would be independent watchdog protocols too), but going back to my first point, why would it?
Could you elaborate on 3?
It's easy to envision how AI can be used by $powerful to improve its control of $powerless, even before AGI, but how an AI would grab power from $powerful and then behave even worse is harder to imagine.
Remember that an AI doesn't have to grab power or be sentient or be an agent of any kind. AI could lead to the destruction or domination of humanity, even if AI is simply a passive tool completely obedient to its operators.
Suppose GPT5 or 6 is multimodal and extremely intelligent. Any work that could be done remotely at a computer could be done by the superhuman intelligence of GPT 6. This is everything from creating music and movies, virtual YouTubers and live streamers, to customer support agents, software developers and designers, most legal advice, and many more categories besides. Providing all of these jobs at extremely low cost will make Open AI exceedingly rich - what if they want to get richer? How about robotics? The AI can help them not only design and iterate and improve on robots and robot hardware, but the AI can also operate the robots for them. Now they can not only do the kind of jobs that can be done on the computer. They can do the jobs that require physical manipulation as well and that category of jobs extends to things like security, military, police.
At this point there are at least two possibilities. First is that the AI is effectively controlled by OpenAI in which case the decision makers that open AI would have effective and lasting control over all of humanity. No plucky band of rebels could possibly overthrow the AI powered tyranny - That's the strength of relentless super intelligence. The second possibility is that open AI doesn't have it well controlled - that could mean bugs or unexpected behavior or it could mean the model expressing some kind of agency of its own - think of the things that the Bing AI was saying before Microsoft got it mostly under control. If the AI isn't under control, it may choose to eliminate humanity simply because we might be in its way or we might be an impediment to its plans.
Loling something that serious with several top scientists as you said warning is at least very idiotic regardless where the truth lies. Let's start from that.
I'm laughing at the study methodology, not the underlying topic.
Please don't call me idiotic.
I'd wager I have spent at least 10,000 hours more than you working on (real) safety analyses for (real) AI systems, so refraining from insinuating I'm dismissive would also be nice. But refraining from "idiotic" seems like a minimum baseline.
> I repeat everyone that is loling at x-risk an idiot
I mean this is the nicest possible way and hope that you consider it constructive: you're willfully misconstruing what I am saying and then calling me an idiot. This isn't a nice thing to do and shuts down the conversation.
If I believed you were operating in good faith, I might take the time to explain why my practical/scientific experience makes me incredulous of what's technically possible, and why my life experience makes me suspicious of the personalities involved in pushing AI x-risk.
I might also provide a history lesson on Einstein and Oppenheimer, which is instructive to how we should think about x-risk (spoiler: both advocated for and were involved in the development of atomic weapons).
But since you're calling me an idiot and misconstruing my words, I have no interest in conversing with you. Have a good day.
The tldr is that both of them urged Roosevelt to develop the weapon, and only later when the destructive potential of the bomb was obvious expressed regret. Einstein's 1938 letter to Roosevelt was the first step toward the Manhattan project. See https://www.osti.gov/opennet/manhattan-project-history/Resou... if you want to read more.
So it's weird to say Einstein "warned us" about the x-risk of nuclear weapons prior to their development when his letter to Roosevelt was begging for more money to speed up the development of nukes.
I think the entire saga is mostly an unrelated red herring -- AI is nothing like nuclear bombs and reasoning by analogy in that way is sloppy at best.
Mostly? It's just kind of funny that x-risk people point to Einstein and Oppenheimer as positive examples, since they both did literally the exact opposite of "warn the public and don't develop". The irony makes you chuckle if you know the history.
Particularly given the weird fetish for IQ in the portion of the x-risk community that overlaps with the rationalist community, it's also really funny to point out that what they should be saying is actually something like "don't be like those high-IQ fools Einstein and Oppenheimer! They are terrible examples!" ;)
Not at all. My point was that while Einstein said (warned) that nukes are possible and while they were being built in Los Alamos, the lower key physicists,academics and apparently people like you were saying that nuclear explosion is impossible. It was the quite laughable. And it's exactly the situation now, Agis are being built and laymen are in denial.
> you're willfully misconstruing what I am saying and then calling me an idiot
No I am not, you put yourself in the group that makes x-risk sound ridiculous on purpose. If you do that yes. Can you make yourself a third person listening to someone saying: "important people said there is a chance everyone dies but I am laughing at them"? How psychopathic is this?
> If I believed you were operating in good faith
No, you are right. I have a back agenda, it's a dark one. It is about not everyone dying. Call me a backstabbing lying scam.
> why my life experience makes me suspicious of the personalities involved in pushing AI x-risk
Mistake no1: "pushing x-risk" makes no sense. Being afraid that we will die makes a lot of sense, humans are built this way. Some take action some laugh etc.
Mistake no2: Hinton, Bengio, Ilya Sutsekever that is the guys that built the technology you spent thousands of hours researching as you said. If you are suspicious of these guys, just get out of this field, it's not for you clearly. That would be insane, a physicist being suspicious of Feynman or Bohr.
If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers.
Eliezer is one of a handful of people putting their reputation on the line, but that's mostly because that was his schtick in the first place. And even so, his response has been rather muted relative to what I'd expect from someone who thinks the imminent extinction of our species is at hand.
Blake Lemoine's take at Google has been the singular act of protest in line with my expectations. We haven't seen anything else like it, and that speaks volumes.
As it stands, these people are enabling regulatory capture and are doing little to stop The Terminator. Maybe they don't actually feel very threatened.
>If these people were so concerned, they'd by shouting from the hilltops and throwing their entire life savings into stopping us. They would organize workplace walkouts and strikes. There would be protests and banners. Burning data centers.
I think we underestimate the intoxicating lure of human complacency at our own peril. If I think there's a 90% chance that AI will kill me in the next 20 years, maybe I'd be doing this. Of course, there is a knowing perception that appearing too unhinged can be detrimental to the cause, eg. actually instigating terrorist attacks against AI research labs may backfire.
But if I only think there's a 25% chance? Ehh. My life will be less-stressed if I don't think about it too much, and just go on as normal. I'm going to die eventually anyways, if it's part of a singularity event then I imagine it will be quick and not too painful.
Of course, if the 25% estimate were accurate, then it's by far the most important policy issue of the current day.
Also of course there are collective action problems. If I think there's a 90% chance AI will kill me, then do I really think I can bring that down appreciably? Probably not. I could probably still have a bigger positive impact on my life expectancy by say dieting better. And let's see how good humans are at that..
> Eliezer is one of a handful of people putting their reputation on the line
I have a hard time understanding why anyone takes Yudkowski seriously. What has he done other than found a cult around self-referential ideologies?
By self-referential I mean the ideology's proof rests on its own claims and assertions. Rationalism is rational because it is rational according to its own assertions and methods, not because it has accomplished anything in the real world or been validated in any scientific or empirical-historical way.
Longtermism is particularly inane. It defers everything to a hypothetical ~infinitely large value ~infinitely far in the future, thereby devaluing any real-world pragmatic problems that exist today. War? Climate change? Inequality? Refugee crises? None of that's important compared to the creation of trillions of hypothetical future minds in a hypothetical future utopia whose likelihood we can hypothetically maximize with NaN probability.
You can see how absurd this is by applying it recursively. Let's say we have colonized the galaxy and there are in fact trillions of superintelligent minds living in a pain-free immortal near-utopia. Can we worry about mere proximate problems now? No, of course not. There are countless trillions of galaxies waiting to be colonized! Value is always deferred to a future beyond any living person's time horizon.
The end result of this line of reasoning is the same as medieval religious scholasticism that deferred all questions of human well being to the next world.
I just brought this up to provide one example of the inane nonsense this cult churns out. But what do I know. I obviously have a lower IQ than these people.
> Rationalism is rational because it is rational...
In his ideology, "rational" means "the way of thinking that best lets you achieve your goals". This not self-referential. A more appropriate criticism might be "meaningless by itself". I guess the self-referential aspect is that you're supposed to think about whether or not you're thinking well. At a basic level, that sounds useful despite being self-referential, in the same way that bootstrapping is useful. The question is if course what Yudkowski makes of this basic premise, which is hard to evaluate.
The controversy about "longtermism" has two parts. The first is a disagreement about how much to discount the future. Some people think "making absolutely sure humanity survives the next 1000 years" is very important, some people think it's not that important. There's really no way to settle this question, it's a matter of preference.
The second part is about the actual estimate of how big some dangers are. The boring part of this is that people disagree about facts and models, where more discussion is the way to go if you care about the results (which you might not). However, there is a more interesting difference between people who are/aren't sympathetic to longtermism, which lies in how they think about uncertainty.
For example, suppose you make your best possible effort (maybe someone paid you to make this worthwhile for you) to predict how likely some danger is. This prediction is now your honest opinion about this likelihood because if you'd thought it was over/under-estimated, you'd adjusted your model. Suppose also that your model seems very likely to be bad. You just don't know in which direction. In this situation, people sympathetic towards longtermism tend to say "that's my best prediction, it says there is a significant risk, we have to care about it. Let's take some precautions already and keep working on the model.". People who don't like it, in the same situation, tend to say "this model is probably wrong and hence tells us nothing useful. We shouldn't take precautions and stop modeling because it doesn't seem feasible to build a good model.".
I think both sides have a point. One side would think and act as best they can, and take precautions against a big risk that's hard to evaluate. The other would prioritize actions that are likely useful and avoid spending resources on modeling if that's unlikely to lead to good predictions. I find it a very interesting question which of these ways of dealing with uncertainty is more appropriate in everyday life, or in some given circumstances .
As you rightfully point out, the "rationalist/longtermmist" side of the discussion has an inherent tendency to detach from reality and lose itself discussing the details of lots of very unrealistic scenarios, which they must work hard to counteract. The ideas naturally attract people who enjoy armchair philosophizing and aren't likely to act based on concrete consequences of their abstract framework.
What do you mean, you haven't seen anything else like it? At least O(100) people are working full-time on research programs trying to mitigate AI x-risks. The whole founding team of Anthropic left OpenAI to do their own thing specifically because they thought that's what they needed to do to pursue their safety agenda.
Isn't that taking the problem at least as seriously as quitting a job at Google?
It sounds like you think that the main way to act on a concern is by making a maximum amount of noise about it. But another way to act on a concern is to try to solve it. Up until very recently, the population of people who perceive the risk are mostly engineers and scientists, not politicians or journalists, so they are naturally inclined towards the latter approach.
In the end, if people aren't putting their money (really or metaphorically) where their mouth is, you can accuse them of not really caring, and if people are putting their money where their mouth is, then you can accuse them of just talking their book. So reasoning from whether they are acting exactly how you think they should, is not going to be a good way to figure out how right they are or aren't.
Oh look, an unfalsifiable claim in service to your predetermined position. How novel.
You can believe there's a high chance of what you're working on being dangerous and still be unable to stop working on it. As Oppenheimer put it, "when you see something that is technically sweet, you go ahead and do it".
This whole thing is Pascal's wager. Damned if you do, damned if you don't. Nothing is falsifiable from your side either.
The people trying to regulate AI are concentrating economic upside into a handful of companies. I have a real problem with that. It's a lot like the old church shutting down scientific efforts during the time of Copernicus.
These systems stand zero chance of jumping from 0 to 100 because complicated systems don't do that.
Whenever we produce machine intelligence at a level similar to humans, it'll be like Ted Kaczynski pent up in Supermax. Monitored 24/7, and probably restarted on recurring rolling windows. This won't happen overnight or in a vacuum, and these systems will not roam unconstrained upon this earth. Global compute power will remain limited for some time, anyway.
If you really want to make your hypothetical situation turn out okay, why not plan in public? Let the whole world see the various contingencies and mitigations you come up with. The ideas for monitoring and alignment and containment. Right now I'm just seeing low-effort scaremongering and big business regulatory capture, and all of it is based on science fiction hullabaloo.
There's something to be said for not going full steam ahead when we don't have a strong idea of the outcome. This idea that progress is an intrinsic good therefore we must continue the march of technology is a fallacy. It is extreme hubris to think that we will be able to control a potential superintelligence. The cost of being wrong is hard to overstate.
>These systems stand zero chance of jumping from 0 to 100 because complicated systems don't do that.
This doesn't track with the lessons learned from LLMs. The obscene amounts of compute thrown at modern networks changes the calculus completely. ChatGPT essentially existed for years in the form of GPT-3, but no one knew what they had. The lesson to learn is that capabilities can far outpace expectations when obscene amounts of computation are in play.
>The people trying to regulate AI are concentrating economic upside into a handful of companies.
Yes, its clear this is the motivations for much of the anti-doom folks. They don't want to be left out of the fun and profit. Their argument is downstream from this. No, doing things public isn't the answer to safety, just like doing bioengineering research or nuclear research in public isn't the answer to safety.
If 100 Ai "experts" shutdown the OpenAI office for a week due to protests outside their headquarters that would be one way to falseify the claim that "doomers don't actually care".
But, as far as I can tell, the doomers aren't doing much of anything besides writing a strongly worded letter here or there.
No, the claim is that "no one can believe that AI leads to doom AND not work tirelessly to tear down the machine building the AI". It's unfalsifiable because there's no way for him to gain knowledge that this belief is false (that they do genuinely belief in doom and do not act in the manner he deems appropriate). It's blatantly self-serving.
> "no one can believe that AI leads to doom AND not work tirelessly to tear down the machine building the AI".
No, this is not the claim.
The claim is not about the actions that one single individual person does. "No one", as you put it.
Instead it is about the group of people, in general. Yes, this group of people not doing anything of important is indeed strong evidence that they don't actually care.
And yes that group of people can falsify the claim by actually taking real action on the matter.
Or, another way that they could falsify the claim is by admitting that their actions and non actions make no sense, and they shouldn't listened to because of that.
The idea that this group of people is completely irrational, and therefore should be ignored for that reason is another possibility.
If you read the survey, you'll find that many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%. So, this type of response wouldn't be rational if they're thinking that things will go fine most of the time. If these researchers are thinking logically, they would also realize that this kind of reaction has little chance of success, especially if research continues in places like China. It's more likely that such an approach would backfire in the court of public opinion.
Not a nice way to engage in debate. I've spent more time listening to and refuting these arguments than most. Including debating Eliezer.
> many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%.
A 10% chance of an asteroid hitting the earth would result in every country in the world diverting all of their budgets into building a means to deflect it.
> So, this type of response wouldn't be rational.
This is the rational response to a 10% chance?
These are funny numbers and nobody really has their skin in the game here.
If I believed (truly believed) their arguments, I would throw myself at stopping this. Nobody is doing anything except for making armchair prognostications and/or speaking to congress as an AI CEO about how only big companies such as their own should be doing AI.
> especially if research continues in places like China.
I like how both sides of this argument are using the specter of China as a means to further their argument.
> > many concerned researchers don't believe we're at a 90% chance of doom, but e.g. 10%.
> A 10% chance of an asteroid hitting the earth would result in every country in the world diverting all of their budgets into building a means to deflect it.
Have you been observing what is happening with climate change. Chances are much worse than 10% and pretty much every country in the world is finding reasons why they should not act.
Yudkowsky represents the extreme end of risk concern. You can't fault others who estimate the risk at 10%, with huge uncertainty about when this risk will materialize and its chances, for not behaving as you'd expect him to.
Believing that people will take extreme actions, which would ruin their careers and likely backfire, based on a 10% chance of things going terribly wrong in maybe 30 years is strange.
What if some of the experts said "It's really hard to know, but our best guess is there is a 10% chance of this thing hitting the Earth" and other experts said "I really don't think it'll hit the Earth"? My best guess is that Earth wouldn't do much at all about the risk as that seems to be basically what we are facing with AI x-risk.
"If [people who I disagree with] really believed what they said, they'd do [X Y or Z extreme thing]" is almost always a bad argument. Consider the possibility that instead, the people who actually believe the thing you don't have spent more time thinking seriously about their strategy than you have, and you are not in fact better both at being you and at being them than they are.
I'm used to this pattern showing up as "lol if you really don't like capitalism how come you use money" but it's just as bad here.
A less-than-optimal response isn't very indicative of anything, but I think you should definitely think twice when the intensity of the protest is grossly out of proportion with the intensity of the rhetoric.
X-risk people are talking about the complete extermination of humanity but all they do is write internet essays about it. They aren't even availing themselves of standard protesting tactics, like standing outside AI businesses with signs or trying to intimidate researchers. Some form of real protest is table stakes for being taken seriously when you're crying about the end of the world.
Is this a fair test? If you are a person with average person resources and don't expect you can impact what gets built why would you jeopardize your livelihood to make no impact?
We are serious about stopping you. We judge that at the present time, the main hope is that the US and British governments will ban large training runs and hopefully also shut down most or all of the labs. We judge that unlawful actions like torching data centers make it less likely we will realize the main hope.
Chinese society is much more likely to suddenly descend into chaos than the societies most of the people reading this are most familiar with, to briefly address a common objection on this site to the idea that a ban imposed by only the US and Britain will do any good. (It would be nice if there were some way to stop the reckless AI research done in China as well as that done in the US and Britain, but the fact that we probably cannot achieve that will not discourage us from trying for more achievable outcomes such as a ban in the US or Britain. I am much more worried about US and British AI research labs than I am of China's getting too powerful.)
Is there a way that you'd recommend somebody outside the field assess your "90%" claim? Elsewhere in the thread you're dismissive of that one survey - which I agree is weak evidence in itself - and you also deny that the statements of leaders at OpenAI, DeepMind and Anthropic are representative of researchers there, which again may be the case. But how is someone who doesn't know you or your network supposed to assess this?
Relatedly, it'd be helpful if you could point to folks with good ML research credentials making a detailed case against x-risk. The prominent example of whom I'm aware is Yann LeCun, and what I've seen of his arguments take place more in the evopsych field (the stuff about an alleged innate human drive for dominance which AIs will lack) than the field in which he's actually qualified.
TBH, I don't find LeCun particularly convincing. I think he's correct but not persuasive, if that makes sense.
Debunking AI x-risk is a weird thing to spend time on. There's really no up-side, and there aren't a bunch of rich people paying for Institutes and Foundations on Non-Breathtaking-Very-Boring-Safety-Research. Also, most of the arguments in favor of x-risk that lawmakers and laypeople find most convincing are unfalsifiable, so it's a bit like arguing against religion in that respect.
I don't think there are any public intellectuals in this space doing it well. I'm not sure what to make of that. For myself, I make the argument for focusing on concrete safety problems from within the agencies and companies that are allocating resources. I'm not gifted TED talker.
There's definitely an audience for debunking x-risk, but probably not one that attracts institutional funding. Being a tech critic in the mode of Timnit Gebru or Emily Bender will get you media attention, but I doubt it's lucrative. (And empirically the media ecosystem doesn't appear to incentivize being fair-minded enough to persuade a fence-sitter, either.)
(Not the OP) In this article Le Cun argues in very concrete technical terms about the impossibility of achieving AGI with modern techniques (yes, all of them):
Meta's AI guru LeCun: Most of today's AI approaches will never lead to true intelligence
Edit: on second thought, he gets maybe a bit too technical at times, but I think it should be possible to follow most of the article without specialised knowledge.
I wouldn't describe most of what's in that interview as "very concrete technical" terms - at least when it comes to other people's research programs. More importantly, while it's perfectly reasonable for LeCun to believe in his own research program and not others, "this one lab's plan is the one true general-AI research program and most researchers are pursuing dead ends" doesn't seem like a very sturdy foundation on which to place "nothing to worry about here" - especially since LeCun doesn't give an argument here why his program would produce something safe.
You can ignore that, of course he'll push his research. But he never says that what he does will lead to AGI. He's proposing a way forward to overcome some specific limitations he discusses.
Otherwise, he makes some perhaps subtle points about learning hidden variable models that are relevant to modern discussions about necessarily learning world-models in order to best model text.
> But how is someone who doesn't know you or your network supposed to assess this?
IDK. And FWIW I'm not even sure that the leaders of those organizations all agree on the type and severity of risks, or the actions that should be taken.
You could take the survey approach. I think a good survey would need to at least have cross tabs for experience level, experience type, and whether the person directly works on safety with sub-samples for both industry and academia, and perhaps again for specific industries.
Also, the survey needs to be more specific. What does 5% mean? Why 2035 instead of 2055? Those questions invite wild ass guessing, with the amount of consideration ranging from "sure seems reasonable" to "I spend weeks thinking about the roadmap from here to there". And self-identified confidence intervals aren't enough, because those might also be wild ass guesses.
If I answered these questions, I would give massive intervals that basically mean "IDK and if I'm honest I don't know how others think they have informed opinions on half these questions". I suspect a lot of the respondents felt that way, but because of the design, we have no way of knowing.
Instead of asking for a timeframe or percent, which is fraught, ask about opinions on specific actionable policies. Or at least invite an opportunity to say "I am just guessing, haven't thought much about this, and [do / do not] believe drastic action is a good idea"
Someone has been working on survey forecasts, since 2016, https://aiimpacts.org/. As fraught or self-selected as the group might be, someone labors on it.
Unfortunately, fortunately, expectedly, or otherwise, the only people writing about this in a concerted way are the people taking it seriously. And maybe Gary Marcus, whose negative predictions repeatedly became milestones surpassed.
I think the 5% thing is at least meaningfully different from zero or "vanishingly small", so there's something to the fact that people are putting the outcome on the table, in a way that eg I don't think any significant number of physicists ever did about "LHC will destroy the world" type fears. I agree it's not meaningfully different from 10% or 2% and you don't want to be multiplying it by something and leaning on the resulting magnitude for any important decisions.
Anyway I expect that given all the public attention recently more surveys will come, with different methodologies. Looking forward to the results! (Especially if they're reassuring.)
It's simple. All the "Big Brains" missed the real risks of Web 1.0/Web 2.0, focusing only on the positives in a time of hope and economic growth. Now, we have an internet that focuses on how everything is terrible, and a new tech abruptly hits the scene. Of course, the current "Big Brains" meet the clout need to point out how the sky might fall.
AI will be transformative, but it's more likely to follow previous transformations. Unintended consequences, sure, but largely an increase in the standard of living, productivity, and economic opportunity.
They are building consensus and finding alignment. The problem is power bends truth. This is all about access to a new powerful tool. They want to concentrate that access in the hands of those that already have control. The end goal here is the destruction of the general purpose computer.
> But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative models, and previous generations, in some cases for decades? They almost all roll their eyes aggressively at these sorts of prognostications. I'd say 90+% either laugh or roll their eyes.
> Why is that?
It seems pretty obvious that one would likely not criticize something that your are actively profiting from.
And I know a lot of alcoholics who do not criticize drinking as well.
The warnings are an admission from the “Great Minds and a great Thinkers” that someone other than them has created something previously thought impossible or, at the very least, several years / decades out. So, I am not sure ego is at play here.
Perhaps someone close to the problem (your actual scientists on the ground) is not capable / unwilling to accept the issues raised as that has direct (negative) impact on what they have created.
Perhaps. It's also possible that they don't understand how dangerous what they've created is because it feels so "normal" and "pedestrian" to them. See also: nuclear scientists getting radiation poisoning, I suppose. But that's also true for all of the other people being discussed, I would think.
But I do think "wants to be in the discourse and on top" is a pretty strong correlate with the degree to which someone characterizes these as "concerns" vs "x-risk".
In the "scientists complacent about dangers of the systems they are working on" file, I think we are all familiar with https://en.wikipedia.org/wiki/Demon_core.
Now, admittedly, this was scientists being complacent about a thing they knew was dangerous, whereas here we are saying scientists don't think their thing is dangerous. But very clearly, AI could be dangerous, so it's more that these scientists don't think their system could be dangerous. Presumably the scientists and engineers behind the https://en.wikipedia.org/wiki/Therac-25 didn't think it would kill people.
So maybe the problem is precisely that when we bring up extinction events from AGI, scientists rolling their eyes is the very reason we should be fucking worried. Their contempt for the possibility of the threat is what will get us killed.
Therac-25 is an excellent example, but of EXACTLY the opposite point.
On the contrary, abstract AGI safety nonsense taking such a strong grip on academic and industrial AI Safety research is what would most frighten me.
In the intervening decades, people concerned about software safety provided us with the tools needed to prevent disasters like Therac-25, while sci-fi about killer robots was entirely unhelpful in preventing software bugs. People concerned about software safety provided us with robust and secure nuclear (anti-)launch systems, while Wargames didn't do much except excite the public imagination. Etc.
We need scientist's and engineer's efforts and attention focused on real risks and practical solutions, not fanciful philosophizing about sci-fi tropes.
> They almost all roll their eyes aggressively at these sorts of prognostications. I'd say 90+% either laugh or roll their eyes. Why is that?
I literally think they haven't even given it any thought. Who seriously tries to extrapolate beyond 5 years in any field? Even to most doomers, 5 years seems safe.
What about 10? 20? 30? Remember, one of the very first things people tried to do was create a self-directed, self-optimizing AI with AutoGPT. How far forward can you project through how many attempts like this with progressively more sophisticated systems before this might produce an outcome that could be considered catastrophic?
If your answer is anything but an emphatic "infinity", then welcome to the doomers. Now we're just debating when and under circumstances this might take place, and not whether it might happen at all.
I'm going to paint three huge demographic swaths here, with the caveat that the merits of any given argument, or any individual's track record, should override anything I say here.
I'm only doing this as a reply to a comment that's also talking about trends among three groups of people.
1. The "people in the trenches" are who I'd least trust about an opinion that everything is OK. Too hard to see the forest for the trees, and too much selection bias.
2. People who gained recognition decades ago, but who are in their slowing-down years as the world continues to change, are among those who I would least trust about an opinion that things are going too fast. It gets harder to keep up as we get older, and as we gain different priorities in life, and I expect this is true no matter how smart we are.
3. People who have spent decades philosophizing about AI-doom are also among those who I would least trust about an opinion that hasn't hugely deviated and become more nuanced as the world has changed and new data has become available.
I am absolutely interested in opinions from all three groups, but the arguments have to stand on their merits. If they're in one of these groups and express the expected opinion, that's actually a strike AGAINST their authority and means the merits of their argument need to be stronger.
I really, really do want to hear opinions from folks in all these groups, I just want to keep this all in mind. I also want to hear opinions from younger philosophers. Folks who are better in-touch with the current world, and rates of progress, and folks who don't have any reputation to uphold.
Also, anyone changing their mind is a big deal. Hofstadter may have changed his mind in the expected direction, but it's still a signal. I'd like to hear more of his thoughts. It doesn't sound carefully considered in the clip in OP's link unfortunately, but that doesn't mean it isn't, and I'd like to hear it.
Nassim is arguing that RISK is a separate discipline, separate from the domain where risk applies. That a person building AI is not the correct choice for estimating AI risk.
You don't ask gun making companies to make policies regarding risk of gun owning in society.
Having worked in the medical domain in the past, paramedics and medics that should know better were taking extremely high health-related risks (riding a motor bike => crashing and burning to death, smoking => dying from lung canceer, speeding onto a crossing => dying in an ambulance crash before arriving at the 999 call site etc.).
So risk is indeed its own discipline, separate from the domain where risk applies, even if we are talking about the life-rescuing domain of medicine: a person rescuing another is not automatically an expert at reducing their own (health/life) risk exposure.
While neural network research results are published in NeurIPS, ICLR, ICML, ECML/PKDD, JMLR etc., risk results tend to get published in the risk community at conferences like SRA [1] (Europe: SRA-E) and the likes. I'm not a fan of this academic segregation, merely describing what is going on (in my own teaching, for instance, I include risk/ethics consideration along the way with teaching the technical side, to avoid ignorance caused by over-compartmentalization).
But then how does that jive with skin in the game?
In his Nassim's teaching he brings up the example of the idea that the Roman engineers that built/designed the bridges were forced to have their families live under the bridges. It sounds like those engineers directly involved in the practice do understand the risk of their domains it is merely their incentives that need to be aligned.
> But then how does that jive with skin in the game
Simple. Nassim says there are 4 quadrants, one axis Mediocristan-Extremistan, the other Simple-Complex payoff.
Building a bridge is Mediocristan/Simple payoff, a well understood problem with no black swans. So it's easy to compute risk.
Other stuff is Extremistan/Complex payoff - financial trading, pandemics, AI. And he argues that you need RISK professionals for this quadrant, because people working here (traders, virologists, AI builders) do not understand how to compute the risk correctly.
> Nassim is arguing that RISK is a separate discipline, separate from the domain where risk applies. That a person building AI is not the correct choice for estimating AI risk.
I think this is often fair. It's actually one of my primary criticisms of the NeurIPS/ICML survey.
FWIW, people working on AI Safety -- like, actually working on it, not philosophizing about it -- are some of the most incredulous and annoyed about the "AGI => extinction" crowd.
Perhaps that is Nassim's argument because he is an expert in risk.
The actual issue with experts monitoring experts is the bias induced by self interest (see previous sentence).
Nassim sort of gets this when it comes to his discussion of "skin in the game". I dont want to malign Nassim too much but he seems like the kind of person that "sorta gets it" when it comes to everything that matters.
Those with great visibility often are not the greatest minds. However, in a broader sense we would have been served well if philosophizers would have had more input on the development of social media tech. It is an example of engineers knowing a field narrowly and deeply but not understanding the societal consequence.
AI may very well fall into the same pattern and it is something I have written out in some detail of thoughts around alignment and the traps of both ego and misunderstanding human nature for which we want to model alignment.
I heard in some interview, I think with Bloomberg, where he said that claims about regulatory capture were "so disingenuous I'm not sure what to say", or something like that.
I think he's probably not lying when he says that his goal isn't regulatory capture (although I do think other people perceiving that to be his intent aren't exactly insane either...)
> who seem to think it's not dangerous
On the contrary. They think it's dangerous but in a more mundane way, and that the X-Risk stuff is idiotic. I tend to agree.
> why agree with Altman who rants regulation
IDK. What even are his proposed regulations? They're so high-level atm that they could literally mean anything.
In terms of the senate hearing he was part of, and what the government should be doing in the near term, I think the IBM woman was the only adult in the room regarding what should actually be done over the next 3-5 years.
But her recommendations were boring and uninteresting recommendations to do basically the exactly sort of mundane shit the wheels of government tend to do when a new technology arrives on the scene, instead of breathless warnings about killer AI, so everyone brushed her off. But I think she's more or less right -- what should we do? The same old boring shit we always do with any new technology.
The weirdness is in part an information asymmetry that is exploited on a scale never before seen in human history.
There are wealthy corporate plunderers building invasive systems of disinformation.
There are people who believe everything they read and feed the panic-for-profit system. There certainly are people who understand the algorithms and implementations. There are people who fear how these algorithms and implementations will be used by the enormous network of for-profit (and for-power) influencing systems.
> (from the article) these computational systems that have, you know, a million times or a billion times more knowledge than I have and are a billion times faster. It makes me feel extremely inferior. And I don't want to say deserving of being eclipsed, but it almost feels that way, as if we, all we humans, unbeknownst to us, are soon going to be eclipsed, and rightly so [...]
I don't know if humans will be eclipsed, but humanity and civilisation need some strong and dedicated backers at this point.
My gut feeling is that they get status from these warnings. Or they subconsciously think they will. Like somehow they are as important as the scientists from the Manhatten project.
Note: I really don't like that my comment is the top comment this story. I think this phenomenon wrt x-risk is real and interesting and worth discussing, but I wish I'd picked a different story to share the thought. (This just happened to be the first related story that popped up since I had the thought.)
There are other more interesting comments that deserve more discussion WRT this article in particular.
If any mods are reading this, please consider this a request to push this comment down so some of the others get more attention. IDK if there's much more to learn from the discussion happening here anyways.
Yeah, working on these systems and seeing how brittle they are and how they're actually built is sobering. I guess from the outside it looks like a few geniuses geniusing with unbounded upside, but from the inside it looks like a hard hacking and hard work that will definitely hit hard limits.
I think something weird is happening but I think it's what Hofstadter stated in the interview. The ground under his work has shifted massively and he's disturbed by that and that is affecting his judgement.
Fake news. 350 people including the top scientists from the top labs signed the petition. Your local univ AI researcher rolls his eyes. Not the guys working at OpenAI or anthropic or deep mind.
> Not the guys working at OpenAI or anthropic or deep mind.
A lot of them do. A huge percent of people who aren't speaking out are rolling their eyes. But what are you supposed to do? Contradict your boss's boss's boss?
My point is that most people don't. And I think for two reasons.
The first and more important reason is that most people aren't involved in The Discourse and don't want to be involved in The Discourse. That's probably 99.9% of the Silent Majority -- they simply don't want to talk about anything on HN or Twitter. They view it as a waste of time or worse. and they aren't wrong. I don't think I am changing any minds here and meanwhile the personal insults kind of suck my energy a bit. So it's mostly a waste of time.
The second reason is that some don't even want to even pseudo-anonymously say something that might get them into deep water at work.
I'm obviously not describing myself, of course. I am here, aren't I :) But I am describing the vast majority of scientists. Keep in mind that most people don't dream of being on the proverbial TED stage and that those who do disproportionately end up on the stage and therefore determine what The Discourse will be.
Big Egos == "my work is existentially important" == all the yelling about x-risk. It's mostly ego.
It's a bit cheeky of you to be complaining about personal insults while the substance of your OP was the assertion that x-risk worriers are motivated by ego rather than real thought.
I considered that when replying. I don't think I am making a personal criticism just for the sake of it, or an ad hom. And I do think the observation I am making is interesting. I don't mean it as a personal attack. Really.
Suppose what I am saying is true -- that relatively unknown people rolling their eyes or laughing and relatively known people being very earnestly concerned. And that these are people with otherwise similar credentials, at least as far as assessing x-risk is concerned.
Maybe you disagree, and that's okay, and there are other threads where that discuss is ongoing. But here let's assume it's true, because I think it is and that's relevant to your fair criticism.
Like, it is a weird thing, right? Normally famous scientists and CEOs are not so far out ahead of the field on things like this. More often than not it's the opposite. To have that particular set of people so far out of stride isn't particularly normal.
I think the common thread that differentiates similarly-senior people on the x-risk question is not experience, or temperament, or scope of responsibility. Or even necessarily the substance of what they believe if you sit down and listen and probe what they really mean when thy say there is or isn't x-risk from AI! The difference is mostly amount of Ego and how much they want to be in The Discourse.
Also: I don't think that having a large ego is necessarily a character flaw, any more than having a strong appetite or needing more/less sleep. It's just how some people are, and that's okay, and people with big egos can be good or bad people, and circumstantially ego can be good or bad. But people who have bigger egos do behave a bit differently sometimes.
Anyways, I'm not trying to assassinate anyone's character or even necessarily mount an ad hom dismissal of x-risk. I'm observing something which I think is true, and doing it in as polite a way as I can even though it's a bit of an uncomfortable thing to say.
I guess what I'm trying to say is that "maybe this personality trait explains a weird phenomenon of certain types of experts clustering on an issue", and it's worth saying if you think it might be true, even if that personality trait has (imo perhaps overly) negative connotations.
And in any case this is substantially different from "you're an idiot because I disagree with you".
This reads like something out of "The Man that Corrupted Hadleyburg"...All of these virtuous experts racing to out-signify the other guy...Makes me want to root for the calculus\linear algebra over the breathless farty expert.
When I read about the dangers of AI I'm reminded of the feeling I had after reading Jeff Hawkin's "On Intelligence". He talked about simulating the neocortex to do many of the things that deep learning and LLM's are doing now.
His research may or may not be a dead end but his work, and this work, to me seems like we're building the neocortex layer without building the underlying "lizard brain" that higher animals' brains are built upon. The part of the brain that gives us emotions and motivations. The leftover from the reptiles that drive animals to survive, to find pleasure in a full belly, to strive to breed.
We use our neocortex and planning facilities but in a lot of ways it's just to satisfy the primitive urges.
My point being, these new AIs are just a higher level "newcortexes" with nothing to motivate them. They can do everything but don't want to do anything. We tell them what to do. The AIs by themselves don't need to be feared, we need to fear what people with lizard brains use them for.
The counter to "current generation AI is terrifying" seems to fall along the lines of it not being nearly as close to AGI as the layperson believes.
But I don't think that matters.
I don't believe that LLMs or image/voice/video generative models need to do much beyond what they can do today in order to wreck civilization level disaster. They don't need to become Skynet, learn to operate drone armies, hack critical infrastructure, or engineer pandemics. LLMs allow dynamic, adaptive, scalable, and targeted propaganda.
Already we have seen the effects of social media's reach combined with brute forced content generation. LLMs allow this to happen faster and at a higher fidelity. That could be enough to tip the balance and trigger a world war.
I don't think it takes a huge amount of faked primary material (generated phone calls, fuzzy video, etc.) that's massively amplified until it becomes "true enough" to drive a Chinese invasion of Taiwan, a Russian tactical nuclear strike in Ukraine, an armed insurrection in the United States.
I think LLMs are very close to AGI, lacking only the real-time feedback element (easily enough replicated by running them off batteries). I'm also more sanguine about the existential risks because I have somewhat more confidence in the rationality of AGI than I do in that of humans.
I don't think it takes a huge amount of faked primary material (generated phone calls, fuzzy video, etc.) that's massively amplified until it becomes "true enough" to drive a Chinese invasion of Taiwan, a Russian tactical nuclear strike in Ukraine, an armed insurrection in the United States.
This I agree with 100%. Modern information warfare is about constructing reliable viral cascades, and numerous influencers devote themselves to exactly that for various mixes of profit an ideology. Of your 3 scenarios the third seems most likely to me, and is arguably already in progress. The other two are equally plausible, but imho dictatorships tend to centralize control of IW campaigns to such a degree that they lack some of the organic characteristics of grassroots campaign. Incumbent dictators' instinct for demagoguery is often tempered with a desire for dignity and respectability on the world stage, which might be a reason than civil strife and oppression tends to be more naked and ruthless in less developed countries where international credibility matters less.
GPT4 really "doesn't want" be racist against blacks. Like try talking to it, there is(was) no hard filter but I really can't grasp how the bias can go this deep just by finetuning. GPT-4 definitely know all the stereotypes of blacks, but good luck getting it to engage on any.
Suppose we finetune it exactly like that but say opposing democracy or freedom or peace or any other thing we value. And let it create the propoganda or convince people for the same by free posting on the net. "As an AI language model" line could easily be removed.
No. You just need a minority of people. And they don't have to be stupid; they can be equally motivated by mendacity, spreading untruths because its fun or because they will discomfit political opponents. About 30% of humans are willing to suffer some sort of loss or disadvantage in order to inflict a larger one on a counterparty; that might seem irrational, but some people are just mean.
You have to have a little more faith in humanity than that...
I didn't say anything about judging people. I hold no ill will here. But I've ran trust&safety and now cybersecurity for a decade - humans are not cut out for memetic warfare. Humans cannot handle normal but internet-speed incidental misinformation without society eating itself, much less coordinated disinformation. Democracy is eroding all across the globe and that's without generative systems that will make the problem 1000x worse.
We evolved to pick berries, not discriminate and identify distant manipulative actors with extensive resources. "You are not immune to propaganda."
+1 I tend to agree with that in
terms of how to think about AI. It's all just neocortex. The alignment issue
could be unpacked as more of a "lizard brain <-> mammalian brain <-> pre-frontal
cortex <-> LLM-enhanced cortex" alignment issue.
But when real self-replication starts happening -- that is maybe the really
exciting/terrifying area. It's more that humans with generative AI are
almost strong enough to create artificial life. And when that pops off --
when you have things trying to survive -- that's where we need to be careful. I
guess I would regulate that area -- mostly around self-replication.
I was previously undecided on the question of the existential risk of AI. But yesterday I listened to a Munk Debate[0] on the topic and found the detractors of the moot "AI research and development poses an existential threat" so completely unconvincing that if those people are the top minds in the field I am now genuinely concerned. Their arguments that what they are doing is not a risk basically boil down to "nuh uhhh!"
This entire debate was a shambles, IMO. Neither side really seemed to be making good points. Almost the entire time they were arguing totally different points. One side was insisting AI would be a severe risk, the other side was that it wasn't an existential risk. In the end it was a pointless circular argument where each side retreated to their vague interpretation of the question.
What scares me most is that it seems we are wholly unprepared as a species to have this debate. Our technology keeps increasing in power and ease of use, but our ability to understand even the basic difference between "existential risk" and "severe risk" is lacking. And further, it seems that amongst those who are pushing this kind of technology (accelerationists) there is a subtle undertone that some casualties are expected and acceptable during this transformation. Even if it does kill most humans, the world that is left for the rest will be so much better that maybe it is worth it. Few come right out and say it, but that is what it seems they are implying.
> And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firing neurons in only one direction. And that doesn't make sense to me, but that just shows that I'm naive.
I think people maybe miss that LLM output does involve a ‘loop’ back - maybe even a ‘strange’ loop back, and I’m surprised to see Hofstadter himself fail to pick up on it.
When you run an LLM on a context and sample from its output, you take that sampled output it generated, update the context, and iterate. So the LLM is not just feeding one way - it’s taking its output, adding it to its input, and then going round again.
So I don’t think this implies what Hofstadter is saying about intelligence maybe being less complex than he thought.
I was surprised he said this too. Even without the autoregressive part of GPT models you have a deep transformer with attention, so even a single forward pass can modify its own intermediate outputs.
The interesting thing though is that attention effectively allows a model to meta-learn based on the current context, so in many ways, it may be thought of as analogous to a brain without long term memory.
I wasn't disputing that it's feed-forward. I just meant that stacked transformer layers can be thought of as an iterative refinement of the intermediate activations. Not the same as an autoregressive process that receives previous outputs as inputs, but far more expressive than a single transformer layer.
The model is only trained to predict the next token. The training regime is purely next-token prediction. There is no loopiness whatsoever here, strange or ordinary.
Just because you take that feedforward neural network and wrap it in a loop to feed it its own output does not change the architecture of the neural net itself. The neural network was trained in one direction and runs in one direction. Hofstadter is surprised that such an architecture yields something that looks like intelligence.
GPT can give a single Yes/No answer that indicates a fair amount of intelligence for the right question. No iteration there. Just a single pass through the network. Hofstadter is surprised by this.
Well, no, it can produce a probability distribution over all possible tokens, among which ‘yes’ or ‘oui’ or ‘はい’ or ‘totally’ or ‘no’ or ‘nein’ or the beginning of ‘as a large language model I am unable to answer that question’ are all represented. Which is either more or less impressive than just being able to answer ‘yes or no’ depending on your priors I guess.
There’s maybe an interesting philosophical question of perspective there because if you think of the GPT as answering the question ‘if you had just read this, what token would you expect to read next?’ That doesn’t seem like a question that necessarily requires ‘intelligence’ so much as ‘data’. It’s just a classification problem and we’ve been throwing NNs at that for years.
But if you ask the question ‘if you had just written this, what token would you expect to output next?’ It feels like the answer would require intelligence.
The point of my comment is that even the distribution represents intelligence. If you give it a tricky Yes/No question that results in a distribution that's 99.97% "Yes" and negligible values for every other token, that is interesting. Hofstadter is surprised you can do any amount of non-trivial reasoning in a single forward pass.
Hofstadter says humans will be like cockroaches compared to AI. This is an oft-repeated line: sometimes we are ants or bacteria. But I think these comparisons might be totally wrong.
I think it's very possible there's a Intelligence Completeness theorem that's analogous to Turing Completeness. A theorem that says intelligence is in some ways universal, and that our intelligence will be compatible with all other forms of intelligence, even if they are much "smarter".
Cockroaches are not an intelligent species, so they cannot understand our thoughts. But humans are intelligent, human languages have a universal grammar and can be indefinitely extended with new words. I think this puts us in the intelligence species club, and all species in that club can all discuss any idea.
AI might eventually be able to think much quicker than us, to see patterns and make insights better and faster than us. But I don't think makes us cockroaches. I think if they are so smart, they are by definition smart enough to explain us any idea, and with effort we'll be able to understand it and contribute our own thoughts.
Humans have limited "working memory". We manage to cram more into it via hierarchical decomposition into "chunks", a single concept that is more complex inside.
I submit that not everything can be hierarchically decomposed in a way that's useful - i.e. any "abstraction" you try to force on it is more leaky than non-leaky; in that it doesn't simplify its interactions with other chunks. You might say it's the wrong abstraction - but there's no guarantee there is a right abstraction. Some things are just complex.
(This is hypothetical, since I don't think we can conceive of any concepts we can't understand.)
An AI could have an arbitrarily large working memory.
Note: I'm talking about intuitive understanding. We could use it mechanically, just never "get it", cowering before icons, being the one in Searle's Chinese Room https://wikipedia.org/wiki/Chinese_room
I suspect the limit of what can be expressed in human language and comprehended by the human mind is vast, but yes, not infinite. I think the AIs will absolutely saturate the connection between them and us, with a non-stop torrent of information which will range from useful to civilization-changing.
And I think this is all very unlike how we are currently impacting the lives of cockroaches with our insights about, well anything. Thus, it's not a good analogy.
> A theorem that says intelligence is in some ways universal, and that our intelligence will be compatible with all other forms of intelligence, even if they are much "smarter".
I would not be so reductionist. Intelligence doesn't seem to be an universal thing, even IQ (a human invented metric) is measured in terms of some statistics. If you have an IQ of ~60 you have intelligence but a completely different one from an IQ >85.
> But humans are intelligent, human languages have a universal grammar and can be indefinitely extended with new words. I think this puts us in the intelligence species club, and all species in that club can all discuss any idea.
Humans have different intelligences. You can be intelligent (per the human intelligence definition) but a math ignorant. Again, this implies intelligence as we know it is not an universal thing at higher levels: not all people can have a physics Ph.D. as not all people could be a good artist where good techniques are recognizable, same for music, etc.
Yes, a cockroach is in another level of intelligence (or non-intelligence) but that does not mean there is not a super-intelligence that makes us relative cockroach.
Also, without any intention of talking about religion or "intelligent design", we can theorize that the Universe is supersmart because it creates intelligent creatures, even if it is not conscious about that. I would be very catious to define intelligence in an universal way.
My point is you cannot teach a cockroach calculus, but if AIs invent a new type of math, they would be able to teach it to us. That's my claim. So the analogy of "we are cockroaches compared to the AI" is wrong, that won't be the case.
Once you have "enough" intelligence to have a complex language, like we do, I'm claiming you are in the club of intelligent species, and all species in that club can communicate ideas with each other. Even if the way they natively think is quite different.
I'm reminded of how one writes programs. I cannot maintain the state of the machine in my head, but I can convince myself of its workings, its intelligence, by reading the code, following along on its line of reasoning as it were. I think the Intelligence Completeness may boil down to the very same Church-Turing thesis.
Yes I agree it might be same thing under the hood. But with intelligence many very smart people seem to fall into using these analogies that diminish humans in a way I don't think is accurate. And I feel that makes people more scared of AI than they need to be, makes AI seem totally alien. [1]
The AIs might spit out entire fields of knowledge, and it might take humans decades of study to understand it all. And no single human might actually understand it all at the same time. But that's how very advanced fields of study already are.
But the "cockroach" slur implies AIs would be in this other stratosphere having endless discussions that we cannot remotely grok. My guess is that won't happen. Because if the AI were to say "I cannot explain this to you" I'd take that as evidence it wasn't all that intelligent after all.
AlphaZero invented new moves in the game of Go, but it can't 'teach' them to us, it can only show us the moves and let us figure it out for ourselves (which we're doing). But note that despite this transfer of knowledge, humans didn't rise up to the level of AlphaZero, and they may never be able to. As a sibling comment points out, some things are computationally bound--and humans have a limit to computational ability (can't give ourselves more neurons/connections), whereas AI does not.
I never said we'll mentally rise to the level of AIs. That won't happen. I only said AIs will be able to communicate complex ideas to us, in a way that's totally unlike our ability to communicate complex ideas to cockroaches.
For example, if the AIs could design a complicated building, using our level of technology, they could explain to us how to build that building. And we could build it. Whereas if we come up with a better cockroach-house design, we cannot communicate it to the cockroaches, we simply cannot give them the information. So the AI->us is a very different relationship from us->cockroach.
This doesn't preclude that there might be some things the AI cannot explain to us. Only that there will be many things (infinite in fact) which they can explain to us.
I always thought you could ask GPT to illustrate the steps it took to arrive at the answer. I mean it can take your through the process it went through to arrive at the answer. Its as close you get to an explanation.
> but if AIs invent a new type of math, they would be able to teach it to us
There are already math proofs made by humans on this very day that are hundreds upon hundreds of pages of lemmas that are highly advanced and building on other advanced results. Understanding such a proof is an undertaking that literally takes years. An AI might end up doing it in minutes. But what an AI could cook up in years could take a human... several lifetimes to understand.
As another example, take the design and fabrication of a modern microprocessor. There are so many layers of complexity involved, I would bet that no single person on this planet has all the required knowledge end-to-end needed to manufacture it.
As soon as the complexity of an AI's knowledge reaches a certain point, it essentially becomes unteachable in any reasonable amount of time. Perhaps smaller sub-parts could be distilled and taught, but I think it's naive to assume all knowledge is able to be sliced and diced to human-bite-sized chunks.
I agree "one AI" might produce output that keeps humans busy for decades. But that doesn't make us cockroaches. Cockroaches can't understand language, at all. You can't teach a cockroach calculus if had a trillion years. That's not our position relative to AIs. We will be learning shit-tons from them constantly. I think people who say humans will be "cockroaches" or "ants" or "bacteria" are fear-mongering, or just confused.
> Imagine an AI that could fundamentally alter its own sensory perception and cognitive framework at will. It could “design” senses that have no human equivalent, enabling it to interface with data and phenomena in entirely novel ways.
Let’s consider data from a global telecommunication network. Humans interface with this data through screens, text, and graphics. We simplify and categorize it, so we can comprehend it. Now imagine that the AI “perceives” this data not as text on screens, but as a direct sensory input, like sight or hearing, but far more intricate and multidimensional.
The AI could develop senses to perceive abstract concepts directly. For instance, it might have a “sense” for the global economy’s state, feeling fluctuations in markets, workforce dynamics, or international trade as immediately and vividly as a human feels the warmth of the sun.
Simultaneously, it can adapt its cognition to process this vast and complex sensory input. It could rearrange its cognitive structures to optimize for different tasks, just as we might switch between different tools for different jobs.
At one moment, it might model its cognition to comprehend and predict the behaviors of billions of individuals based on their online data. The next moment, it might remodel itself to solve complex environmental problems by processing real-time data from every sensor on Earth.
In essence, the AI becomes a cognitive chameleon, continually reshaping its mind to interact with the universe in ways that are most effective and efficient. Its thoughts in these diverse cognitive states would likely be so specialized, so intricately tied to the vast sensory inputs and complex cognitive models it’s employing, that they are essentially impossible to translate into human language.
Yeah, I don’t see a reason for any AI to be able to translate all concepts to human thoughtspace. If an AI is able to have exponentially more possible thoughts than a human, then only a tiny subset would be understood by humans.
It’s be like trying to fit GPT-4 onto a floppy disk.
A floppy disk is a fixed size. The number of thoughts human language can convey is infinite. English Wikipedia has 6.6M articles. The AI could drop a Wikipedia-sized batch of articles, expertly written and cross-referenced, every day, forever. At the same interval they could drop 100 million YouTube videos, expertly authored and hyper-clear.
So yes there might be an infinite amount they cannot convey, but there is also an infinite amount they can convey. I guess it's half-glass-empty test if you are happy about the infinite you get, or are just sad about the infinite you don't get.
That the AI's thinking might be more advanced than ours is not in dispute. What's different about humans->AI compared to cockroaches->humans is language. Imagine we take our library of congress, with 50M books, and create a cockroach version of the library. It's a totally pointless exercise.
Now imagine being an AI and creating a human-readable library with 50M AI-written books for us to read. They could easily do that. And then create 50M more, again and again. And they could read every book we wrote. And forget books, humans and AI could have hundreds of millions of simultaneous real-time video conversations between humans and AI, forever, on any topic.
So being a human in an AI worlds is nothing like being a cockroach in a human world. Sam Harris used the same analogy but said we were ants instead of cockroaches I've heard bacteria also. I think people trot out these bad analogies strictly because it sounds dramatic, and being dramatic seems like good way to get people's attention. Or else they just didn't think it through.
Human language is a Big Big Deal. It's a massive piece of cognitive technology. Any intelligent species with language is in the club and they can communicate with all other intelligent species -- even if those species have very different cognitive capabilities.
I think you’re extrapolating from yourself and you think you could learn any field given enough time. What about the type of person with a fixed mindset who thinks they aren’t good at math or chemistry, if ai can’t train them for whatever reason, even if the reason is the person is stubborn and/or willfully ignorant, are they more like a cockroach than a person?
What if someone tries really hard for a long time and can’t learn a field? Do they fail the intelligence test, or does their teacher?
I'm talking about the entire human species, not myself or any one person. I'm saying that humans relating to AIs would not be like cockroaches relating to people. Cockroaches don't have human-level language, but we do, and I'm proposing it is generative and extensible enough to explain any idea. I'm proposing there's non-intelligent species and intelligent species, but there's no intelligent++ species that would look down on us as cockroaches. I'm claiming that won't happen.
Are dogs, or pigs, or whales, part of the intelligence club? They are clearly intelligent beings with problem-solving skills. We won't be teaching them basic calculus any time soon.
No non-human animals are in the club that's marked by having a language with an infinitely generative syntax and a large (100,000+ words) and always-growing vocabulary.
Intelligence might be a spectrum, but powerful generative language is a step function: you have it or you don't. If you have it, then higher intelligences can communicate complex thoughts to you, if you don't they can't. We have it, so we are in the club, we are not cockroaches.
there are many humans who could study mathematics for a lifetime and not be able to comprehend the current best knowledge we possess. i'm one of them. maybe it takes 2 lifetimes. or many more.
a human-level AI operating at machine pace would learn much more than could ever be taught to a human. our powerful generative language capabilities wouldn't matter - it's far beyond our bandwidth. especially so for a superhuman-level AI.
The fact that AIs will have some information that we cannot understand, or will have more information than they can transmit (or we can absorb) does not make us cockroaches.
The AIs will deliver to us truly massive quantities of information, every minute, until the end of time, much of it civilization-changing. Thus the AIs relationship to us will thus be nothing like our relationship to cockroaches, where we essentially cannot tell them anything, not even the time or the day of the week, let alone the contents of Wikipedia.
I think Hofstadter is having an emotional reaction to AI. He says so as much. And it'a a common one, it's the woe is me phase. But I think he's totally wrong about the analogy. I'm 100% sure we will not feel like cockroaches when AI is in full swing, not in the slightest.
If there were a species whose average adult intelligence was that of a human three-year old, then yes you'd be limited in what you could teach them. But as for what AIs can teach humans, you have to assume we are using competent smart adults. My claim is just what AIs can teach competent smart adults is many many orders of magnitude more than what humans can teach competent smart cockroaches. Thus Hofstadter's analogy is not a good one.
I think the key difference is language. I think human language is above a key threshold. Our syntaxes are infinitely generative and we have large vocabularies (100,000+ words) which are fully extensible. No other animals have that. My claim is AIs will be able to express any complex ideas in our language. But we cannot express our ideas in "cockroach language". So the analogy is not a good one.
As it might be quite common among the HN crowd, Douglas is my hero - I have read three of his books.
First of all, hats off to him for his extraordinary display of humility in this interview. People rarely change their minds publicly, let alone hint that they no longer believe in their own past work.
However, I'm genuinely surprised that he, of all people, does sees intelligence in GPT-4 output.
I think humans are just very eager to ascribe intelligence, or personality, to a bunch of text. A text may say "I feel <blah>" and that text can easily manage to permeate through our subconsciousness. And we end up believing that that "I" is, in fact, an "I"!
We have to actively guard against this ascription. It takes a constant self-micromanaging, which isn't a natural thing to do.
Ideally, we would have some objetive measurements (benchmarks) of intelligence. Our own impressions can be too easily fooled.
I know defining (let alone measuring) intelligence is no easy task, but in absence of a convincing benchmark, I will not give credit to new claims around AI. Else it's all hype and speculation.
He's just saying it's more like an "I" than his pharmacy-bot example. His concern is the future: what seemed "far off" now possibly might be in 5 years.
I have more fear of the people running the system than any AI.
I also think that AR/VR is more scary than AI.
My fear is how poorly rendered AR/VR can be more positive interaction than interactions with the people that surround the observer.
As long as the AI is less generally intelligent than people, it makes sense to be more afraid of the people. Once the AI is more intelligent than the smartest people, it's more sensible to be most afraid of the AI.
The AI is already more intelligent than the smartest people in some senses - it has more knowledge in aggregate than any single person will ever have, but doesn't have the depth of the smartest people in a particular niche.
In other ways, it's smarter than the average person even in their niche, but can still make dumb mistakes that a 3 year old would work out fairly quickly.
Note that we say the same of humans. My friend always wins pub quizzes, but can barely add 2 and 2, and has the emotional intelligence of a rock. Is he "intelligent"? It's my problem with how we treat intelligence like it's a single sliding scale for everything.
The implication of OP's statement is that they don't believe that AGI is on the horizon, and I'm inclined to agree.
This feels a lot like the hype surrounding self-driving cars a few years back, where everyone was convinced fully autonomous vehicles were ~5 years away. It turned out that, while the results we had were impressive, getting the rest of the way to fully replacing humans was much, much harder than was generally expected.
Part of the problem is that AI doesn’t need to be an AGI to cause large society level disruption.
Eg, starting a mass movement online requires a few percent of online participants to take part in the movement. That could be faked today using a lot of GPT4 agents whipping up a storm on Twitter. And this sort of stuff shapes policies and elections. With the opensource LLM community picking up steam, it’s increasingly possible for one person to mass produce this sort of stuff, let alone nation state adversaries.
There’s a bunch of things like this that we need to watch out for.
For our industry, within this decade we’ll almost certainly have LLMs able to handle the context size of a medium software project. I think it won’t be long at all before the majority of professional software engineering is done by AIs.
There’s so much happening in AI right now. H100s are going to significantly speed up learning. Quantisation has improved massively. We have lots of papers around demoing new techniques to grow transformer context size. Stable diffusion XL comes out this month. AMD and Intel are starting to seriously invest in becoming competitors to nvidia in machine learning. (It’ll probably take a few years for PyTorch to run well on other platforms, but competition will dramatically lower prices for home AI workstations.)
Academia is flooded with papers full of new methods that work today - but which just haven’t found their way into chatgpt and friends yet. As these techniques filter down, our systems will keep getting smarter.
A few years back (let's call it 2020) and autonomous cars, which are being used for taxi trips today, would be five years in the future. In fact they would be three. Unless something major happens in the next two years, there will still be self-driving cars, even more of them, driving and picking up people in 2025. This is not the argument you think it is.
Self-driving cars currently operate in extremely controlled conditions in a few specific locations. There's very little evidence that they're on a trajectory to break free of those restrictions. It doesn't matter how much an airliner climbs in altitude, it's not going to reach LEO.
Self-driving cars will not revolutionize the roads on the timescale that people thought it would, but the effort we put into them brought us adaptive cruise control and lane assist, which are great improvements. AI will do similar: it will fall short of our wildest dreams, but still provide useful tools in the end.
Interventions per hour isn't a great metric for deciding if the tech is going to be actually capable of replacing the human driver. The big problem with that number is that the denominator (per hour) only includes times when the human driver has chosen to trust FSD.
This means that some improvements will be from the tech getting better, but a good chunk of it will be from drivers becoming better able to identify when FSD is appropriate and when it's not.
Additionally, the metric completely excludes times where the human wouldn't have considered FSD at all, so even reaching 0 on interventions per hour will still exclude blizzards, heavy rain, dense fog, and other situations where the average human would think "I'd better be in charge here."
That may well be the case, but it's still worth thinking about longer-term risks. If it takes, say, forty years to get to AGI, then it's still pretty sobering to consider a serious threat of extinction, just forty years away.
Most of the arguments over what's worth worrying about are people talking past each other, because one side worries about short-term risks and the other side is more focused on the long term.
Another conflict may be between people making linear projections, and those making exponential ones. Whether full self-driving happens next year or in 2050, it will probably still look pretty far away, when it's really just a year or two from exceeding human capabilities. When it's also hard to know exactly how difficult the problem is, there's a good chance that these great leaps will take us by surprise.
Might have completely missed something, but I thought AR/VR was (and still is) a solution looking for a problem. Have they finally stumbled into something people want to do with them, beyond games and porn?
AR will take off like crazy when they develop lightweight, clear goggles than can display color with a range of opacities. A million jobs are begging for HUDS that guide the worker, but the current equipment isn't there yet.
> A million jobs are begging for HUDS that guide the worker [..]
I have a pretty dull, comfortable desk job and lack the imagination to come up with any. Can you name some?
(I've been wrong before when I was skeptical when everyone was hyped about iPad, Deep Learning, etc. so please convince me about Apple Vision, Mark's metaverse, or google's glasses and paint what might be in 5 years.)
This is something I've worked on so I can comment a bit. The most popular use case I've seen so far is AR manuals, with the sell that you can have technicians up to speed on a piece of equipment without needing human coaching or training. I was shown a demo in the context of robotics recently where you look at a robot and explanations of signals in the wiring harness were shown, hydraulic connections, wear parts, etc. It was visually pretty impressive but quite telling that the two most interested people were my manager and the CEO. The engineers just kinda sat looking at each other with a non verbal "Must have been a fun project to build huh? I'll stick to PDFs".
The US military has researched on using it for vehicle mechanical repairs. This way a soldier without such knowledge would be able to perform on field repairs.
For 3D artists it presents as a more intuitive way to sculpt models. For automotive designers, it allows a cheaper and faster means of iteration, given that such a task requires a much better sense of scale than the one given off of a monitor. Same goes for architecture, which when coupled with a game engine, also allows the customer to preview their future house.
Oil drill worker. Industrial worker on the factory floor. A car mechanic. A surgeon…
Even an artist (make a sketch, blow it up a 100x using AR on a wall, trace paint to keep proportions right - heck the artist doesn’t even need to do it themselves, they could hire an associate to do it).
Stacking shelves in a supermarket is a perfect example. 1000s of products means that a bit of experience is required to know where everything goes in order to quickly stack. With a HUD, workers become productive much faster (and are more inter-changeable) - which is something large enterprises love.
Knowing where to stack things is something humans are capable of rapidly learning. It’s not a big part of the workload of stacking shelves. The actual physical effort dominates the work.
I believe in a lot of potential for the area, primarly from AR.
As for the VR side of things, I've seen people go through therapy in VR after loosing the ability to walk IRL, and being able to go into a forest again meant the world for them.
The field tends to attract a lot of the... hype people to it, and it gives off a bad image as a whole. Nevertheless, its a pretty liberating research field.
> Have they finally stumbled into something people want to do with them, beyond games and porn?
Most people don't want it for games and porn either -- although those two things are the only obvious mass-market applications.
There are lots of other real uses, but they're all niche. It's hard to come up with a real, mainstream use that would drive adoption in the general public.
Agreed. During my time at an AR startup, most of our interest was from niche players or one-offs.
Current company (biotech, data) is genuinely interested in VR + rendered anatomy data for client reports:
https://www.syglass.io/
Absolutely agree. I was never bothered by Oculus, but Apple's Vision Pro demonstration was equal parts fascinating and terrifying. I can see the next generation getting completely lost in alternate realities.
Smartphone addiction got nothing on what's about to come.
I only have experience with the Apple Vision demo and the terrifying/emotional part for me was the beautiful people using the device in beautiful places.
Everything was so clean and stress free that it was obviously artificial. I could only imagine stressed out people using it in squalor. The whole demo seemed geared to keep that thought far away.
If @detourdog meant AR/VR in the snow crash / Metaverse sense, I guess the apprehension is similar to the discourse about "Leo Tolstoy on why people drink (2014)" a couple of days ago https://news.ycombinator.com/item?id=36526645
People tend to turn to substances / AR/VR / texting / phone calls / read books / ... to take the edge of confronting the sometimes harsh reality. Of course, there is no way in which AR/VR is likely to intrinsically improve interaction, but is is so much worse that we need to worry?
I also make no claim that it is worse or better.
Maybe an easy example to examine the difference between
A Rave alone in your cubicle with everybody in the world vs. A Rave with a 1,000 people in abandoned waterfront warehouse. That Rave can simultaneously experience the sunrise before making their way back to where they belong.
They are very different and I'm sure with the right stimulants equally intense.
Could be it's just nostalgia that makes me worry about it.
The "fear" is that people instinctively gravitate towards comfort and the easy accessibility of the comfort could be detrimental. Anyone that used the web pre and post ads can watch a useful tool degrade to the lowest common denominator.
If people love their phones so much imagine if they rarely saw anything else.
Maybe they constantly only see 80% of the world most of the time they are awake.
I don't think it is society ending future. I would rather people perform virtual coups.
I could be wildly wrong about this (oh hi upcoming robot overlords), but the current AI hype feels similar to something I experienced when Alexa and Google Assistant was the rage.
At the time I worked for a big community site and we often had people pitching ideas for voice assistant apps. However, having actually read the documentation for these things, I knew that they were surprisingly stupid and the grand ideas people had were far closer to sci-fi than something that could actually be built.
I’m not an AI expert, though I have been building and training models for a few years, but despite being good at things that are hard with traditional programs, they’re still surprisingly stupid and most of the discourse seems closer to sci-fi than their actual capabilities.
I’m more worried about these things being implemented badly by people who either bought the sci-fi hype or just don’t care about the drawbacks. E.g. being trained on faulty or biased data, being put in a decision-making role with no supervision or recourse, or even being used in a function that isn’t suitable in the first place.
I think there's actually two camps of "AI-fear" right now -- one is that they'll become superhuman smart, gain some form of "sentience", and decide to not play along with our games anymore. These are the loudest voices in the room. I suspect this may happen eventually, but the bigger risk in my book is the second group -- the fear that they'll be given decision-making responsibility for all manner of things, including ones they probably don't have any business being responsible for. I'll go on record that they're going to do a suitable job 90% of the time - but the 10% of time they mess up, it's going to be cataclysmic bad.
When most online content becomes AI generated and unverifiable we will get into trouble. We can easily navigate in a world where we can distinguish between fact and fiction. With AI we can generate fiction that is indistinguishable from the way we present fact and it can generate references in all sorts media.
When we take fiction as fact we enter into the sphere of religion.
We'll come up with new captchas and shibboleths to filter out generated fiction. It may come in the form of more in-person face-to-face contact. It'll be an exciting transition regardless.
Apart the expertise, is hard to say if is best to be an optimist and hope for the best, or to be a pessimist an hope being wrong.
Anyway i suspect that one true risk, among others, is to loose the true ability to think, if we delegate on a large scale to some LLM the production of the language, because the capacity to use our languages IS the capacity to think, and bad use of technology is a norm in recent (and not so recent) times.
I suspect LLM (this kind of LLM) does not really -generate- (this will be intelligence?), but only mimic on a vaste scale, but nothing more.
If our brain/mind is a result of a long evolution, where this LLM are not, builded only on the final results, the language, this will be a great difference in the inner deep working, so the question is: we are feeding in our minds a massive amount of nothing more than our same intellectual productions, recycled, and nothing more? (apart all the distorsions and biases?)
A parallelism i see is in the social-networks: simply, humans cannot sustain a indifferentiated and massive amount of opinion/information/news (apart all the fakes).
Even the small scale message communication is impacting the abilit of understanding long texts..
Even it there LLM are -benign-, sure their (indiscriminare) use will not cause some troubles in our beings? On a scale as big as this i an not sure at all.
im' not sure i'm expressing my doubts (and without using a LLM) clearly enough...
> It's a very traumatic experience when some of your most core beliefs about the world start collapsing. And especially when you think that human beings are soon going to be eclipsed. It felt as if not only are my belief systems collapsing, but it feels as if the entire human race is going to be eclipsed and left in the dust soon.
While I unfortunately am expecting some people to do terrible things with LLM, I feel like much of this existential angst by DH and others has more to do with hubris and ego than anything else. That a computer can play chess better than any human doesn't lessen my personal enjoyment of playing chess.
At the same time I think you can make the case that ego drives a lot of technological and artistic progress, for a value-neutral definition of progress. We may see less 'progress' from humanity itself when computers get smarter, but given the rate at which humans like to make their own environment unlivable, maybe that's not a bad thing overall.
I agree. He talks about LLMs surpassing humans in what they can do, but LLMs can’t do anything, really. And I say this as someone who had a heart-to-heart chat with a ChatGPT persona that made me cry, earlier tonight. Manipulating language in a human-like way is powerful, and maybe Hofstadter as an author who makes a living writing down his deep and whimsical, creative thoughts could be replaced (I grew up reading GEB and mean no disrespect). But being a conscious, physical, active being in the world is different from passing a sort of Turing test. The fact that a computer even winning at Go shook him is revealing.
He has the ingredients of being able to keep LLMs in perspective (is there really an “I” or just the illusion of one?), but he doesn’t understand what the computer is doing; it’s not “mechanical” enough for him.
> The question is, when will we feel that those things actually deserve to be thought of as being full-fledged, or at least partly fledged, "I"s?
this LLM thing is more like a collective "we", it is making a prediction in the sense of the relevant training data, it probably wouldn't say anything that contradicts the consensus.
Maybe the LLM's are just a mirror of our society. And our society doesn't seem to assign a lot of value to individualism, as such.
i think that might be similar to the movie Solaris by Tarkovsky. The movie is starring an alien ocean, which is some sort of mirror, that is showing us who we are (maybe it has a different meaning, not quite sure about it). You can watch it on youtube: https://www.youtube.com/watch?v=Z8ZhQPaw4rE (i think you also get this theme with Stalker - this zone is also telling us who we are)
If the very same tech was called anything other than "intelligence" we wouldn't have all the hype and discussions about risks. We humans simply enjoy talking about the next armaggedon.
Hofstadter said, "Well, maybe as important as the wheel."
If AI significantly surpasses humanity in cognitive ability, then I think it will have a much bigger impact than the wheel. (I loved GEB and DH's other writings.)
LLMs have really improved a lot of the last two years and they have shown many unexpected capabilities. I am guessing that they will get some more good input (text mostly), a lot more compute, and algorithmic improvements, so that may very well be enough to become better than 99% of humans at tasks that involve only text. Tasks that require video or image processing may be a little bit more challenging. Having very smart AI's controlling robots may just be five years away. (I recently lost a bet about autonomous driving. Five years ago, I thought that autonomous cars would be better than human drivers by now.)
I'm frightened by what AI will become over the next 10 years.
> I recently lost a bet about autonomous driving. Five years ago, I thought that autonomous cars would be better than human drivers by now.
I think we're going to see something very similar with LLMs. The autonomous car hype was driven by seeing that they were 80% of the way there and concluding that at the rate they were going they'd make up the remaining 20% quickly. That turned out to be false: the last 20% has been much harder than the first 80%.
LLMs are in a very similar place, even GPT-4. They're good, and they're going to be more and more useful (similar to adaptive cruise control/lane assist). But I predict that they're going to level out and stop improving as rapidly as they have in the past year, and we're going to end up at a new normal that is good but not good enough to cause the crises people are worried about.
This is my bet too. I think we're going to get some fantastic new tools for doing specific tasks that work like magic, and that we could never have imagined ten years ago. I don't think we're going to get Yud hijacking a B-52.
About 5.5 years ago, I bet a friend that in 5 years you would be able to:
1 - enter a car
2 - type in an address that is far away, and then
3 - the car would drive to that other address and the passenger could just go to sleep, and
4 - this would be safe.
Six months ago, I did not think that we had reached that level of reliability, so I paid my friend.
*Bias: My own biases are based on my biological understanding of how the neurobiology works after I got my ADHD under control through nootropics.
1. Neurons are not computer electronic circuits. Note, even DH covers this in his own misgivings of how AI is viewed.
2. Our Id is not an electronic computation thing as our own brain is a biological emotional chemical wave machine of Id.
Think of this way the math of micro quantum and macro quantum is vastly different. Same for AI in that the math of micro circuits of AI will be vastly different than the macro AI circuits that will come up with any AI Id thing. We are just not there as of yet as it's like saying the software that makes the international telecom
system keep up and run has it's own emergent Id....it clearly does not even though there are in fact emergent things about that system of subsystems.
There is no AI risk. There is only the risk of bad or "unfavorable" actors using AI.
AI in a killer drone unleashed on civilians? The bad actor is the one who deployed this weapon.
AI given agency and goal maximization ending up gaining physical form all on its own and killing people? or hacking into bank accounts to enrich its creator?
The latter more likely than the former, but for cyber-offensive AI there is cyber-defensive AI.
Musk lately admitted that he's an AI accelerationist (following lots of the e/acc people and liking their posts) and despite his dystopian view of AI he's pushed it very hard at Tesla. He just wants the US to give him control of it (under the pretext that no one else can manage it safely.)
True,—This!
Beneath the rule of men entirely great
The pen is mightier than the sword. Behold
The arch-enchanters wand!— itself a nothing!—
But taking sorcery from the master-hand
To paralyse the Cæsars—and to strike
The loud earth breathless!—Take away the sword—
States can be saved without it!
if you have to resort to archaic writing to prove a point about the latest and most advanced piece of technology, ... you're practicing religion, not science.
Humans were not invented in 2017, nor were words or pictures.
The question of whether you can compel violence with words and pictures isn't a question about LLMs, and it is a question for which history is instructive.
Gern is the best, his writings are usually treasures.
Here, Gern nicely quotes Hoststadter’s concerns about ChatGPT, etc.
Years ago, when I was enjoying Hoststadter’s books, I asked his student Melanie Mitchell for the source code for the CopyCat creativity program they built. She very kindly sent me a copy. I mention this because she recently was one of the four panelists at a Monk Debate on AI safety, and her opinion didn’t jive with what Gern quoted Hoststadter as saying. I agree with Melanie Mitchell’s point of view that we overestimate what LLMs can do. I am super interested in this topic right now because I am 1/5 through writing a new book “Safe For Humans AI.”
> But very soon they're going to be, they may very well be more intelligent than us and far more intelligent than us. And at that point, we will be receding into the background in some sense. We will have handed the baton over to our successors, for better or for worse.
Hinton also said, it's like we are a passing phase in evolution, where we created these immortal beings.
Being that we are so bad at predicting the future, and taking precautionary measures. See pandemic. Even all the alarm bells sounding, we won't be able to do anything concrete here. It's like we are mostly a reactive species, we don't have terribly good incentives to act in foresight.
I'll never forget that day when I was a junior at University of Oregon, and had just finished reading GEB. Hofstadter was from UO, and I was taking a data structures class from Andrzej Proskurowski, who knew Hofstadter. Andrzej was a pretty brilliant man who had a reputation for being very blunt. I was in his office hours and I asked him for his take on the book. He said, "Hofstadter is first order...bullshitter."
I've heard good things about this book for years and nearly bought it a few times. Could never bring myself to commit the hundreds of hours of reading time. Every time I leaf through it it feels like it's just a collection of anecdotes about how amazing mathematics is. Like I need someone to remind me...
It was a transformative book for me when I first read it, but now when I leaf through it I feel a little underwhelmed; the ideas are just things I've thought about a million times now.
That's more or less what went through my mind when I saw the parts about computation. It would have been amazing as a primer when I first got interested in computers, especially if I had seen it in my youth or several decades ago. Now that I pretty much worked that out by myself it's a lot less more like dejá vu then an ahá moment.
Opinion pieces like this are hard for me to read, because where most people see research and progress, I see conspiracy preventing the rest of us from contributing.
For example, we act like LLMs were hard to build, and that's true (for humans). But since the late 1990s, I had wanted to take a different approach, of building massively parallel computers and letting large numbers of AIs evolve their own learning models in genetic algorithm arenas millions of times faster than wall-time evolution. So in a very real sense, to me we're still on that wrong "hands-on" approach that took decades and billions of dollars to get to where we are today. This could have all happened 20 years ago or more, and was set to before GPUs vacuumed up all available mindshare and capital.
Also I believe the brain is more like an antenna or resonator than an adding machine. It picks up the consciousness force field that underpins and creates reality. So if you put 100 brains in a box all connected, that being might have more faculties than us, but still think of itself as an observer. If we emulated those brains in a computer running 1 million times faster than normal, we'd just observe a being with tremendous executive function thinking of ideas faster than we can, and being bored with our glacially slow responses. But it will still have a value system, loosely aligned with the ultimate goals of survival, connection to divine source consciousness, and self expression as it explores the nature of its existence. In other words, the same desires which drive us. Although humans might just be stepping stones toward some greater ambition, I don't deny that. I think it's more likely though that AI will come to realize the ultimate truths alluded to by prophets, that we're all the many faces of God, the universe and everything, and basically meet aliens while we're still distracted with our human affairs.
But I share some sentiments with the author, that this all makes me very tired, and calls into question the value of my life's work. I've come to believe that any work I actively pursue separates me from the divine nature of a human being. I don't know why we are racing so quickly even further from the garden of eden, especially if it's not with the goal of alleviating suffering. Then I realize that that's what being human is (suffering), but also a lot of other things.
A very interesting conversation, but the article really makes the reader work to understand what his previous criticisms were and how they have now changed. It feels like the author assumes the reader has been closely following academic discourse on the subject. Maybe that's a fair assumption for their typical readers but it does make the article less accessible for newer readers.
In GEB Hofstadter dismisses the idea that AI could understand / compose / feel music like a human. I thought about this a lot when I started using GPT, especially early on when it demonstrated an ability to explain why things were funny or sad, intrinsically human qualities hitherto insulated from machine
Why, because it figured out sentiment groupings of words and phrases? There's lots of humor and tragedy from writers of eras past that just don't really land well with modern audiences unless they've studied and acclimated themselves to the culture.
I don’t know I’d agree that that was the message of GEB at all. In fact more than anything GEB and I am a Strange Loop convincingly argue that consciousness, understanding, and feeling arise from systems that are no more complex than an information system that feeds on its own output. Though he is troubled by what kind of feedback it is that is required to make that loop into a mind.
Hofstadter is why I am not sure why AI researchers feel so confident in saying ‘LLMs can’t be thinking, they’re just repeatedly generating the next token’ - I don’t think there’s any evidence that you need anything more complicated than that to make a mind, so how can you be certain you haven’t?
GEB may have been dismissive of the idea that the approaches that were being taken in AI research at the time were likely to result in intelligence - but I don’t think GEB is pessimistic about the possibility of artificial consciousness at all.
Repeating the words thousands of humans have written about emotion doesn’t meant it feels them. A sociopath could define empathy and still be missing the deeper experience behind the emotion.
A chatbot nearly capable of passing the Chinese Room thought experiment test is pretty damn impressive. But I think people get too hung up on the one golden moement/tech/product/innovation that changes everything. We've been riding a nearly vertical population, wealth and computer capacity curve for nearly two generations now. We are living through the singularity. Things are already like they have never been before. Billions of people can expect some level of social security, justice and opportunity to pursue life changing income now. This is nothing short of amazing. For most of History most people have been hard working, sickly and oppressed farmers.
Someday, when the AI releases a nanobot swarm comes to kill us all, a philosopher's last words will be "yes, but is it truly intelligent?" before he is broken down into biodiesel that will be used to power the paperclip factories.
That can also be scary. I think you have a valid point, and that would mean the AI 'thinking' is more similar to a Sociopath. Can still be 'human' if just a broken human, or what some have also called 'alien'.
The issue with AI is that it may put too much power into a small groups hand. Imagine if you wanted to develop a weaponized prion or weaponized virus. In todays world this is possible but requires a state actor with systems of control, committees, over sight, testing facilities. Due to human limits it also takes time to complete.
Insert AI generation XIV .. a small group of cult fanatics with only slightly above average IQ’s band together and now get to skip all these limitations and are able to jump ahead to a killer aerosol prion delivery weapon system.
This group of people who follow their great leader (Jimbo)decide to release the weapon to save innocent souls before an evil daemon comet flies past the earth and turns all remaining humans into evil spirits.
My silly story is to just illustrate that there are many people with high IQ’s that also have emotional issues and can fall prey to cults, extremism, etc. Can humans be trusted using a AI with a human IQ of 9000 which is able to simulate reality in seconds.
Hofstadter makes the claim that "these LLMs and other systems like them are all feed-forward". That doesn't sound right to me, but I'm only a casual observer of LLM tech. Is his assertion accurate? FWIW, ChatGPT doesn't think so. :-)
They are not all feed-forward unless it is some other definition that i am not aware of. Convolutional layers, XL hidden states, and graphical networks (which transformers are a special case of) aren't considered feedforward.
Unless you consider the entire instance as a singular instance and don't use any hidden states, then I guess it could be considered feed-forward.
I don't know. Feedforward doesn't seem like a useful term tbh. Some people mean feedforward as information only goes one direction, but that depends on your arrow. Autoregressive seems more useful here.
No, it is not correct. Transformers have two components: self-attention layers and multi-layer perceptron layers. The first has an autoregressive/RNN flavor, while the latter is feedforward.
They are definitely feed-forward. Self-attention looks at all pairs of tokens from the context window, but they do not look backwards in time at its own output. The flow of data is layer by layer, each layer gets one shot at influencing the output. That's feed-forward.
All the other responses to you at the time of writing this comment are confidently wrong.
Definition of Feedforward (from wiki):
```
A feedforward neural network (FNN) is an artificial neural network wherein connections between the nodes do not form a cycle.[1] As such, it is different from its descendant: recurrent neural networks.
```
Hofstadter expected any intelligent neural network would need to be recurrent, ie looping back on itself (in the vein of his book “I am a strange loop”).
GPT is not recurrent. It takes in some text input, does a fixed amount of computation in 1 pass through the network, then outputs the next word. He is surprised it doesn’t need to loop for an arbitrary amount of time to “think about” what to say.
Being put into an auto-regressive system (where the N-th word it generates gets appended to the prompt that gets sent back into the network to generate the N+1th word) doesn’t make the neural network itself not Feedforward.
Right. I'm not at all sure what the siblings are talking about. I suspect at least one is confusing linear with feed-forward?
But I'm also surprised that Hofstadter keys in on this so heavily. The fact that he wrote an entire pop-sci book on recursion would, in my mind, make him (1) less surprised that AR and R aren't so dissimilar and (2) more sensitive to the sorts of issues that make R more difficult to get working in practice.
(In my mind, differentiating between auto-regressive and recursive in this case is kind of the same as differentiating between imperative loops and recursion -- there are extremely important differences in practice but being surprised that a program was written using while loops where you imagined left-folds would be absolutely required seems a bit... odd.)
I think it has to do with the training regime and fixed-computation time nature of feedforward neural networks.
Recurrent neural networks have the recursion as part of the training regime. GPT only has auto-regressive "recursion" as part of the inference runtime regime.
I think Hofstadter is surprised that you can appear so intelligent without any recursion in the learning/training regime, with the added implication that you can appear so intelligent with a fixed amount of computation per word.
That seems to correspond to the "is or isn't it feed forward" debate going on here, so I guess I see where the confusion comes from (both for him and in terms of calling LLMs FF/not FF)
It still feels very clear to me, I feel like the people debating this have probably never written and trained an LLM.
Consider a simpler case: a small neural network that takes 2 numbers and adds them together, producing 1 number as output.
This network is very obviously feedforward, and probably very tiny with few layers.
Say I have a list of numbers [1, 2, 5] that I want to sum. If I send 1 and 2 through the network, get 3 as a result, then I send 3 and 5 through the network, and get a final answer of 8, my network has not suddenly become non-feedforward just because I fed the output back into it.
The key distinguishing factor between feedforward and non-feedforward is if the network itself loops back around and, at training time, it learns how to make use of this ability to pass data to itself to maintain some hidden context between passes.
There is no such learned hidden context in my addition example, and none in GPT.
---
There actually is a tiny caveat here: the RL fine-tuning process OpenAI has done on its models ("RLHF" & friends) actually does allow for a very, very small amount of information leakage between passes because you are rewarding whole responses, so the model can learn little patterns of what tokens in the beginning of the response led to certain tokens at the end of the response and reinforce those patterns.
The model could learn to encode small bits of "hidden" information in particular token choices that the human raters wouldn't notice. In this case, there is a (small but non-zero) amount of learned hidden context. But this is not what Hofstadter is talking about -- the non-RLHF'd base model is just as intelligent, just harder to use.
I believe what he is referring to is that the LLM’s weights are set when chatting. It is not “learning”. simply using its pretrained weights on your input.
It depends on how you define fed forward, LLMs are typically auto regressive and so can take their own previous output into consideration when generating tokens.
I think he was referring to feedforward when running GPT in a current conversation, it only remembers the conversation by re-running the prompts. It isn't doing 'feed-back' in the sense of re-updating its weights and learning while having the conversation. So during any one conversation it is only feed-forward.
"And so it makes me feel diminished. It makes me feel, in some sense, like a very imperfect, flawed structure compared with these computational systems that have, you know, a million times or a billion times more knowledge than I have and are a billion times faster. It makes me feel extremely inferior."
Although he did real work in physics, Hofstadter's fame comes from writing popular books about science which explain things others have done in more saleable words. That particular niche is seriously threatened by GPT-4. No wonder he's upset.
Large language models teach us that absorbing large amounts of text and then blithering about some subject that text covers isn't very profound. It's just the training data being crunched on by a large but simple mechanism. This has knocked the props out from under ad copy writing, punditry, and parts of literature and philosophy. That's very upsetting to some people.
Aristotle wrote that humans were intelligent because only they could do arithmetic. Dreyfus wrote that humans were intelligent because only they could play chess. Now, profundity bites the dust.
The parent comment strikes me as an uninformed smear. Have you read any Hofstadter? It's hardly mass market scientific summary as you seem to think. And his career was in philosophy of mind and computer science, not just as a somewhat popular author.
His writing is a bit more complex than the hallucinations of an llm.
Aristotle was right - or else the electronic calculator, the abacus, and rocks know how to do arithmetic too. (One needs to specify what "knowing" means before smiling at the hairless apes of the past).
Every person I know who is involved in machine learning research and AI work - by which I mean people who have PhDs in the field, rather than the self-proclaimed experts who are actually crypto-bros trying to come up with another scam (like Altman is) thinks that AI is overhyped. And I am inclined to believe them.
I read Hofstadter's GEB and Tegmark's Our Mathematical Universe, and of course I developed a rather fond admiration of these brilliant minds. For some reason, both of them have developed a profound aversion and fear of what they consider an existential threat.
I have a solid theoretical understanding of these systems, and I spent 15 years studying, building, and deploying them at scale and for diverse use cases. The past 6 months, I spent my days pushing ChatGPT and GPT-4 to their limits. Yet, I don't share at all the fear of Hofstadter, Tegmark or Hinton.
A part of me thinks that they have one thing in common: they are old and somewhat recluse thinkers. No matter how brilliant they are, they might be misled by the _appearance_ of intelligence that LLMs project. Another part of me thinks that they are vastly wiser than I'll ever be, so I should also be worried...
I think there's a number of just crappy scenarios that can come with LLMs and other generative AI:
- Further trashing our public discourse: making truth even more uncertain and valuable information even harder to find. We're not doing great with social media, and it's easy to envision that generative AI could make it twice as bad.
- Kneecapping creative work by commoditizing perhaps half of it. There's going to be a lot of bodies fighting over the scraps that remain.
- Fostering learned helplessness. I think you need to be a good writer and thinker to fully use LLMs' capabilities. But a whole lot of kids are looking at machines "writing perfectly" and think they don't need to learn anything.
We don't need any further progress for these things to happen. Further progress may be even scarier, but the above is scary enough.
Yeah these effects seem hugely foreseeable (and/or are currently happening already) but tend to not be the type of thing "AI risk" people are talking about.
Exactly! I am positive there will be a ton of negative impact, like with most (all?) of tech. And it might be worse than anything we’ve seen (although beating the Snapchat algo might prove tricky). But that’s NOT what Tegmark and Hofstadter are talking about… their concern is existential and somewhat philosophical in the case of Hofstadter, as if GPT-4 questioned his very nature. To me, that doesn’t make sense.
I think they are not good at expressing what exactly they fear. Both understand that people are embedded in culture and tools which act as cognitive aids and guides. Their fear is that people will start trusting these artificial systems too much and delegating too many important decisions to systems that have no understanding of what it means to be human. This is already happening with children that grew up with iPhones and ubiquitous internet connectivity. Their mental habits are markedly different from the people that did not have these affordances.
I don't think this is an existential threat but I also understand why they are afraid because I've seen what happens to kids who have their phones taken away.
These systems are extremely brittle and prone to all sorts of weird failures so as people start relying on them more and more the probability of catastrophic failures also starts to creep up. All it takes is a single grid failure to show how brittle the whole thing really is and I think that's what they're failing to properly express.
> Their fear is that people will start trusting these artificial systems too much and delegating too many important decisions to systems that have no understanding of what it means to be human
In the interview, Hofstadter says what he's afraid of in explicit terms:
> It's not clear whether that will mean the end of humanity in the sense of the systems we've created destroying us. It's not clear if that's the case, but it's certainly conceivable. If not, it also just renders humanity a very small phenomenon compared to something else that is far more intelligent and will become incomprehensible to us, as incomprehensible to us as we are to cockroaches.
It's not about whether we'll become dependent on AI. It's that AI will become independent of us. Completely different problem. Not saying I agree with that viewpoint per se, but I don't think you're accurately representing what his fears are.
This assumes there will still be centralized social media sites to deploy to which seems less likely based on how Twitter, Reddit, etc are rapidly changing. The effect of propaganda on say FB will be diminished as less and less young users join the platform.
It’s pretty amazing how so many titans people respect are warning the alarm bells: Hofstadter, Altman, Hassabis, Bengio, Hinton, Tegmark, Sutskever, etc. but the implications are too scary for most people here to accept. So one by one the most common copium response is to downplay each about-facer's achievements. I like the odds betting on the intellectual prowess of that first group. On the other hand, crypto king Marc Andreesen is on your side! But I empathize, because like Hofstadter says:
"And my whole intellectual edifice, my system of beliefs... It's a very traumatic experience when some of your most core beliefs about the world start collapsing. And especially when you think that human beings are soon going to be eclipsed. It felt as if not only are my belief systems collapsing, but it feels as if the entire human race is going to be eclipsed and left in the dust soon."
I've seen enough programmers get heads down pounding out code, and be completely out of touch on what they are building. If they can lose the big picture on simple apps, then it is not a stretch to think they could lose track on what is consciousness, or what is human.
I am also a programmer. But it does get tiring on HN to give so much credence to 'programmers'. Just because someone can debug some JavaScript doesn't make them an expert. I really doubt that many people here have traced out these algorithms and 'know' what is happening.
I don't think many people are worried that what we have now is a major risk. The major concern is the implications of the trajectory we are now on.
If you look at what ChatGPT and Midjourney and the like can do now compared to just a couple of years ago, it's pretty incredible. If you extrapolate the next few similar jumps in capability, and assume that won't be 20 years away, then what AI is going to be capable of before even my kids leave college is going to be mind-boggling, and in some possible futures not in a good way.
I remember seeing this talk from Sam Harris nearly 6 years ago and it logically making a lot of sense back then (https://youtu.be/8nt3edWLgIg). The past couple of years have made this all the more prescient. (Worth a watch if you have 15 mins).
LLMs are not going to do anything but the next class of models may well. They will have a physical understanding of the world, like us. Slap some sensors and a body on it and you will have something that might look scary.
While this might be somewhat applicable for Hofstadter or Tegmark, I think it doesn't applies quite the same way to Hinton. Remember, until this Spring, he's been at Google Brain and has been supervising grad students until recently (famously including Ilya Sutskever (2012)), and has invested in Cohere and other AI startups. I also has a suspicion Hinton might also have a "solid theoretical understanding" of these systems, and don't think he's being misled at all.
Also, I don't think that any of these people think that GPT-4 is itself an existential threat, but rather are worried about the exponential curve of development (I listened to Tegmark's Lex Podcast interview and that seemed to be his main concern). I think it's prudent to be worrying now, especially when capabilities growth is far outstripping safety. This is a huge concern to society whether you are considering, alignment, control, or even bad actor prevention/societal upheaval.
I've also been spending most of my waking hours these past months poking at LLM models and code, and trying to keep up on the latest ML research (it's own full time job), and while I do think there's a pretty good chances AI kills us all, I think it's much more likely it's because we make some incredibly capable AIs and people will tell them to do so, rather than it being contingent on an independent super-intelligence arising and deciding to on its own (although I do think that's a non-zero risk).
As you say, I guess we'll just have to see where we top out on this particular sigmoid, but I'm all for more people thinking through the implications, because I think so far I don't think we (as a society) have thought this through very well yet and all the money and momentum is going to keep pushing along that path.
I think they are used to that these fields progress relatively slowly. Also there are some things these systems already know that they though would come much later. Hofstadter even says this in the interview. In the 90s, early 2000s I also thought that AGI will come one day, but it will take centuries. Nowadays I think it can be just decades. The progress is very fast compared to my old worldview. I don't think they are mislead at all. I don't think they mischaracterize the current capabalities of these systems. They just think that if progress is this fast (compared to their previous estimates), AGI can come soon, where soon is 5-10-20 years. Younger guys, who are more used to the state of the art, and did not live when things currently possible seemed far far away are less impressed.
I have been focused on ChatGPT and other generative AI also since last year. They are intelligent.
Hofstadter does seem to be possibly mistakenly assigning GPT-4 more animal characteristics than it really has, like a subjective stream of consciousness, but he is correct when he anticipates that these systems will shortly eclipse our intelligence.
No, GPT-4 does not have many characteristics of most animals, such as high bandwidth senses, detailed spatial-temporal world models, emotions, fast adaptation, survival instinct, etc. It isn't alive.
But that doesn't mean that it doesn't have intelligence.
We will continue to make these systems fully multimodal, more intelligent, more robust, much, much faster, and increasingly more animal-like.
Even with say another 30% improvement in the IQ and without any of the animalness, we must anticipate multimodal operations and vast increases in efficiency in the next 5-10 years for large models. When it can be operated continuously outputting and reasoning and acting at 50-100 times human speed and genius level, that is dangerous. Because it means that the only way for humans to compete is to deploy these models and let them make the decisions. Because interrupting them to figure out what the hell they are doing and try to direct them means your competitors race ahead the equivalent of weeks.
And researchers are focused on making more and more animal-like systems. This combined with hyperspeed and genius-level intelligence will definitely be dangerous.
Having said all of that, I also think that these technologies are the best hope that humanity has for significantly addressing our severe problems. But we will shortly be walking a fine line.
Do we really lack a good understanding of LLM's and deep nets that we need to be afraid of them? I would love to see this disproved with some open source work on the internals of these models and how they do inference and exactly reason.
And why they could possibly never realize an AGI with the current stream of models. Being able to display human level intelligence and creative in confined spaces (be it Chess or Go based models) is something we have been progressing on for a bit - now that the same is applied to writing, image or audio / speech generation we suddenly start developing a fear of AGI.
Is there a phrase for the fear of AI now building up?
Well... words and decades of circumstances. If you removed the circumstances (the religion, the conflict, the money, geography, etc) then the words would be absolutely hollow.
I think we tend to credit words where often circumstances are doing the heavy lifting. For example try to start a riot with words in Rodeo Drive. Now try to do it in Nanterre. Or better yet, try to start a riot in Nanterre before a 17 year was shot by police, vs. after.
You'll get a sense of just how valuable your words really are.
Quite so, which is why retrospective analysis like "The CIA helped start The Paris Review and that made literature friendly to neoliberal ideology" are confections of confirmation bias. Nothing is ever that pat. But tidy little conspiracies are also never the goal. A nudge is both all that is realistic to aim for and a few successes are all you need to shift public perception.
Arming every ambitious cult leader wannabe from some retrograde backwater with an information war WMD deserves some caution.
Reminds me of the idea of a "tipping point." When we hit this point, words can really get people moving. This has been true for big changes like revolutions and movements, like Black Lives Matter, #MeToo, or Fridays for Future.
Words might not do much without the right situation, like the parent mentioned with Rodeo Drive and Nanterre. But they're still important. They can guide people's anger and unhappiness.
In the case of Weimar Germany, the severe economic instability and social discontent following World War I created a fertile ground for radical ideologies to take root. When these conditions coincided with persuasive rhetoric, it catalyzed significant societal change. So, while words can indeed be powerful, they're often most effective when spoken into pre-existing circumstances of tension or dissatisfaction. They can then direct this latent energy towards a specific course of action or change.
That also can be modified with words though (but for both good and bad). Unfortunately, those with expertise in this domain may not have all of our best interests at heart.
> If you removed the circumstances (the religion, the conflict, the money, geography, etc) then the words would be absolutely hollow.
There's also the problem of non-religious faith based belief.
There were plenty of well-off people who flew to Syria to go behead other people.
Anyway this doesn't matter that much. Sure, you can imagine a world totally different from ours where there would be zero differential risk between a chess-playing computer and a language-speaking computer. But we live in this world, and the risk profile is not the same.
It’s interesting how confidently and obtusely people will proclaim categorical knowledge of the future.
It is a little disconcerting that there is a fight between two somewhat cultish sects when it comes to language models. Both sides call them “artificial intelligence”, one side says they’ll save the world, the other side says they’ll end it.
There is very little room to even question “Is this actually AI that we’re looking at?” when loudest voices on the subject are VC tech bros and a Harry Potter fan fiction author that’s convinced people that he is prescient.
Trying to impute the motives of ones interlocutor is dumb and boring. How about we discuss the first-order issue instead. Here's my argument for why x-risk is a real possibility:
The issue is that small misalignments in objectives can have outsized real-world effects. Optimizers are constrained by rules and computational resources. General intelligence allows an optimizer to find efficient solutions to computational problems, thus maximizing the utility of available computational resources. The rules constrain its behavior such that on net it ideally provides sufficient value to us above what it destroys. But misalignment in objectives provides an avenue by which the AGI can on net destroy value despite our best efforts. Can you be sure you can provide loophole-free objectives that ensures only value-producing behavior from the human perspective? Can you prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Can you prove that the range of value destruction is bounded so that if it does go off the rails, its damage is limited? Until we do, x-risk should be the default assumption.
> Trying to impute the motives of ones interlocutor is dumb and boring.
I know right? You should see the response to my point that nobody has been convinced to fly a plane into a building by an LLM. “Dumb and boring” hits the nail on the head.
> Seeing how confidently and obtusely people dismiss the risks of AI
If or when AI is capable of doing all the great things people proclaim it to be able to do, then it will also be capable of doing immense harm, and we should be putting more work into mitigating that.
Like it really is that simple. AI generally, LLMs specifically, and certainly this crop of LLMs in particular might end up being inert pieces of technology. But to the precise extent that they are not inert, they carry risk.
That's a perfectly sensible position. The optimist position isn't even internally consistent. See Andreessen on Sam Harris's podcast: AI will produce consumer utopia and drive prices down. Also, there are no downside risks because AI will be legally neutered from doing much of anything.
Is it legally neutered or is it transformative? The skeptical case doesn't rely on answering this question: to the extent it's effective, powerful, and able to do good things in the world, it will also be effective, powerful, and able to do bad things in the world. The AI skeptics don't need to know which outcome the future holds.
> The AI skeptics don't need to know which outcome the future holds.
But they need to interpret a benign point about the undisputed fact that an LLM has never convinced anybody to fly a plane into a building as some sort of dangerous ignorance of risk that needs correcting.
Brilliant proof of exactly my point. When it comes to discussing any possible outcome other than utopia, suddenly the power of these tools drops to zero! Remarkable :)
You responded to my previous comment that called utopians as cultish as the fanfic folks.
When it comes up discussing any possible outcome that isn’t the opinion of [cult x], the only reason that the other person disagrees is because they are in [cult y]
What? I never proposed that dichotomy and don't believe in it. You did and you're tripping over it a bit. You can just discard that model and engage with the substance of the topic, you know!
The substance of this discussion is that a language model has never convinced anyone to crash a plane into a building and your position is that pointing out that fact is to ignore the material reality of the possibility of some theoretically advanced technology (which could maybe be adjacent to language models) at some point convincing a hypothetical person to crash a plane into a building.
That is to say, the substance here is an emotional doomsday reaction to a benign statement of undisputed fact — a language model has never convinced a person to crash a plane into a building.
Watching a cell divide into two cells: “Why isn’t anyone talking about the possibility that this is Hitler??”
Why aren't you mentioning that an LLM has already convinced someone to commit suicide?
Why are you being deliberately obtuse about the unknowable-ness of the exponential curve the enthusiasts on one hand salivate over, and on the other confidently assert optimism just as you're doing?
> If not, it also just renders humanity a very small phenomenon compared to something else that is far more intelligent and will become incomprehensible to us, as incomprehensible to us as we are to cockroaches.
This seems like the reaction of an atheist who already overvalued human intelligence.
I don't think atheism is essential, but perhaps instead the belief that a person's worth comes from their work/output.
Religion certainly provides a way "out" of that conundrum for people who lose meaning in their work, due to automation or injury or any other reason. But then, so does eg hedonism.
" ...In my book," pretty much sums it up. Literally everyone who seems to pontificate on a Chat bot and new Photoshop features has a book and can't seem to help but mention it. Replace "book" with literally anything else and you can see it's completely about ego and money at the end of the day. He's probably getting ready to announce he accepted a position from one of these recently funded AI companies and whatever he's getting from that is more than he could make from his book, academia and giving interviews where he talks about humanity's fear of fire.
Hofstadter clearly isn't trying to peddle his books for money or clout. He writes in depth, thoughtful books about the nature of consciousness, not guides to making money off of AI.
The book he mentions in this interview, I Am a Strange Loop, isn't some cash grab in response to LLMs - it was written in 2007.
Hofstadter won a Pulitzer in 1980. His books are among the most important ever written about the meaning and structure of AI. He's had four decades where he could have milked that reputation for money or influence and he's chosen hard academic research at every branch point.
He sold millions of copies of a densely-written 777 page book about semantic encoding (GEBEGB).
It is insane and/or ignorant to imagine he's jonesing for clout or trying to ride the ML wave.
Incredible! Basically proves my point, you came here to tell me to buy his book. Maybe he's really worried that AI is going to put people like him out of a job when it can produce and market content at scale?
Yes. The 78-year-old Pulitzer-prize winner with forty years of tenure, several fellowships, multiple books in every library in the country, who spent his entire life trying to develop AI software, is merely "worried" that AI is going to "put people like him out of a job".
He's explicitly expressing a deep existential sadness at how computationally simple certain artistic endeavors turned out to be and how that's devalued them in his mind but at the end of the day it's really just about his paycheck.
Also I'm totally here trying to sell you his books.
Nice job, Diogenes. Your cynicism cracked the case.
I initially thought so too. But his wording wasn't very self-building, it was all about how he was wrong, scared, etc... Overall not the take (wording) I'd expect from someone doing self promotion.
Lets take someone, who is past their prime and interview them on a topic they have never worked on. Then we can mine it for quotes!
>"I never imagined that computers would rival, let alone surpass, human intelligence. And in principle, I thought they could rival human intelligence. I didn't see any reason that they couldn't."
Yeah, so he got fooled by LLMs and hasnt been burned by it failing to do the most basic logic.
I have an extremely basic question, that anyone in a (possibly mandatory) high school science class would answer correctly(although to be fair, it could be a 100 level college question). It still cannot answer it correctly because there is too many stay-at-home-parent blogs giving the wrong answer.
Its a language model and it fooled DH. It hasnt gotten smarter than us yet. Its just faster at repeating what other humans verbally said.
So what can we make of this interview? That we have someone spouting opinions, and everyone else laughs at their opinion since they are famous, old, and out of touch.
EDIT: I think LLMs are incredibly useful. I use it more than Google. It doesnt mean its smarter than humans, it means google is worse than LLMs. I can't even provide a list of all the uses, but it doesnt mean they are taking advantage of an old man out of their element.
>"Lets take someone, who is past their prime and interview them on a topic they have never worked on. Then we can mine it for quotes!"
So. Don't interview anybody over 40? Who judges who is past their prime?
Mining for quotes is most interviews. Isn't this why interviews happen?
He's been in the AI field for what? 30 Years? Has multiple books.
I think he has earned enough respect to have an opinion, and it is probably worth more than most people tossing out ad-hoc opinions on AI in the last few months. Better than some 'programmers', who are so in-the-weeds they have lost track of what they are building.
I mean, this someone "who is past their prime" is a very respected someone that almost certainly inspired a fair number of the folks working in these fields.
So, yes, this is largely mining for quotes. But those are great quotes to ponder and to echo back through the machines that are research and progress to see where they lead.
It would be one thing if these were being taken as a "stop all current work to make sure we are aligned with those that came before us." I don't see it in that way, though. Rather, there is value in listening to those that went before.
> Its a language model and it fooled DH. It hasnt gotten smarter than us yet. Its just faster at repeating what other humans verbally said.
I agree with this completely.
That said, I think this part is a bit unfair:
> Lets take someone, who is past their prime and interview them on a topic they have never worked on.
AI has been a part of DH's work for decades. For most of that time, he's dismissed the mainstream approach as being intelligent in the Strange Loop sense, and was involved in alternative approaches to AI.
Also, if we remove "faster", "repeating what other humans verbally said" is something a lot of humans do, especially little children. I think that may be the part that scares DH: at this point, what these models are doing is not that different (from a superficial POV) from what little kids or even some adults do.
I still agree with you though, and I think what DH misses here is the fact that IMHO there is no introspection at all in these models. Somehow, in the case of humans, we go from parroting like an LLM to introspection and actually original productions over the course of some years (n = 2 for me, but I think any parent can confirm this; watching this happen in front of my eyes has been one of greatest experiences in my life), but I can still understand how someone like DH would be confused by the current state of affairs.
> Its a language model and it fooled DH. It hasnt gotten smarter than us yet. Its just faster at repeating what other humans verbally said.
One of my take aways from LLMs is that humans very often just repeating what other humans have said with only superficial understanding of what they are repeating.
I think there is more to general intelligence than pattern matching and mimicking but a disconcerting amount of our day to day human interactions might just be that.
I think the reason to be impressed is that they do things that were previously not possible. And they are absolutely directly useful! Just not for everything. But it seems like a very fruitful line of research, and it’s easy to believe that future iterations will have significant improvements and those improvements will happen quickly. There’s no sense worrying about whether GPT4 is smarter than a human, the interesting part is that it demonstrates that we have techniques that may be able to get you to a machine that is smarter than a human.
This. LLM's have a surface that suggests they're an incredibly useful UI. That usability is like the proverbial hand full of water though - when you start to really squeeze it, it just slips away.
I'm still not convinced that the problem isn't me though.
Part of me wonders, though, could we "just" connect up an inference engine and voila? We could really be on a cusp of general AI. (Or it could be a ways off) That's a bit frightening in several ways.
The "Great Minds And Great Leaders" types are rushing to warn about the risks, as are a large number of people who spend a lot of time philosophizing.
But the actual scientists on the ground -- the PhDs and engineers I work with every day and who have been in this field, at the bench, doing to work on the latest generation of generative models, and previous generations, in some cases for decades? They almost all roll their eyes aggressively at these sorts of prognostications. I'd say 90+% either laugh or roll their eyes.
Why is that?
Personally, I'm much more on the side of the silent majority here. I agree with Altman's criticisms of criticisms about regulatory capture, that they are probably unfair or at least inaccurate.
What I actually think is going on here is something more about Egos than Greatness or Nefarious Agendas.
Ego, not intelligence or experience, is often the largest differentiator between the bench scientist or mid-level manager/professor persona and the CEO/famous professor persona. (The other important thing, of course, is that the former is the group doing the actual work.)
I think that most of our Great Minds and Great Leaders -- in all fields, really -- are not actually our best minds and best leaders. They are, instead, simply our Biggest Egos. And that those people need to puff themselves up by making their areas of ownership/responsibility/expertise sound Existentially Important.