> The article skirts around a central question: what defines humans? Specifically, intelligence and emotions?

> The entire article is saying "it looks kinds like a human in some ways, but people are being fooled!"

> You can't really say that without at least attempting the admittedly very deep question of what an authentic human is.

> To me, it's intelligent because I can't distinguish its output from a person's output, for much of the time.

I think the article does address that rather directly, and that it is also is addressing very specifically your setence about what you can and can't distinguish.

LLMs are not capable of symbolic reasoning[0] and if you understand how they work internally, you will realize they do no reasoning whatsoever.

Humans and many other animals are fully capable of reasoning outside of language (in the former case, prior to language acquisition), and the reduction of "intellgence" to "language" is a catagory error made by people falling vicim to the ELIZA effect[1], not the result of a sum of these particular statistical methods being equal real intelligence of any kind.

0: https://arxiv.org/pdf/2410.05229

1: https://en.wikipedia.org/wiki/ELIZA_effect

> LLMs are not capable of symbolic reasoning[0]

Despite the citation. I think this is still being studied. And others have found some evidence that it forms internal symbols.

https://royalsocietypublishing.org/doi/10.1098/rsta.2022.004...

Or maybe, can say, an LLM can do symbolic reasoning, but can it do it very well? People forget that humans are also not great at symbolic reasoning. Humans also use a lot of cludgy hacks to do it, it isn't really that natural.

Example often used, about it not doing math well. But humans also don't do math well. How humans are taught to do division and multiplication, really is a little algorithm. So what would be difference between human following algorithm to do a multiplication, and an LLM calling some python to do it. Does that mean it can't symbolically reason about numbers? Or that humans also can't?

> the reduction of "intellgence" to "language" is a catagory error made by people falling vicim to the ELIZA effect[1], not the result of a sum of these particular statistical methods being equal real intelligence of any kind.

I sometimes wonder how many of the people most easily impressed with LLM outputs have actually seen or used ELIZA or similar systems.

> isn't this also telling us what "understanding" is?

When people start studying theory of mind someone usually jumps in with this thought. It's more or less a description of Functionalism (although minus the "mental state"). It's not very popular because most people can immediately identify an phenomenon of understanding separate from the function of understanding. People also have immediate understanding of certain sensations, e.g. the feeling of balance when riding a bike, sometimes called qualia. And so on, and so forth. There is plenty of study on what constitutes understanding and most healthily dismiss the "string of words" theory.

A similar kind of question about "understanding" is asking whether a house cat understands the physics of leaping up onto a countertop. When you see the cat preparing to jump, it take a moment and gazes upward to its target. Then it wiggles its rump, shifts its tail, and springs up into the air.

Do you think there are components of the cat's brain that calculate forces and trajectories, incorporating the gravitational constant and the cat's static mass?

Probably not.

So, does a cat "understand" the physics of jumping?

The cat's knowledge about jumping comes from trial and error, and their brain builds a neural network that encodes the important details about successful and unsuccessful jumping parameters. Even if the cat has no direct cognitive access to those parameters.

So the cat can "understand" jumping without having a "meta-understanding" about their understanding. When a cat "thinks" about jumping, and prepares to leap, they aren't rehearsing their understanding of the physics, but repeating the ritual that has historically lead them to perform successful jumps in the past.

I think the theory of mind of an LLM is like that. In my interactions with LLMs, I think "thinking" is a reasonable word to describe what they're doing. And I don't think it will be very long before I'd also use the word "consciousness" to describe the architecture of their thought processes.

That’s interesting. I thought your cat analogy (which I really liked) was going to be an example of how LLMs do not have understanding the way a cat understands the skill of jumping. But then you went the other way.

> Another thing that I hear from time to time is an argument along the line of "it just predicts the next word, it doesn't actually understand it". Rather than an argument against AI being intelligent, isn't this also telling us what "understanding" is? Before we all had computers, how did people judge whether another person understood something? Well, they would ask the person something and the person would respond. One word at a time. If the words were satisfactory, the interviewer would conclude that you understood the topic and call you Doctor.

You call a Doctor 'Doctor' because they're wearing a white coat and are sitting in a doctor's office. The words they say might make vague sense to you, but since you are not a medical professional, you actually have no empirical grounds to judge whether or not they're bullshitting you, hence you have the option to get a second or third opinion. But otherwise, you're just trusting the process that produces doctors, which involves earlier generations of doctors asking this fellow a series of questions with the ability to discern right from wrong, and grading them accordingly.

When someone can't tell if something just sounds about right or is in fact bullshit, they're called a layman in the field at best or gullible at worst. And it's telling that the most hype around AI is to be found in middle management, where bullshit is the coin of the realm.

Hmm, I was actually thinking of a viva situation. You sit with a panel of experts, they talk to you, they decide whether you passed your PhD in philosophy/history/physics/etc.

That process is done purely by language, but we supposed that inside you there is something deeper than a token prediction machine.

> The entire article is saying "it looks kinds like a human in some ways, but people are being fooled!"

The question is, what's wrong with that?

At some level there's a very human desire for something genuine and I suspect that no matter the "humanness" of an AI, it will never be able to close that desire for genuine. Or maybe... it is that people don't like the idea of dealing with an intelligence that will almost always have the upper hand because of information disparity.

We cannot actually judge whether something is intelligent in some abstract absolute way; we can only judge whether it is intelligent in the same way we are. When someone says “LLM chatbot output looks like a person’s output, so it is intelligent”, the implication is that it is intelligent like a human would be.

With that distinction in mind, whether an LLM-based chatbot’s output looks like human output does not answer the question of whether the LLM is actually like a human.

Not even because measuring that similarity by taking text output at a point in time is laughable (it would have to span the time equivalent of human life, and include much more than text), but because LLM-based chatbot is a tool built specifically to mimic human output; if it does so successfully then it functions as intended. In fact, we should deliberately discount the similarity in output as evidence for similarity in nature, because similarity in output is an explicit goal, while similarity in underlying nature is a non-goal, a defect. It is safe to assume the latter: if it turned out that LLMs are similar enough to humans in more ways than output, they would join octopus and the like and qualify to be protected from abuse and torture (and since what is done to those chatbots in order for them to be useful in the way they are would pretty clearly be considered abuse and torture when done to a human-like entity, this would decimate the industry).

That considered, we do not[0] know exactly how an individual human mind functions to assess that from first principles, but we can approximate whether an LLM chatbot is like a human by judging things like whether it is made in a way at all similar to how a human is made. It is fundamentally different, and if you want to claim that human nature is substrate-independent, I’d say it’s you who should provide some evidence—keeping in mind that, as above, similarity in output does not constitute such evidence.

[0] …and most likely never could, because of the self-referential recursive nature of the question. Scientific method hinges on at least some objectivity and thus is of very limited help when initial hypotheses, experiment procedures, etc., are all supplied and interpreted by the very subject being studied.

Maybe it needs blood and flesh to be able for us to happily accept it.

This kind of mockery is unproductive and doesn't constitute an actual argument against the position it describes.

Sadly there are plenty of arguments that boil down to "AI can't be reasoning because they don't do everything humans do", including things such as being embodied, "having consciousness" or some postulated quantum effects in the brain making humans special[0].

Drawing a line around the bag of things that humans do and calling that reasoning isn't all that conductive to discussion either because it's a rather large bag, some parts are idiosyncratic and others aren't well-defined.

[0] https://en.wikipedia.org/wiki/Orchestrated_objective_reducti...

This isn’t that hard, to be honest. And I’m not just saying this.

One school of thought is - the output is indistinguishable from what a human would produce given these questions.

Another school of thought is - the underlying process is not thinking in the sense that humans do it

Both are true.

For the lay person, calling it thinking leads to confusions. It creates intuitions that do not actually predict the behavior of the underlying system.

It results in bad decisions on whether to trust the output, or to allocate resources - because if the use of the term thinking.

Humans can pass an exam by memorizing previous answer papers or just memorizing the text books.

This is not what we consider having learnt something. Learning is kinda like having the Lego blocks to build a model you can manipulate in your head.

For most situations, the output of both people is fungible.

Both people can pass tests.

This is maybe the best response thus far. We can say that there's no real modelling capability inside these LLMs, and that thinking is the ability to build these models and generate predictions from them, reject wrong models, and so on.

But then we must come up with something other than opening up the LLM to look for the "model generating structure" or whatever you want to call it. There must be some sort of experiment that shows you externally that the thing doesn't behave like a modelling machine might.

Heh, this phrasing has resulted in 2 “best response” type comments in the 2 times I’ve used it.

I think maybe it makes sense for people who already have the building blocks in place and just require seeing it assembled.

Aren't halluciantions enough proof for you that they don't think/understand? At least not in the same way as humans?

If a student was on a regular basis hallucinating and giving complete nonsense as an answer, I don't think they'll pass their studies.

To me, it's empathetic and caring. Which the LLMs will never be, unless you give money to OpenAI.

Robots won't go get food for your sick, dying friend.

That implies that people who aren't empathetic and/or caring aren't human, which I guess could be argued too, but feels too simplistic.

> Which the LLMs will never be

I'd argue LLMs will never be anything, they're giving you the text you're asking for, nothing more and nothing less. You don't tell them "to be" empathic and caring? Well, they're not gonna appear like that then, but if you do tell them, they'll do their best to emulate that.

A robot could certainly be programmed to get food for a sick, dying friend (I mean, don't drones deliver Uber Eats?) but it will never understand why, or have a phenomenal experience of the act, or have a mental state of performing the act, or have the biological brain state of performing the act, or etc. etc.

Interesting. I wonder why?

Perhaps when we deliver food to our sick friend we subconsciously feel an "atta boy" from our parents who perhaps "trained" us in how to be kind when we were young selfish things.

Obviously if that's all it is we could of course "reinforce" this in AI.

"Never" is a very broad word.