> We (the authors of this website) have at times sought insight into the inner workings of an LLM by asking it “why did you just do that?”
> But the LLM can’t tell us. It’s not a person. It doesn’t have the metacognitive abilities necessary to reflect on its past actions and report the motivations underlying them*.
> With no clue why it did whatever it just did, the LLM is forced to guess wildly at a plausible explanation, like the ill-fated Leonard Shelby in Christopher Nolan's film Memento.
> And we, gullible humans that we are, often believe its bullshit.
---
I am almost convinced that we ourselves are a narrator riding along inside an animal's mind, trying desperately to put together explanations for our actions, mostly just to convince others, just as though an LLM were running on our own senses trying to portray some deep semblance of consciousness. I don't think we'll find a super smart AI, we'll just realize we were not very sophisticated all along. The power of speech for information and culture transfer, writing, inspiration, and coordination is just awe inspiring, evolutionary speaking, so once we could talk we had to, because the "better" talker almost always won. It's an arms race.
Related, since the advent of LLMs I've become acutely aware how any argument of a considerable length with another person quickly starts meandering and how topics change seemingly of no one's volition - almost as if our own internal token limit has been exceeded.
Tangentially related, I became aware how my (in)ability to reason 3 intertwined different programming languages across different files can be conveniently called as my own "context window". (the example here is HTML/CSS/JS where LLM greatly exceeds my own capacity).
> I am almost convinced that we ourselves are a narrator riding along inside an animal's mind
I think this is one of those "some truth, but not the whole truth" things. Yes, we trick ourselves, such as with mis-remembered reflex actions: "I felt it, it hurt, therefore I decided to move", even though the nerve-impulse speeds means your limb was moving before your brain even knew about the pain.
But the "narration" seems to be very important. We create stories to capture cause-and-effect about the world (unclear how much that requires language) and it seems to be beneficially adaptive. In fact this drive is so important that we do it even when we abstractly know it's wrong, like when flipping 50/50 coins and imagining a particular coin is luckier than another, or that you're on a "hot streak", or "now that other outcome is overdue."
> I am almost convinced that we ourselves are a narrator riding along inside an animal's mind, trying desperately to put together explanations for our action
This is more or less the position of Daniel Dennett.
Also, it’s the premise of the one of Greg Egan’s (IMO) best short stories, Mr. Volition.
I am completely convinced that what we call consciousness is as you say.
this means that it really exists in retrospect (20ms ? i recall some neuroscience articles from the 00s). nonetheless its whole reason for existing (retrospectively) is planning the future
> But the LLM can’t tell us. It’s not a person. It doesn’t have the metacognitive abilities necessary to reflect on its past actions and report the motivations underlying them*.
> With no clue why it did whatever it just did, the LLM is forced to guess wildly at a plausible explanation, like the ill-fated Leonard Shelby in Christopher Nolan's film Memento.
> And we, gullible humans that we are, often believe its bullshit.
---
I am almost convinced that we ourselves are a narrator riding along inside an animal's mind, trying desperately to put together explanations for our actions, mostly just to convince others, just as though an LLM were running on our own senses trying to portray some deep semblance of consciousness. I don't think we'll find a super smart AI, we'll just realize we were not very sophisticated all along. The power of speech for information and culture transfer, writing, inspiration, and coordination is just awe inspiring, evolutionary speaking, so once we could talk we had to, because the "better" talker almost always won. It's an arms race.