An LLM is not a conscious entity, and while it can babble and assemble meaningful sentences, it has no guarantee of correctness, a baseline honesty, and other small bits and pieces we value in a conversation. It's just a sentence builder.
As a result, it can't replace a real human being backed by real experience and thoughts, hence it can't be as useful. I converse with myself, too, but it's not the same either with a different human being or an artificial construct which can babble meaningful sentences.
Currently, we simply ignore or don't understand the fact that the meaning of a sentence is not solely built with word order. There are higher orders of information carried implicitly, and they are not articulated in words. Hence an LLM can not replace, even mimic a real human being in a conversation.
> no guarantee of correctness, a baseline honesty, and other small bits and pieces
Humans for sure do not guarantee this either.
I think of LLMs as an amazing rubber duck. It's heard of everything, and it always responds with something that sounds like it came from the same sphere. You have to use your own mind to figure out if it's meaningful, but this is not so different from conversing with a person. People can babble too.
> There are higher orders of information carried implicitly
You're not always looking for those in a conversation. Sometimes you really are just checking that you've thought things through. Like if someone asks you what the arguments in favor of democracy are, you want a list of points so you can check you haven't forgotten something.
However, humans are more nuanced than that. One might remember wrongly, or act in bad-faith. This is why I said "baseline honesty". One people's traits definitely affects how their words are perceived, and this is not carried in the sentence itself.
> I think of LLMs as an amazing rubber duck. ... People can babble too. (Snipped for brevity)
Humans are not as random as an LLM.
> You're not always looking for those in a conversation. Sometimes you really are just checking that you've thought things through.
Again the person you're asking or answering has their own character and their or your words are affected by that implicit knowledge. This is a background process we're not aware of unless you dig into yourself and look for it.
> Humans you think you're conversing with online, not so much.
Two decades of experience shows me otherwise. Even your two comments shows a consistent tone. We're just started to discuss, yet I have started to build an image of you thanks to your comments.
This is one of the mechanisms we don't fully understand and don't dare to dig much, because tinkering with people's minds are dangerous.
But it's a pretty amazing rubber duck that has read the entire internet and thus can correlate your ideas with the ideas of all of humanity in a split second on the spot.
Whilst by no means a new idea, physical and digital zettelkasten may interest some folks more than conversing with an LLM. Some people describe it as conversing with yourself or having a second brain. I don't find discourse with an LLM to be productive, but I do find surfing my zettelkasten to yield new and novel ideas, especially when it comes to problem-solving and research.
However, I'm still on industry forums and communities like HN to hopefully have new and conflicting thoughts thrown my way by peers. That's my primary concern with LLMs, a lack of fresh perspective that has all the nuance of experience and understanding behind its output.
If convincingly arguing about something was the only thing we needed for proving correctness, we wouldn't have the scientific method.
Or, if word order built the meaning and it's devoid of the character itself, we wouldn't be praising authors for their character building skills and embedding things not written or spoken into their stories, regardless of the medium it's presented.
To apply the scientific method and prove correctness, you need to define the subject first. Meaning is ill-defined, just like consciousness, intelligence, and others.
LLMs can clearly grasp higher order abstractions and concepts just by reordering the words. In fact, embeddings (developed much earlier) are specifically intended to represent the semantical meaning extracted from text using just statistical methods; until the introduction of transformers they lacked a good architecture to demonstrate the usefulness of that.
This makes many people argue that being a sentence builder is enough to be intelligent, as they also received most of their intelligence from the same source (concentrated experience of someone else - social intelligence).
"Honesty" and other forms of self-introspection is just a high level construct which current models aren't trained for, and likely not a fundamental issue with being a stochastic parrot.
I agree. Instance from today; I have a very novel task at work that nobody seems to know how to approach, I had a hunch that metaheuristics could play a role; I used ChatGPT to help me better formulate the problem and at least now there is a way to tackle the problem.
It made a few errors in at least presenting my ideas, but these errors were a consequence of my misunderstanding and lack of clarity.
The errors themselves, in my opinion, are invaluable because they force you to think, to be clear, and to guide the model into giving me useful ideas to look into.
For all intents and purposes, it was an example of the Socratic method, albeit inverted, where the student is asking questions to an all knowing teacher, and the teacher / LLM responds with ideas and hints. Ultimately, it's up to the student to synthesize the solution, be critical of the data, and tie everything together.
