A significant fraction of ML-adjacent people think it can do more.
I'll say that it's more than just words. LLMs can learn patterns, and patterns of patterns, recursively to a degree. They can represent real knowledge about the real world to the degree that this is revealed through the text they train on. This means LLMs can make inferences based on similarities, sometimes similarities at a surprisingly abstract level. And reasoning at the basic logical step by step can of course be done, since that can be reduced to textual pattern matching and string substitution.
But LLMs have no computational space to, for example, read about the description of a novel computation, and then perform the computation without using generated text as a scratchpad, if the computation physically takes more steps than are available in its feedforward stack. It would need to call out to a subsystem in that case. And callable subsystems are ripe for abuse through confused deputy - LLMs are not reliable deputies.
There's a lot of people, text-oriented people, who mistake authorial voice for animus. To me this is like mistaking a CGI animation for a real person behind frosted glass. Text is a low bandwidth medium and it relies on the reader bringing their own mental model to the party. So a machine which produces convincing text has a high leverage tool to seem more capable than it is.
In a sense, LLMs- particularly "conversation-shaped" LLMs like ChatGPT- harvest the benefit of the doubt we are all, as readers, used to providing to text.
For most of our lives, most of the text we have encountered was an intentional communication, self-evident through its own existence. LLMs challenge us with something new: text that has the shape of communication, but no intent.
The proliferation of generative "AI" for text will profoundly alter the human relationship to the printed word, and perhaps ultimately dispell that benefit of the doubt.
I'll say that it's more than just words. LLMs can learn patterns, and patterns of patterns, recursively to a degree. They can represent real knowledge about the real world to the degree that this is revealed through the text they train on. This means LLMs can make inferences based on similarities, sometimes similarities at a surprisingly abstract level. And reasoning at the basic logical step by step can of course be done, since that can be reduced to textual pattern matching and string substitution.
But LLMs have no computational space to, for example, read about the description of a novel computation, and then perform the computation without using generated text as a scratchpad, if the computation physically takes more steps than are available in its feedforward stack. It would need to call out to a subsystem in that case. And callable subsystems are ripe for abuse through confused deputy - LLMs are not reliable deputies.
There's a lot of people, text-oriented people, who mistake authorial voice for animus. To me this is like mistaking a CGI animation for a real person behind frosted glass. Text is a low bandwidth medium and it relies on the reader bringing their own mental model to the party. So a machine which produces convincing text has a high leverage tool to seem more capable than it is.