Going through this, it has the old "A full understanding of an utterance or a question requires understanding the one and only one thought that a speaker is trying to convey. " claim, which continues to not make any sense, because obviously people don't do that; as much as I would like to be understood in precisely the way I mean, down to the most subtle nuance/shade of meaning and connotation, at least much of the time, this is not something we can actually get across in an at all reasonable amount of time.
Also, claiming that natural language is infinite, if taken literally, would imply a large and contrary to the common consensus claim about physics, contradicting the Bekenstein bound and all that.
But one thing which seemed, at least initially, like a point that could have some merit, was the point about compression vs decompression.
But the alleged syllogism about it, is, pretending to be much more formal/rigorous than it is, and is also kind of nonsense? Or, like, it conflates "NLU is about decompression" with, "NLU \equiv not COMP" which I assume is meant to mean, --
well, actually, I'm not sure what it is supposed to mean. Initially I thought it was supposed to mean "NLU is nonequivalent to compression", but if so, it should be written as like, "NLU \not\equiv COMP" (where \not\equiv is the struckthrough version of the \equiv symbol) , but if it is supposed to mean "NLU is equivalent to the inverse or opposite of compression" (which I suppose better fits the text description on the right better), then I don't think "not" is the appropriate way to express that.
And, if by "not" the author really means "the inverse of", then, well, there's nothing wrong with something being equivalent to its own inverse!
Nor, does something being equivalent to the inverse of something else imply that it is "incompatible" with it.
For something talking about communicating ideas and these ideas being understood precisely by the recipient of the message, the author sure did not work to communicate precisely.
The value in formalization comes not in its trappings, but in actually being careful and precise, etc., not merely pretending to be.
The part on intensional equality vs extensional equality was interesting, but the claim that neural networks cannot represent intension is, afaict, not given any justification (other than just "because they are numeric").
> Also, claiming that natural language is infinite, if taken literally, would imply a large and contrary to the common consensus claim about physics, contradicting the Bekenstein bound and all that.
Natural language is infinite in the pretty straightforward sense that, say, chess is infinite (there is an infinite number of valid chess games - if you ignore arbitrary restrictions such as the 50 move rule). This of course doesn’t mean that a chess computer has to be infinitely large or violate any known laws of physics. Similarly, practical C compilers can exist despite their being an infinite number of valid C programs.
Natural language as understandable by humans, is finite, because humans are finite, at least in this life.
There are only finitely many distinct utterances that a person could possibly hear that have length less than one lifetime.
Any utterance which takes longer than a lifetime to make/hear, is not really natural language, so much as an extrapolation of what natural language would/might be if people had unlimited lifespans, memory, and attention-spans.
How would I represent intensions in a neural network?
Well, you can encode text with a sequence of 1-hot vectors. (Is this trivial? Yes. Still counts.) If you can encode intensions on a computer, you can encode it as text. If you can encode it as text, you can encode it as a vector.
Do I think that (a sequence of 1-hot vectors) is the best way to do it? Not really, no. I'd need a bit more detail on what is meant to be represented in order to give a better shot at describing what I think could be a good approach for encoding it.
But also, I don't think the burden of proof is on me here. The author claimed that it is impossible, I said I don't see any justification for that claim.
Personally, I'm not entirely sure what they are saying is impossible. Do they have a particular task in mind?
____
Also, the set of possible states in a game of chess is finite, and a complete understanding of how to play chess optimally would, as such, also be finite. The fact that you can repeat some subset of states in a bunch of orders for an arbitrary amount of time, before you stop messing around and actually finishing the game, doesn't grant chess infinite complexity in any meaningful way.
A finite state machine can recognize a language which has infinitely many strings in it, yes. This does not mean that a RNN can't emulate such an FSM.
Sure, but this is missing the point. There’s also a finite number of C programs that can actually exist in the physical universe, but that’s an arbitrary limit, not part of the definition of C. Similarly, there’s no non-arbitrary limit on, say, the length of an English sentence.
All of this ‘debate’ about the infinity of language just reduces to a misunderstanding of what people are saying, as far as I can see. No-one thinks that more than a finite number of English sentences will ever be uttered; conversely, no-one thinks that we will ever discover such a thing as a complete list of all English sentences (since trivially the conjunction of all these sentences would be a new sentence not on the original list).
Note that if you view English as a regular tree language, your point about chess also applies to English. You don’t need to remember all of the preceding state, just the congruence class. No-one is saying that English is “infinitely complex”. The grammar of English is finite, but there is no limit on the number of sentences that it can assign a structure to.
Of course you can encode intensions using numbers in the broad sense that you can encode pretty much anything using numbers.
Also, claiming that natural language is infinite, if taken literally, would imply a large and contrary to the common consensus claim about physics, contradicting the Bekenstein bound and all that.
But one thing which seemed, at least initially, like a point that could have some merit, was the point about compression vs decompression.
But the alleged syllogism about it, is, pretending to be much more formal/rigorous than it is, and is also kind of nonsense? Or, like, it conflates "NLU is about decompression" with, "NLU \equiv not COMP" which I assume is meant to mean, -- well, actually, I'm not sure what it is supposed to mean. Initially I thought it was supposed to mean "NLU is nonequivalent to compression", but if so, it should be written as like, "NLU \not\equiv COMP" (where \not\equiv is the struckthrough version of the \equiv symbol) , but if it is supposed to mean "NLU is equivalent to the inverse or opposite of compression" (which I suppose better fits the text description on the right better), then I don't think "not" is the appropriate way to express that. And, if by "not" the author really means "the inverse of", then, well, there's nothing wrong with something being equivalent to its own inverse! Nor, does something being equivalent to the inverse of something else imply that it is "incompatible" with it.
For something talking about communicating ideas and these ideas being understood precisely by the recipient of the message, the author sure did not work to communicate precisely.
The value in formalization comes not in its trappings, but in actually being careful and precise, etc., not merely pretending to be.
The part on intensional equality vs extensional equality was interesting, but the claim that neural networks cannot represent intension is, afaict, not given any justification (other than just "because they are numeric").