> Also, claiming that natural language is infinite, if taken literally, would imply a large and contrary to the common consensus claim about physics, contradicting the Bekenstein bound and all that.
Natural language is infinite in the pretty straightforward sense that, say, chess is infinite (there is an infinite number of valid chess games - if you ignore arbitrary restrictions such as the 50 move rule). This of course doesn’t mean that a chess computer has to be infinitely large or violate any known laws of physics. Similarly, practical C compilers can exist despite their being an infinite number of valid C programs.
Natural language as understandable by humans, is finite, because humans are finite, at least in this life.
There are only finitely many distinct utterances that a person could possibly hear that have length less than one lifetime.
Any utterance which takes longer than a lifetime to make/hear, is not really natural language, so much as an extrapolation of what natural language would/might be if people had unlimited lifespans, memory, and attention-spans.
How would I represent intensions in a neural network?
Well, you can encode text with a sequence of 1-hot vectors. (Is this trivial? Yes. Still counts.) If you can encode intensions on a computer, you can encode it as text. If you can encode it as text, you can encode it as a vector.
Do I think that (a sequence of 1-hot vectors) is the best way to do it? Not really, no. I'd need a bit more detail on what is meant to be represented in order to give a better shot at describing what I think could be a good approach for encoding it.
But also, I don't think the burden of proof is on me here. The author claimed that it is impossible, I said I don't see any justification for that claim.
Personally, I'm not entirely sure what they are saying is impossible. Do they have a particular task in mind?
____
Also, the set of possible states in a game of chess is finite, and a complete understanding of how to play chess optimally would, as such, also be finite. The fact that you can repeat some subset of states in a bunch of orders for an arbitrary amount of time, before you stop messing around and actually finishing the game, doesn't grant chess infinite complexity in any meaningful way.
A finite state machine can recognize a language which has infinitely many strings in it, yes. This does not mean that a RNN can't emulate such an FSM.
Sure, but this is missing the point. There’s also a finite number of C programs that can actually exist in the physical universe, but that’s an arbitrary limit, not part of the definition of C. Similarly, there’s no non-arbitrary limit on, say, the length of an English sentence.
All of this ‘debate’ about the infinity of language just reduces to a misunderstanding of what people are saying, as far as I can see. No-one thinks that more than a finite number of English sentences will ever be uttered; conversely, no-one thinks that we will ever discover such a thing as a complete list of all English sentences (since trivially the conjunction of all these sentences would be a new sentence not on the original list).
Note that if you view English as a regular tree language, your point about chess also applies to English. You don’t need to remember all of the preceding state, just the congruence class. No-one is saying that English is “infinitely complex”. The grammar of English is finite, but there is no limit on the number of sentences that it can assign a structure to.
Of course you can encode intensions using numbers in the broad sense that you can encode pretty much anything using numbers.
Natural language is infinite in the pretty straightforward sense that, say, chess is infinite (there is an infinite number of valid chess games - if you ignore arbitrary restrictions such as the 50 move rule). This of course doesn’t mean that a chess computer has to be infinitely large or violate any known laws of physics. Similarly, practical C compilers can exist despite their being an infinite number of valid C programs.