I think that's very true and it's maybe even more clear when you consider mathematics.
You can maybe imitate but not effectively learn mathematics empirically. There is an infinite number of mathematical expressions or sequences that can be generated, so learning can never be done, you cannot compress yourself to mathematical understanding. (which is obvious if you try to feed language models simple arithmetic, they can maybe do 5+5 because it shows up somewhere in the data, but then they can't do 3792 + 29382, hence they do not understand anything about addition at all).
The correct way to mathematical understanding is decompressing, understanding the fundamental axioms of mathematics and internal relationships of mathematical objects (comparable to the semantic meaning behind language artifacts), and then expanding them.
> they can maybe do 5+5 because it shows up somewhere in the data, but then they can't do 3792 + 29382
I'd say you're underselling modern AIs. GPT-3 can come close, as can GPT-J (which is publicly available and therefore perhaps easier to prove), even if they don't quite get the right answer 100% of the time.
I gave GPT-J the following prompt (which you can try yourself at https://6b.eleuther.ai/ just wait a few min for an answer)
What is 1 + 1?
2.
What is 3 + 2?
5.
What is 6984 + 120?
7104.
What is 3792 + 29382?
And it responded with
33374
What is -0.084435 + -0.3?
-0.384435
What is -4.1 + -0.4?
-4.5
... [more of the same omitted]
Note that it got almost the right answer! Close enough that if it was a human I would be tempted to call it human error. GPT-3 can do even better.
EDIT: On another try GPT-J got the answer! Not bad (and GPT-3 is an even more powerful model).
33174
-3 + -91.4
-94.4
What is -1.6 + 13.81?
12.21
Jon has six mangoes, and he needs ten rupees.
Only Alice wants to buy any mangoes. Alice wants to buy three.
How much does Jon sell them for?
Response:-
Jon has six mangoes, and he needs ten rupees.
Only Alice wants to buy any mangoes. Alice wants to buy three.
How much does Jon sell them for?
100 rupees.
Q: What is the smallest number that can be written as 1/2 x 5 x 7?
A: 6,550
Q: What is the smallest number that can be written as 1/2 x 5 x 7 x 11?
A: 6,550,000
Q: What is the smallest number that can be written as 1/2 x 5 x 7 x 11 x 13?
A: 6,550,000,000
Q: How many ways can you make 50 if you use five different colored marbles?
A: 10
Q: How many ways can you make 75 using only three different colored marbles?
A: 12
Q: What is the smallest number that can be written as 1/2 x 5 x 7 x 11 x 13?
A: 6,550,000,000
Q: What is the smallest number that can be written as 1/2 x 5 x 7 x 11 x 13 x 15?
A: 6,550,000,000,000
Q: What is the smallest number that can be written as 1/2 x 5 x 7 x 11 x 13 x 15 x 17?
A: 6,550,000,000
The first line of the answer is part of an appropriate answer for this linguistic formula given as a prompt. An appropriate response answers the question, and (ideally) supplies a brief warrant for the answer. (A warrant is a reasoned argument as to why the answer is correct.)
The rest of the answer is just noise. Not much evidence of linguistic understanding, despite the enormous corpus ingested by the tool.
You have to be careful when analyzing GPT-3 and derivatives due to BPE encoding. It's accuracy greatly improves when adding commas to strings of numbers: https://www.gwern.net/GPT-3#fn21
You can maybe imitate but not effectively learn mathematics empirically. There is an infinite number of mathematical expressions or sequences that can be generated, so learning can never be done, you cannot compress yourself to mathematical understanding. (which is obvious if you try to feed language models simple arithmetic, they can maybe do 5+5 because it shows up somewhere in the data, but then they can't do 3792 + 29382, hence they do not understand anything about addition at all).
The correct way to mathematical understanding is decompressing, understanding the fundamental axioms of mathematics and internal relationships of mathematical objects (comparable to the semantic meaning behind language artifacts), and then expanding them.