So why is not intelligence when I ask an LLM to solve a problem, and it proceeds to review code, search external documentation and write a solution, test it and iterate until the problem is solved? It seems very much like we we do. It seems like the difference to you guys is “hard to define magic that we can’t explain / it just is different”.

So far, it's been pretty easy to find examples of how it is different from human intelligence.

For one, it needs multiple orders of magnitude more tokens (millions? billions? more?) to start following grammar rules: watch any kid pick it up with exposure to maybe 10,000 words by the age of 2 (look how they apply the rule to exceptions and while it does not work, it shows their internalization of the rule).

Next, "how" LLMs work is evident when it barfs in gotcha questions like "if yesterday was Tuesday, what day was yesterday" (I believe all major players have overcome this one in the last 12 months, but you can always find these LLM-twisters: yes, you can trick humans too by distracting them, but these are computers which "don't pay attention"?)

And it can frequently undo an improvement it added earlier, or mess something new up that you did not call out specifically.