GPT can give a single Yes/No answer that indicates a fair amount of intelligence for the right question. No iteration there. Just a single pass through the network. Hofstadter is surprised by this.
Well, no, it can produce a probability distribution over all possible tokens, among which ‘yes’ or ‘oui’ or ‘はい’ or ‘totally’ or ‘no’ or ‘nein’ or the beginning of ‘as a large language model I am unable to answer that question’ are all represented. Which is either more or less impressive than just being able to answer ‘yes or no’ depending on your priors I guess.
There’s maybe an interesting philosophical question of perspective there because if you think of the GPT as answering the question ‘if you had just read this, what token would you expect to read next?’ That doesn’t seem like a question that necessarily requires ‘intelligence’ so much as ‘data’. It’s just a classification problem and we’ve been throwing NNs at that for years.
But if you ask the question ‘if you had just written this, what token would you expect to output next?’ It feels like the answer would require intelligence.
The point of my comment is that even the distribution represents intelligence. If you give it a tricky Yes/No question that results in a distribution that's 99.97% "Yes" and negligible values for every other token, that is interesting. Hofstadter is surprised you can do any amount of non-trivial reasoning in a single forward pass.
