Yea it was a wake up call for me when I was asking about the volume of a cube vs a dodecahedron and Claude+ hit me with "The diameter of a dodecahedron passes through 3 pentagonal faces", just no ability to reason about geometry.
Sidenote, poe has a bug that mis-reports this as a conversation with Claude-2-100k, but the conversation took place on March 24, about a week after Claude+ was made public. Can't even rely on the portals to be truthful about what model was used.
"Reasoning" about faculties which the bot does not possess (visual, audial, or individual character domains) is simply not going to happen until we hook it up to some symbolic model, or the LLM becomes advanced enough to infer a world model from text alone.
Until then, we're basically asking a blind & deaf guy, who happens to be very well-read, to reason about senses he doesn't have.
Though the mistake in your example does seem kind of egregious and I'd be curious to see whether GPT-4 would make similar mistakes.
https://poe.com/lookaroundyou/1512927999895666
Sidenote, poe has a bug that mis-reports this as a conversation with Claude-2-100k, but the conversation took place on March 24, about a week after Claude+ was made public. Can't even rely on the portals to be truthful about what model was used.