I've never had any time for Penrose. Gödel’s theorem "merely" asserts that in any system capable of a specific form of expression there are statements which are true but not provable. What this has to do with (a) limits to computation or (b) human intelligence has never been clear to me, despite four decades or more of interest in the topic. There's no reason I can see why we should think that humans are somehow without computational limits. Whether our limits correspond to Gödel’s theorem or not is mildly interesting, but not really foundational from my perspective.
At the end of the day Penrose's arguments is just Dualism.
Humans have a special thingy that makes the consciousness
Computers do not have the special thingy
Therefore Computers cannot be consciousness.
But Dualism gets you laughed at these days so Dualists have to code their arguments and pretend they aren't into that there Dualism.
Penrose's arguments against AI has always felt to me like special pleading that humans (or to stretch a bit further, carbon based lifeforms) are unique.
While I don't like Penrose's argument and I think it stands on very shaky ground, I very much disagree it's a form of dualism. His argument is simply that human thinking is not reducible to a Turing machine, that it is a form of hyper-computation.
If this were to be true, it would follow that computers as we build them today would fundamentally not be able to match human problem-solving. But it would not follow, in any way, that it would be impossible to build "hyper computers" that do. It just means you wouldn't have any chance of getting there with current technology.
Now, I don't think Penrose's arguments for why he thinks this is the case are very strong. But they're definitely not mystical dualistic arguments, they're completely materialistic mathematical arguments. I think he leans towards an idea that quantum mechanics has a way of making more-than-Turing computation happen (note that this is not about what we call quantum computers, which are fully Turing-equivalent systems, just more efficient for certain problems), and that this is how our brains actually function.
> I think he leans towards an idea that quantum mechanics has a way of making more-than-Turing computation happen (note that this is not about what we call quantum computers, which are fully Turing-equivalent systems, just more efficient for certain problems), and that this is how our brains actually function.
That was my understanding on Penrose's position as well which is just a "Consciousness of the Gaps" argument. As we learn more about quantum operations the space for Consciousness as a special property of humans disapears.
Penrose doesn’t think that consciousness is special to humans. He thinks most animals have it and more importantly to your point, he thinks that there is no reason that we won’t someday construct artificial creations that have it.
I just watched an interview where he made that exact statement nearly word for word.
His only argument is that it is not computable, not that it’s not physical. He does think the physical part involves the collapse of the wave function due to gravity, and that somehow the human brain is interacting with that.
So to produce conciseness in his view, you’d need to construct something capable of interacting with the quantum world the same way he believes organic brains do (or something similar to it). A simulation of the human brain wouldn’t do it.
He proposed a proof of platonism: Mandelbrot set has a stable form and is not subjective, because it doesn't fit in human mind due to its sheer complexity, consequently it exists objectively. His beliefs are pretty transparent.
