I wonder if this is an example of "it works in practice but the important question is whether it works in theory."
Perhaps Penrose is right about the nature of intelligence and the fact that computers cannot ever achieve that (for some tight definition of the term). But in a practical sense, these LLMs that are popular are doing things that we generally considered "intelligent". Perhaps it's faking it well but it's faking it well enough to be useful and that's what people will use. Not the theoretical definition.
Perhaps Penrose is right about the nature of intelligence and the fact that computers cannot ever achieve that (for some tight definition of the term). But in a practical sense, these LLMs that are popular are doing things that we generally considered "intelligent". Perhaps it's faking it well but it's faking it well enough to be useful and that's what people will use. Not the theoretical definition.