You can copy an existing book, word for word (maybe not even that if it uses a different character set, unless you're doing photocopies).

Write a new book without understanding the language? No way - not one that makes any sense. Not unless you're going with the "million monkeys" approach (and if you did try that approach, you wouldn't live long enough to succeed in writing one actual coherent book in the foreign language).

So, we could think about trying to simulate a human brain, neuron by neuron. That's the "making a photocopy" approach. But that's not the approach we're pursuing. We're trying to write a new book (create a new, non-human intelligence) in a language we don't understand (that is, not knowing what intelligence/consciousness actually is).

(Side topic: What would happen if you asked GPT-4 to write a full-length novel? Or even a story as long as the token limit? Would you get anything coherent?)

As a tangent on this, it'd be such an interesting experiment to see how far one could go in deciphering/understanding a new language and attempting to write a new book in that language based on the content of a single, probably fairly long, book.

It feels like it should be theoretically possible, but I doubt it's ever been tried.

Maybe something like understanding aincent languages from limited, fragmented sources is the closest natural experiment we have in practice to have tested it, but it's hardly the same as a full, substantial text in a consistent style.

Of course you can. When the brits first met the Chinese, the Chinese didn’t speak a word of English and the British not a word of Chinese.

The Chinese room thought experiment is flawed; this is exactly how we learn languages!

Congratulations, you have just invented large language models.

IOW you are hardly conscious at all of what the book is about.

That’s not because it’s some deep philosophical problem, but because people refuse to define what they mean by consciousness.

Do they mean memory? Attention? Awareness? Self awareness? The inner voice? Qualia? All of these can be explained somewhat, but they keep moving the goalpost

I would say it's not because people refuse to define, it's that science has not defined (and cannot define it)