> There's a reason it can give a chess move when prompted and explain all the rules and notation, but can't actually play chess: it's not in the training data.

I don't understand how you can claim an LLM can't play chess. Just as one example, see: https://dynomight.net/chess/