I have zero doubt that transformers can construct sensible models of what they are taught.
My concern about LLM is that there's so little of actual knowledge in human language that it's easily drowned in the rest of the human language and neural network trained on not strictly restricted set of human language has very little chance of modelling knowledge.
In case of Othello game if you teach the neural network to predict all moves you get NN to learn how to play legal moves, not necessarily winning moves.
You'd have to train NN on only the moves of the winning side. Or even create some negative training data and method from the moves of loosing side to have any hopes of creating NN that plays Othello well.
Same should be true for LLMs. To have any hope of getting them to model knowledge you'd have to curate input to strictly represent knowledge and perhaps develop a negative reinforcement training method and feed it with all the language that doesn't represent truth.
Knowledge is doing, language is communicating about it. Think about it this way:
Ask the bot for a cooking recipe. Knowledge would be a cook who has cooked the recipe, evaluated/tasted the result. Then communicated it to you. The bot gives you at best a recording of the cook's communication, at worst a generative modification of a combination of such communications, but skipping the cooking and evaluating part.
