Except that we currently have no ability at all to make sure these models tell the truth? You could ask a LLM about a historical event and get weird BS as an answer. I think Facebook had a serious problem with its science language model a few months ago? Great teachers don’t scale is true, but LLM’s are not even minimally competent teachers.
LLMs in a simple chat interface where you rely on the training data to provide the answer, are indeed unable to provide reliable answers.
LLMs hooked together with search engines, fact repositories (Wikipedia, etc) and other agent tools (for math, theorem proving, code checking, etc) can provide not only a reliable answer, but also a research plan it followed to provide it, and details on the fact-checking, and then discuss it with you.
