I disagree. Claude may fail at running a vending machine business but I have used it to read 10k reports and found it to be really good. There is a wealth of information in public filings that is legally required to be accurate but is often obfuscated in footnotes. I had an accounting professor that used to say the secret was reading (and understanding) the footnotes.
That’s a huge pain in the neck if you want to compare companies, worse if they are in different regulatory regimes. That’s the kind of thing I have found LLMs to be really good for.
This is why it’s important to follow the studies comparing LLMs’ performance in “needle-in-a-haystack” style tasks. They tend to be pretty good at finding the one thing wrong in a large corpus of text, though it depends on the LLM, the flavor (Sonnet, Opus, 8B, 27B, etc) and the size of the corpus, and there are occasional performance cliffs.
Did you go and look at the correctness of the information?
Because I have seen Claude, as recently as a week ago, completely inventing and citing whole non existent paragraphs from the documentation of some software I know well. I only because of that, I was able to notice...
All models hallucinate. The likelihood of hallucinations are however strongly influenced by the way you prompt and construct your context.
But even if a human went through the documents by hand and tried to make the analysis, they're still likely to make mistakes. That's why we usually define the scientific method as making falsifiable claims, which you then try to disprove in order to make sure they're correct.
And if you can't do that, then you're always walking on thin ice, whatever tool or methodology you choose to use for the analysis.
Are you serious unaware what a RAG is and still speak with authority on the topic?
It's automatically retrieving information and adding it to the context.
It's -in spirit- a convenience function so you don't have to manually provide it during the prompt. It's just a lot harder to pull off well automatically, but the fundamental practice is "just" context optimization
You're essentially saying "but that's not driving!!!!" After someone goes by in an EV, because it's ain't an ICE
Did you made any technical argument on how to reduce hallucinations? Because I fail to see one from you on this thread except: "it's the fault of your prompt""
I haven't tried SuperHeavy, but why would it? all transformer based LLM's are pretty prone to hallucinations even with RAG... it can be pretty good I guess
That’s a huge pain in the neck if you want to compare companies, worse if they are in different regulatory regimes. That’s the kind of thing I have found LLMs to be really good for.