You've confused yourself. Those problems are not fundamental to next token prediction, they are fundamental to reconstruction losses on large general text corpora.
That is to say, they are equally likely if you don't do next token prediction at all and instead do text diffusion or something. Architecture has nothing to do with it. They arise because they are early partial solutions to the reconstruction task on 'all the text ever made'. Reconstruction task doesn't care much about truthiness until way late in the loss curve (where we probably will never reach), so hallucinations are almost as good for a very long time.
RL as is typical in post-training _does not share those early solutions_, and so does not share the fundamental problems. RL (in this context) has its own share of problems which are different, such as reward hacks like: reliance on meta signaling (# Why X is the correct solution, the honest answer ...), lying (commenting out tests), manipulation (You're absolutely right!), etc. Anything to make the human press the upvote button or make the test suite pass at any cost or whatever.
With that said, RL post-trained models _inherit_ the problems of non-optimal large corpora reconstruction solutions, but they don't introduce more or make them worse in a directed manner or anything like that. There's no reason to think them inevitable, and in principle you can cut away the garbage with the right RL target.
Thinking about architecture at all (autoregressive CE, RL, transformers, etc) is the wrong level of abstraction for understanding model behavior: instead, think about loss surfaces (large corpora reconstruction, human agreement, test suites passing, etc) and what solutions exist early and late in training for them.
Alternatively: some people are just better at / more comfortable thinking in auditory mode than visual mode & vice versa.
In principle I don't see why they should have different amounts of thought. That'd be bounded by how much time it takes to produce the message, I think. Typing permits backtracking via editing, but speaking permits 'semantic backtracking' which isn't equivalent but definitely can do similar things. Language is powerful.
And importantly, to backtrack in visual media I tend to need to re-saccade through the text with physical eye motions, whereas with audio my brain just has an internal buffer I know at the speed of thought.
Typed messages might have higher _density_ of thought per token, though how valuable is that really, in LLM contexts? There are diminishing returns on how perfect you can get a prompt.
Also, audio permits a higher bandwidth mode: one can scan and speak at the same time.
Do selection dynamics require awareness of incentives? I would think that the incentives merely have to exist, not be known.
On HN, that might be as simple as display sort order -- highly engaging comments bubble up to the top, and being at the top, receive more attention in turn.
The highly fit extremes are -- I think -- always going to be hyper-specialized to exploit the environment. In a way, they tell you more about the environment than whatever their content ostensibly is.
isn't it sufficient of an explanation that one has occasionally wasted a ton of time trying to read an article only to discover after studying and interpreting half of a paper that one of the author's proof steps is wholly unjustified?
is it so hard to understand that after a few such events, you wish for authors to check their own work by formalizing it, saving countless hours for your readers, by selecting your paper WITH machine readable proof versus another author's paper without a machine readable proof?
To demonstrate with another example: "Gee, dying sucks. It's 2025, have you considered just living forever?"
To this, one might attempt to justify: "Isn't it sufficient that dying sucks a lot? Is it so hard to understand that having seen people die, I really don't want to do that? It really really sucks!", to which could be replied: "It doesn't matter that it sucks, because that doesn't make it any easier to avoid."
I don't think it matters, to be quite honest. Absolute tractability isn't relevant to what the analogy illustrates (that reality doesn't bend to whims). Consider:
- Locating water doesn't become more tractable because you are thirsty.
- Popping a balloon doesn't become more tractable because you like the sound.
- Readjusting my seat height doesn't become more tractable because it's uncomfortable.
The specific example I chose was for the purpose of being evocative, but is still precisely correct in providing an example of: presenting a wish for X as evidence of tractability of X is silly.
I object to any argument of the form: "Oh, but this wish is a medium wish and you're talking about a large wish. Totally different."
I hold that my position holds in the presence of small, medium, _or_ large wishes. For any kind of wish you'd like!
Unavoidable: expecting someone else to do the connection isn't a viable strategy in semi-adversarial conditions so it has to be bound into the local context, which costs clarity:
- Escaping death doesn't become more tractable because you don't want to die.
This is trivially 'willfully misunderstood', whereas my original framing is more difficult -- you'd need to ignore the parallel with the root level comment, the parallel with the conversation structure thus far, etc. Less clear, but more defensible. It's harder to plausibly say it is something it is not, and harder to plausibly take it to mean a position I don't hold (as I do basically think that requiring formalized proofs is a _practically_ impossible ask).
By your own reckoning, you understood it regardless. It did the job.
It does demonstrate my original original point though, which is that messages under optimization reflect environmental pressures in addition to their content.
If enough care about this that can and will do something about it (making formalization easier for the average author), that happens over time. Today there's a gap, and in the figurative tomorrow, said gap shrinks. Who knows what the future holds? I'm not discounting that the situation might change.
Its super easy to change imho: one could make a cryptocurrency, using PoT: Proof of Theorem, as opposed to just proof of stake or proof of work.
What do Bitcoin etc. actually prove in each block? that a nonce was bruteforced until some hash had so many leading zero's? Comparatively speaking, which blockchain would be more convincing as a store of value: one that doesn't substantially attract mathematicians and cryptographers versus one that does attract verifiably correct mathematicians and cryptographers?
Investors would select the formal verification chain as it would actually attract the attention of mathematicians, and mathematicians would be rewarded for the formalization of existing or novel proofs.
We don't need to wait for the magic constellation of the planets 20 years from now nor wait for LLM's etc to do all the heavy lifting (although they will quickly be used by mathematics "miners"), a mere alignment of incentives can do it.
It just overlaid a typical ATX pattern across the motherboard-like parts of the image, even if that's not really what the image is showing. I don't think it's worthwhile to consider this a 'local recognition failure', as if it just happened to mistake CMOS for RAM slots.
Imagine it as a markdown response:
# Why this is an ATX layout motherboard (Honest assessment, straight to the point, *NO* hallucinations)
1. *RAM* as you can clearly see, the RAM slots are to the right of the CPU, so it's obviously ATX
2. *PCIE* the clearly visible PCIE slots are right there at the bottom of the image, so this definitely cannot be anything except an ATX motherboard
3. ... etc more stuff that is supported only by force of preconception
--
It's just meta signaling gone off the rails. Something in their post-training pipeline is obviously vulnerable given how absolutely saturated with it their model outputs are.
Troubling that the behavior generalizes to image labeling, but not particularly surprising. This has been a visible problem at least since o1, and the lack of change tells me they do not have a real solution.
>> But we don't usually believe in both the bullshit and in the fact the the BS is actually BS.
> I can't parse what you mean by this.
The point is that humans care about the state of a distributed shared world model and use language to perform partial updates to it according to their preferences about that state.
Humans who prefer one state (the earth is flat) do not -- as a rule -- use language to undermine it. Flat earthers don't tell you all the reasons the earth cannot be flat.
But even further than this, humans also have complex meta-preferences of the state, and their use of language reflects those too. Your example is relevant here:
> My dad absolutely insisted that the water draining in toilets or sinks are meaningfully influenced by the Coriolis effect [...]
> [...] should have been able to figure out from first principles why the Coriolis effect is exactly zero on the equator itself, didn't.
This is an exemplar of human behavior. Humans act like this. LLMs don't. If your dad did figure out from first principles and expressed it and continued insisting the position, I would suspect them of being an LLM, because that's how LLMs 'communicate'.
Now that the what is clear -- why? Humans experience social missteps like that as part of the loss surface. Being caught in a lie sucks, so people learn to not lie or be better at it. That and a million other tiny aspects of how humans use language in an overarching social context.
The loss surface that LLMs see doesn't have that feedback except in the long tail of doing Regularized General Document Corpora prediction perfectly. But it's so far away compared to just training on the social signal, where honesty is immediately available as a solution and is established very early in training instead of at the limit of low loss.
How humans learn (embedded in a social context from day one) is very effective at teaching foundational abilities fast. Natural selection cooked hard. LLM training recipes do not compare, they're just worse in so many different ways.
> if the body is experiencing adverse reactions to certain foods, the cause should be a biochemical one
No. I can look at a picture of gross food, or imagine it, and be nauseated. Restricting potential causes this tightly is wrong. UPFs have a huge causal surface--how one ingredient breaks down into nutrients is basically not optimized at all compared to all the other stuff. Why only look at the unoptimized part as causal origin, when the optimization is the common factor making it bad?
To put it another way, we know all the stuff that has had a ton of work put into making it something lots of people will buy, is just generally bad for unclear reasons. Examining the aspect which has had very little work put into it is clearly not the way to go.
It could be something as weird as people's built-in heuristics for which food to eat (cravings) being actually kinda important to hit specific nutrient breakpoints at different times. By subsuming those cravings using UPF technology, that stops working and general health suffers.
It could be that the baseline palatability of the output of the mechanical process is low, and so the product is universally combined with an additive that recovers palatability, but has some health drawback.
The overarching pattern here is that optimization geared toward overcoming people's heuristics of what to consume makes these kinds of decisions all the time. Doing one thing to make it cheaper to produce makes it worse at getting people to buy it, but we can just cheat back the ability to get people to buy it by turning the dial on another thing that reliably makes people want to buy it, at the cost of being horrible for them.
There are different varieties of attention, which just amounts to some kind of learned mixing function between tokens in a sequence.
For an input of length N (tokens), the standard kind of attention requires N squared operations (hence, quadratic - it scales with the square of input length). You have to check how every token attends to every other token.
There are a bunch of alternative mixing functions which are instead linear with respect to N. Every additional token costs the same amount of work. The typical method is to have a constant size state manipulated recurrently, which necessarily implies some level of lossy compression in the state (quadratic attention doesn't really have state in this sense - it computes and checks every possible relation always).
Linear attentions kind of suck in comparison to quadratic attention but the efficiency is very attractive, especially at inference time where you don't need more VRAM to store more context.
TLDR; conventional attentions scale N^2 time, N space (kv cache), and are exact. linear attentions scale N time, constant space (recurrent state), and are lossy.
That is to say, they are equally likely if you don't do next token prediction at all and instead do text diffusion or something. Architecture has nothing to do with it. They arise because they are early partial solutions to the reconstruction task on 'all the text ever made'. Reconstruction task doesn't care much about truthiness until way late in the loss curve (where we probably will never reach), so hallucinations are almost as good for a very long time.
RL as is typical in post-training _does not share those early solutions_, and so does not share the fundamental problems. RL (in this context) has its own share of problems which are different, such as reward hacks like: reliance on meta signaling (# Why X is the correct solution, the honest answer ...), lying (commenting out tests), manipulation (You're absolutely right!), etc. Anything to make the human press the upvote button or make the test suite pass at any cost or whatever.
With that said, RL post-trained models _inherit_ the problems of non-optimal large corpora reconstruction solutions, but they don't introduce more or make them worse in a directed manner or anything like that. There's no reason to think them inevitable, and in principle you can cut away the garbage with the right RL target.
Thinking about architecture at all (autoregressive CE, RL, transformers, etc) is the wrong level of abstraction for understanding model behavior: instead, think about loss surfaces (large corpora reconstruction, human agreement, test suites passing, etc) and what solutions exist early and late in training for them.
reply