I agree with the substance, but would argue the author fails to "understand how AI works" in an important way:
LLMs are impressive probability gadgets that have been fed nearly the entire internet, and produce writing not by thinking but by making statistically informed guesses about which lexical item is likely to follow another
Modern chat-tuned LLMs are not simply statistical models trained on web scale datasets. They are essentially fuzzy stores of (primarily third world) labeling effort. The response patterns they give are painstakingly and at massive scale tuned into them by data labelers. The emotional skill mentioned in the article is outsourced employees writing or giving feedback on emotional responses.
So you're not so much talking to statistical model as having a conversation with a Kenyan data labeler, fuzzily adapted through a transformer model to match the topic you've brought up.
While thw distinction doesn't change the substance of the article, it's valuable context and it's important to dispel the idea that training on the internet does this. Such training gives you GPT2. GPT4.5 is efficiently stored low- cost labor.
I don't think those of us who don't work at OpenAI, Google, etc. have enough information to accurately estimate the influence of instruction tuning on the capabilities or the general "feel" of LLMs (it's really a pity that no one releases non-instruction-tuned models anymore).
Personally my inaccurate estimate is much lower than yours. When non-instruction tuned versions of GPT-3 were available, my perception is that most of the abilities and characteristics that we associate with talking to an LLM were already there - just more erratic, e.g., you asked a question and the model might answer or might continue it with another question (which is also a plausible continuation of the provided text). But if it did "choose" to answer, it could do so with comparable accuracy to the instruction-tuned versions.
Instruction tuning made them more predictable, and made them tend to give the responses that humans prefer (e.g. actually answering questions, maybe using answer formats that humans like, etc.), but I doubt it gave them many abilities that weren't already there.
Modern chat-oriented LLMs are not simply statistical models trained on web scale datasets. Instead, they are the result of a two-stage process: first, large-scale pretraining on internet data, and then extensive fine-tuning through human feedback. Much of what makes these models feel responsive, safe, or emotionally intelligent is the outcome of thousands of hours of human annotation, often performed by outsourced data labelers around the world. The emotional skill and nuance attributed to these systems is, in large part, a reflection of the preferences and judgments of these human annotators, not merely the accumulation of web text.
So, when you interact with an advanced LLM, you’re not just engaging with a statistical model, nor are you simply seeing the unfiltered internet regurgitated back to you. Rather, you’re interacting with a system whose responses have been shaped and constrained by large-scale human feedback—sometimes from workers in places like Kenya—generalized through a neural network to handle any topic you bring up.
> and then extensive fine-tuning through human feedback
how extensive is the work involved to take a model that's willing to talk about Tianamen square into one that isn't? What's involved with editing Llama to tell me how to make cocaine/bombs/etc?
It's not so extensive so as to require an army of subcontractors to provide large scale human feedback.
Ya I don’t think I’ve seen any article going in depth into just how many low level humans like data labelers and RLHF’ers there are behind the scenes of these big models. It has to be millions of people worldwide.
There's a really fascinating article about this from a couple years ago that interviewed numerous people working on data labeling / RLHF, including a few who had likely worked on ChatGPT (they don't know for sure because they seldom if ever know which company will use the task they are assigned or for what). Hard numbers are hard to come by because of secrecy in the industry, but it's estimated that the number of people involved is already in the millions and will grow.
Interestingly, despite the boring and rote nature of this work, it can also become quite complicated as well. The author signed up to do data labeling and was given 43 pages (!) of instructions for an image labeling task with a long list of dos and don'ts. Specialist annotation, e.g. chatbot training by a subject matter expert, is a growing field that apparently pays as much as $50 an hour.
"Put another way, ChatGPT seems so human because it was trained by an AI that was mimicking humans who were rating an AI that was mimicking humans who were pretending to be a better version of an AI that was trained on human writing..."
Right now there are top tier LLMs being produced by a bunch of different organizations: OpenAI and Anthropic and Google and Meta and DeepSeek and Qwen and Mistral and xAI and several others as well.
Are they all employing separate armies of labelers? Are they ripping off each other's output to avoid that expense? Or is there some other, less labor intensive mechanisms that they've started to use?
There are middle-men companies like Scale that recruit thousands of remote contractors, probably through other companies they hire. There are of course other less known such companies that also sit between the model companies and the contracted labelers and RLHF’ers. There’s probably several tiers of these middle companies that agglomerate larger pools of workers. But how intermixed the work is and its scale I couldn’t tell you, nor if it’s shifting to something else.
I mean on LinkenIn you can find many AI trainer companies and see they hire for every subject, language, and programming language across several expertise levels. They provide the laborers for the model companies.
I'm also very interested in this. I wasn't aware of the extent of the effort of labelers. If someone could point me to an article or something where I could learn more that would be greatly appreciated.
Just look for any company that offers data annotation as a service, they seem happy to explain their process in detail[0]. There's even a link to a paper from OpenAI[1] and some news about the contractor count[2].
I added a reply to the parent of your comment with a link to an article I found fascinating about the strange world of labeling and RLHF -- this really interesting article from The Verge 2 years ago:
> produce writing not by thinking but by making statistically informed guesses about which lexical item is likely to follow another
What does "thinking" even mean? It turns out that some intelligence can emerge from this stochastic process. LLM can do math and can play chess despite not trained for it. Is that not thinking?
Also, could it be possible that are our brains do the same: generating muscle output or spoken output somehow based on our senses and some "context" stored in our neural network.
I'm sympathetic to this line of reasoning, but “LLM can play chess” is overstating things, and “despite not being trained for it” is understating how many chess games and books would be in the training set of any LLM.
While it's been a few months since I've tested, the last time I tested the reasoning on a game for which very little data is available in book or online text, I was rather underwhelmed with openai's performance.
Many like the author fail to convince me because they never also explain how human minds work. They just wave their hand, look off to a corner of the ceiling with, "But of course that's not how humans think at all," as if we all just know that.
Well, if there's one thing we're pretty sure of about human cognition, it's that there's very few GPUs in a human brain, on account of the very low percentage of sillicon. So, in a very very direct sense, we know for sure that human brains don't work like LLMs.
Now, you could argue that, even though the substrate is different, some important operations might be equivalent in some way. But that is entirely up to you to argue, if you wish to. The one thing we can say for sure is that they are nothing even remotely similar at the physical layer, so the default assumption has to be that they are nothing alike period.
First off, there's an entire field attempting to answer that question, cognitive science.
Secondly, the burden of proof isn't on cog-Sci folk to prove the human mind doesn't work like an llm, it'd be to prove that it does. From we do know, despite not having a flawless understanding on the human mind, it works nothing like an llm.
Side note: The temptation to call anything that appears to act like a mind a mind is called behavioral ism and is a very old cog-Sci concept, disproved many times over.
* direct causal contact with the environment, e.g., the light from the pen hits my eye, which induces mental states
* sensory-motor coordination, ie., that the light hits my eye from the pen enables coordination of the movement of the pen with my body
* sensory-motor representations, ie., my sensory motor system is trainable, and trained by historical envirionemntal coordination
* heirachical planning in coordination, ie., these sensory-motor representations are goal-contextualised, so that I can "solve my hunger" in an infinite number of ways (i can achive this goal against an infinite permutation of obstacles)
* counterfactual reality-oriented mental simulation (aka imagination) -- these rich sensory motor representatiosn are reifable in imagination so i can simulate novel permutaitons to the environment, possible shifts to physics, and so on. I can anticipate these infinite number of obsatcles before any have occured, or have ever occured.
* self-modelling feedback loops, ie., that my own process of sensory-motor coordination is an input into that coordination
* abstraction in self-modelling, ie., that i can form cognitive representations of my own goal directed actions as they succeed/fail, and treat them as objects of their own refinement
* abstraction across representation mental faculties into propositional represenations, ie., that when i imagine that "I am writing", the object of my imagination is the very same object as the action "to write" -- so I know that when I recall/imagine/act/reflect/etc. I am operating on the very-same-objects of thought
* facilities of cognition: quantification, causal reasoning, discrete logical reasoning -- etc. which can be applied both at the sensory, motor and abstract conceptual level (ie., i can "count in sensation" a few objects, also with action, also in intellection)
* concept formation: abduction, various various of induction, etc.
* concept composition: recursion, composition in extension of concepts, composition in intension, etc.
One can go on and on here.
Decribe only what happens in a few minutes of the life of a toddler as they play around with some blocks and you have listed, rather trivially, a vast universe of capbilities that an LLM lacks.
To believe an LLM has anything to do with intelligence is to have somewhat quite profoundly mistaken what capabilities are implied by intelligence -- what animals have, some more than others, and a few even more so. To think this has anything to do with linguistic competence is a proudly strange view of the world.
Nature did not produce intelligence in animals in order that they acquire competence in the correct ordering of linguistic tokens. Universities did, to some degree, produce computer science departments for this activity however.
Yes, 100% this. And even more so for reasoning models, which have a different kind of RL workflow based on reasoning tokens. I expect to see research labs come out with more ways to use RL with LLMs in the future, especially for coding.
I feel it is quite important to dispel this idea given how widespread it is, even though it does gesture at the truth of how LLMs work in a way that's convenient for laypeople.
So it's still not really "AI", it's human intelligence doing the heavy lifting with labeling. The LLM is still just a statistical word guessing mechanism, with additional context added by humans.
The transformers are accurately described in the article. The confusion comes in the Reinforcement Learning Human Feedback (RLHF) process after a transformer based system is trained. These are algorithms on top of the basic model that make additional discriminations of the next word (or phrase) to follow based on human feedback. It's really just a layer that makes these models sound "better" to humans. And it's a great way to muddy the hype response and make humans get warm fuzzies about the response of the LLM.
There are in fact several steps. Training on large text corpora produces a completion model; a model that completes whatever document you give it as accurately as possible. It's kind of hard to make those do useful work, as you have to phrase things as partial solutions that are then filled in. Lots of 'And clearly, the best way to do x is [...]' style prompting tricks required.
Instruction tuning / supervised fine tuning is similar to the above but instead of feeding it arbitrary documents, you feed it examples of 'assistants completing tasks'. This gets you an instruction model which generally seems to follow instructions, to some extent. Usually this is also where specific tokens are baked in that mark boundaries of what is assistant response, what is human, what delineates when one turn ends / another begins, the conversational format, etc.
RLHF / similar methods go further and ask models to complete tasks, and then their outputs are graded on some preference metric. Usually that's humans or a another model that has been trained to specifically provide 'human like' preference scores given some input. This doesn't really change anything functionally but makes it much more (potentially overly) palatable to interact with.
So you're not so much talking to statistical model as having a conversation with a Kenyan data labeler, fuzzily adapted through a transformer model to match the topic you've brought up.
While thw distinction doesn't change the substance of the article, it's valuable context and it's important to dispel the idea that training on the internet does this. Such training gives you GPT2. GPT4.5 is efficiently stored low- cost labor.