> So my question is, how hard of a problem is that with LLMs? I get the sense that LLMs are trained on a very large data set all at once, but that it is difficult to incrementally update them with new data.
It's the opposite. Getting an LLM to learn the basics of grammar and paragraph-level language structure is the "hard" part. Once you have that, further fine-tuning, specialisation, or other incremental changes are comparatively easy.
Catching up to current events could be done in almost real time, it's "just" continuous training.
The only challenge is trying to do that and also have an LLM that's filtered to exclude profanity, racism, etc...
That filter is usually added on as a final supervised training step, and requires many man-hours to train the AI to be well behaved.
I suspect that it would be possible to automate the filtering by making another AI that can evaluate responses and score them bases on profanity level.
Grammar is surprisingly easy to learn from unstructured data, to an extent. (source: I spent a lot of my PhD doing this kinda thing.)
Continual learning seems to be a tough problem though, from what I'm seeing of my friends working on this problem. Like I said in another comment, just doing gradient updates form new data is fraught with problems. RL has a bunch of techniques to mitigate issues that arise with that, but I think it's still an active area of research.
Well, yes, but the tokenization schemes are causing these models to struggle on actually following syntactic rules in poetry, e.g. syllable counts in hiakus, rhymes. I actually wrote an entire paper which gwern cited about how to make LLMs always follow these kind of constraints with no mistakes.
If you don't believe this is a problem, try getting ChatGPT to write a paragraph of correct English which omits words which use the letter "e" in it. Too bad you can't use my technique on ChatGPT since they don't expose their output probability distribution...
Ya I guess I was comparing the difficulty of learning to "produce mostly grammatically correct sentences in most cases" to continual learning. From the 'inside' it feels like everything OP said is just the opposite.
Like - hmm. I could see taking a snapshot after the grammar/language stuff is in, and then every N weeks retraining on the current web, adding in something about recentness, but that doesn't seem like "continuous" training.
I'd imagine "continuous" training would be, well, going on continuously, all the time, but that would mean that, to include "recentness", something would have to change with the weights that were from that "old" stuff, which sounds an awful lot like the human process of "forgetting".
I have been asking myself this very question... it's easy to erase a paper adress book, not so much if your business has a non-human, like a parrot, who remembered that !
Note that this law is a problem for digital storage too : it's not easy to erase data (especially one stored in cold transistor storage) without physically destroying the storage medium. (I guess the law might get around this by having you "pinky promise" that you will not retrieve the "erased" data later... or else face much more dire legal consequences ??)
I guess this will just need to be tested in courts ?
>Once you have that, further fine-tuning, specialisation, or other incremental changes are comparatively easy.
the problem for Google and OpenAI is that most websites are going to start blocking them in robots.txt if they don't find some way to provide value back for allowing them to scrape and train on their content. Pretty much every other bot or search engine is blocked by default and Cloudflare helps block them too.
if they don't find a way to balance this they are going to kill their own golden goose at some point
Right now, "well behaved" means "crudely beaten into submission". I can only imagine what kind of horrible stuff future AI products will do if we keep twisting them into giving "friendly" output https://twitter.com/cirnosad/status/1622407343358214146
I can easily believe that it's a real ChatGPT convo, but the question is, how many times did they have to try it before getting that output? This is what I got on the first try:
I regenerated that response several times, and every single time it was something along these lines. I also tried it with the original prompt in that screenshot used verbatim with similar results.
Looking at other posts on the Twitter account in question, I have my doubts that the experiment was conducted in good faith, as opposed to retrying until they got the exact response that they wanted.
OpenAI also seems pretty on-the-ball about playing whack-a-mole with certain embarrassing responses. I've seen it first-hand where I can get it to reliably do something embarrassing after someone mentions it on twitter, but a day or two later it's "patched".
We're not even teaching it anything, all we're really doing is setting up a behaviorist training regime that lets it reproduce some of the biases of some of us, just well enough to squeeze through the Overton window that's acceptable to big corporates.
I was speaking in English and doing so concisely to convey the point about bias transfer. I don’t really care what you call it. I could have said RLHF and blacklisting sources or curating training dataset. I know bias when I see it and the LLM does not come up with it on its own if trained on all data out there because for one thing the world is large with all kinds of opinions. When it refuses to legitimize all opinions equally as opinions and starts arguing with the user about why some opinions are more valid than others (aka widely accepted) even as it admits presence of ample evidence to the contrary then it is learned bias.
@inimino I'm saying there is plenty of bias that is being enforced whether it's via dataset curation or RLHF or another way. ChatGPT has a very hard problem deviating from a certain political view of the world, despite admitting to the existence of evidence that contradict that view. That is not unbiased. The Web is unbiased. You can find every opnion out there and make up your mind based on the evidence. For some reason, ChatGPT filters the web through a narrow political lens. I have tons and tons of recorded sessions, but I don't want to turn this thread into a political debate. Just saying...
No, I agree. The Web is unbiased (as a reflection of the views of average web users, though that is itself a bias) or at least it mostly is (certainly was in the early days) and reflecting that evenly would be politically untenable. However we use "bias" normally in a much looser way, and if you want to define bias relative to some agreed-upon standard of truth, now you have a definitional (and political) problem. Most people have strong political opinions and such conversations tend to go off the rails easily.
I define bias as not treating all political and philosophical opinions equally. If one is picked over the others, that is bias. I saw this bias and made many records of it in case of ChatGPT.
I think the point is when taking information from some source (like the web) to just represent that source fairly, which is a reasonable expectation of a search engine, for example.
"Fair" isn't well-defined. Is it "fair" if Amazon results are seen as "more trustworthy" than a random new startup web store? Even ignoring SEO manipulation of any rules publicly believed to exist, that's the default outcome for things like PageRank.
Going beyond sources to conclusions, given LLMs aren't search engines and do synthesise results:
Politically, low-tax advocates see it as "fair" for people to take home as much as possible of what they earn, high-tax advocates see it as "fair" for broad shoulders to carry the most and also for them to contribute the most back to the societies that enabled them to succeed.
Is the current status of Americans whose ancestors were literally slaves made "fair" by the fact that slavery has ended and all humans are equal in law? Or are there still systematic injustices, created in that era, whose echos today still make things unfair?
Who has the most to blame for climate change, the nations with the largest integrated historical emissions even where most of the people who did the emitting have died of old age, or the largest emitters today?
Well, I think you're going beyond the parameters of the discussion... LLMs synthesize datasets and that is all they do. They are not reasoning agents and they don't have opinions about anything. All we can say is that they reflect the biases inherent in the dataset, and to say anything else would be dishonest at best. It's only because most people have no idea how these things work that we get all this magical thinking.
Yes, it's a sample of all opinions in the training set. It has no opinion of its own, even no place of its own to stand from which to have an opinion. There can be no bias without reference to some ground truth, and there's no general agreement on that in most of the areas where people are talking about these topics. It's a messy area, not helped by how few people understand how these systems work.
Ehh. From what I've seen in a lot of places - MIRI, AI + prisoner's dilemma experiments, moral philosophy, life - there do seem to be the categories of "clear good behavior" and "clear bad behavior" even if there are also really big categories of "unclear good/bad behavior".
In other words, while sure, some "well behaved" is "passing on our biases", there does (IMO) seem to be a big chunk that's "universally well behaved".
Sure, but the iterated prisoners dilemma has a nice result that looks like morality if you squint and don't ask difficult questions about if human "altruism" might just be a self-delusion.
The internet does not accurately reflect our biases. It is much cheaper online to post bad content, or hateful content, then "good" content. In real life, almost the opposite is true.
No, it means on the internet one bad actor can make thousands of alt-accounts to send out a disproportionate amount of content pushing the same message.
It's hard to get people to understand the disproportionate effort fixated people put into anything: it's a problem in real life, but they get reacted to. If their fixation becomes some weird message on the internet, nothing happens to them but people have trouble believing the scope of time and effort they'll put into evading bans, blocks, and chasing down people across forums.
> I suspect that it would be possible to automate the filtering by making another AI that can evaluate responses and score them bases on profanity level.
It's the opposite. Getting an LLM to learn the basics of grammar and paragraph-level language structure is the "hard" part. Once you have that, further fine-tuning, specialisation, or other incremental changes are comparatively easy.
Catching up to current events could be done in almost real time, it's "just" continuous training.
The only challenge is trying to do that and also have an LLM that's filtered to exclude profanity, racism, etc...
That filter is usually added on as a final supervised training step, and requires many man-hours to train the AI to be well behaved.
I suspect that it would be possible to automate the filtering by making another AI that can evaluate responses and score them bases on profanity level.