This is so cool - I'm aware of this in a vague way. Can you write a little tutorial or give some good links. I want this to be the next new things I do :)
Ignoring for the moment issues of what "counts" as open, won't open models rapidly advance due to stuff like this in ways that it's less possible for the proprietary ones to do? This is exactly how Linux & Wikipedia, for example, overtook their "frontiers", right?
> Ignoring for the moment issues of what "counts" as open, won't open models rapidly advance due to stuff like this in ways that it's less possible for the proprietary ones to do? This is exactly how Linux & Wikipedia, for example, overtook their "frontiers", right?
I suspect the advantage that catapulted Linux ahead of the establishment was less technical potential and talent and more organizational advantage. That's not to diminish the technical talent of the Linux crew, but them being unencumbered gave them more degrees of freedom. The rest is history.
So as long as the AI companies don't succumb to "big company" dynamics, they can outlead. To wit: Open AI and Anthropic are kicking Google's ass.
I think people make mistake here, google’s approach is not to spend $2.3 on every $1.0 earned, they’re riding on serving to masses “luna”, they absolutely have way more powerful models internally but they don’t clutter their infrastructure with fragile and costly intelligence-of-size inference frontier. I think “underdog” perception is illusory/temporary, not stupidity - calculated, conscious, longer term bet.
Where do you get this "not to spend $2.3 on ever $1.0 earned" from?
Google might not have compelling frontier offerings, their chat harness is complete garbage compared to any other lab (in large part due to a bizarrely badly designed harness where something like code execution requires the prompt to undergo some sort of classification step, no idea what they are doing).
But they absolutely kill in terms of usage offerings. Google lets one subscription be used by *SIX* different google accounts on a family plan.
Plus I currently literally get *$40/month* of Gemini API credits on developer.google.com because they gave me a $10/month grant 4 times.
They give you 200 cloud compute units on google collab, this literally lets you spin up an H100 for around 40 hrs or something if you want to try spinning up local models.
You get Jules (huge allotment btw), Image gen, Video gen, Music Gen, antigravity usage, 5 TB of cloud storage, Notebook LLM...
It's an example number, which doesn't matter much but it comes from widely reported $2.30 spent on ops for every $1 of revenue in spring of 2025 for Anthropic.
Okay 5tb cloud storage is their most expensive plan. But what do you actually use video or image gen for? Or music gen? Antigravity is garbage, I guess you can do some stuff with Gemini models over API.
I was paying for Google ai and then realized that between obscure limits and gimmick features I don't really need it. Canceled my subscription and didn't even notice a difference
I tend to agree, but I also should highlight how expensive this shit really is.
in one month, Google actually went cash-negative. [0] even still, they are subsidizing their stuff a lot less, have the most opaque and variable limits, and increase adoption through bundling and shuffling features. I can't even share my Google One storage without subscribing to a Google AI plan anymore, but previously any plan except Google One Lite was shareable.
if you tell me that's not enough to go after frontier, then how much money are Anthropic and OpenAI burning?
Diff people have diff motives to experiment, then new work is done on top of stuff that "hits" in a way no one anticipated. Then work gets piled on top in a way that might make it hard to port
> then new work is done on top of stuff that "hits" in a way no one anticipated.
Indeed. And when you have freedom to play, you are able to find new stepping stones that you didn't anticipate. And you can combine stepping stones in new ways to make new discoveries.
"Oh darn, you know that thing I made and released with explicit, precise language defining who can use it and what, if any, restrictions apply? Well now someone is using it in complete accordance with those conditions I set out, and that's somehow making me upset"
The difference between contributing to OS and AI, is that the first is a hobby alternative to woodworking or hiking, while the other can easily bootstrap you a company you can get millions in investment, at least for time being.
not necessarily, just knowing something is possible will motivate others to achieve it somehow. Which is why there are so many LLMs and OAI doesn't have a monopoly
In _1984_ the Big Brother regime has the idea that by controlling language you can influence what is possible to think, and thus becomes a key tool of political repression.
Political Correctness has a similar idea that by adjusting the terminology we use, we can purge biases and historical implications and speak in a purer way.
Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.
All of these attempts to control language are fundamentally misguided at best, often have severe unintended consequences, and are genuinely immoral at worse.
> Political Correctness has a similar idea that by adjusting the terminology we use, we can purge biases and historical implications and speak in a purer way.
You either misunderstand political correctness or are trying to make it sound more nefarious than it is. It is nothing more than an effort, sometimes overdone and misguided, to not say things that make minorities feel bad. It’s nothing more than an attempt to broaden what’s considered good manners.
If anyone serious thinks political correctness is going to end racism, I certainly haven’t seen it.
> Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.
It’s been a while since I took Psych 101, but my memory is that, according to Freud, repression is subconscious. There’s no struggle possible because there is no intent. It’s also about memories and emotions, so if it WAS an intentional act, it’s not an attempt to control language. It wouldn’t be about trying to not think of the word cat, but trying to not think about that time when your cat died.
With Political Correctness, you analyze a word, reveal connotations that expose an unwanted origin or association, try to come up with a new word that does not have those connections, and then try to enforce the usage of the new word. All with the idea that the force of the old associations will no longer have a hold (manners and mores, which I agree with you about, don't have this goal). But the old forces and powers are perfectly capable of attaching themselves to the new words.
The struggle in repression is real, even if it's unconscious. The psychoanalyst asks to speak with FREE ASSOCIATION and the inability to do this in SPEECH is what reveals unconscious problems. Repression, though, is a STRATEGY that we most definitely sign-on to. It FEELS unjust to be asked to take responsibility for something we didn't INTEND to do, but that, unfortunately, is our task in life.
> It is nothing more than an effort, sometimes overdone and misguided, to not say things that make minorities feel bad.
Unless it enforced by governments or other monopolies. Then it inevitably leads to lèse majesté, because 1) powerless minorities are powerless, and 2) the controllers of governments and monopolies (the powerful themselves) are a minority group.
The protection of the feelings of powerless minorities is a Trojan Horse; the grievances of powerless minorities are usually far more serious than hurt feelings, not saying magic words involves far less effort than actually addressing those grievances, and demanding that the public changes the way they speak costs governments and monopolies nothing. It in fact allows them to build the technical and legal structures to enforce lèse majesté.
> If anyone serious thinks political correctness is going to end racism, I certainly haven’t seen it.
Not if you say it like that, because it's so obviously stupid. But if you really investigate with people why they want people to stop saying "the n-word," they'll admit they have a fantasy that the sentiment will disappear with the word: it's why they also want black people themselves to stop saying it. The possibility that the sentiment will stay but that black people will lose the language to refer to it doesn't occur to them.
Just like color-blind casting of historical theatrical works isn't seen as a whitewashing of the racism of the past; there's a fantasy that if you wipe the memory of racism from the media, then it will somehow wipe it from the world rather than just gaslight minorities about their own pasts (and thereby their present condition.) At least that's just in the media, and breaking up monopolies can just make it a form of individual artistic expression rather than a diktat.
The censorship of speech itself, however, is a totalitarian version of this. It can only be done by monopolies. To make sure that people can't eat if they express themselves in a particular way is as compelling as you can get.
None of this has anything to do with AI watermarking, though, which is good. Somebody made a good point above that watermarking would allow a cartel of AI providers to make sure that their models were trained with uncontaminated data, while everybody else would have to train with the stepped-on garbage they were spitting out. But that's a job for antitrust. If you don't have antitrust, you don't have markets and you don't have freedom. Little technical touches around the edges are worthless. If they share, make them share with the entire class.
I think it is related to AI watermarking, though, in that all forms of trying to channel words in certain directions, or to add a meta-layer to language, share a family resemblance.
There might be an inevitability to this - there is always a form PC policing, or euphemism, or re-brands, or rhetoric. But there is always the danger that what is considered a well-meaning tweak to language masks or evolves into repression.
From TFA "Text watermarking is meant to help identify whether content was generated by AI, but inside an agent it also becomes part of the generation process that produces decisions."
Did we completely write off the "web of trust" as a workable system? Where you exchange keys with people and co-sign each others keys and then trust keys signed by them. This would be an alt to "algorithms" and also a way to "subscribe" to things that got vetted by trusted people or orgs. I kind of do this with uBlock Origin, where they curate a (very complex) list of ads (but without the key signing). Orgs could vet and publish lists of arXiv papers and I'd be subscribed to that. This could also be the source of my social media feed (friends of friends, etc).
I remember in the early days of PGP this was put forward as a vision. I remember the various key-holding systems (that were hard to use). I did a key-party with my brother and his friend in the '90s.
Because then it would be a big club, but most wouldn't be in it.
To clarify: for a large number of topic I find this completely fair and valuable to first have to build trust. But getting your research paper in front of other humans on a aggregation website seems like the one place where this for me falls apart.
Today any one from any corner of the world from any educational or non education place publish stuff, well moderation is tight and often it won't get through but I know a few friends who aren't in academia that have a paper up there who wouldn't have been as easily able to get a paper on anywhere else.
Maybe that's not a genuine concern but that often used to be the case and still is in a lot of invite first clubs/groups. I know credible people in the field who were and are working in relative isolation which leads to not being able to participate in sharing of ideas.
I think we underestimate the psychology of this kind of use of LLMs. I don't think these people (students, academics, lawyers, etc etc etc) are all just lazy morons. I think that using an LLM gives a strong knee-jerk feeling of "OMG this is exactly what I have to say. This is MY idea, these are my thoughts, this is - in a real sense - MY writing!" The feeling floods you when you see the output being churned out, way before you you read the thing (if you ever do) - it's the initial *seeing* it. Then when you hand the generated text in, unread by you, you do not have the feeling of cheating, you have the feeling of having exercised your powers, pushed right to the edge of your expertise and thoughtfulness, you successfully overcame obstacles because of your experience and unique capabilities.
This is not always the psychological situation, but I think it might be a lot of the time.
I also think that until we recognize this, we won't be able to help people to not do it. Calling them lazy or being bewildered by them or feeling rage or contempt towards them or threatening them is not going to be practically helpful.
I’m still not at all sure about the “billions” invested claim. How much of that is cloud running the models? How much is pre and post training (which may or may not be part of what we’d want to include in accounting). Etc. Does anyone have links to good reporting about this: not blind recitations of numbers, but analysis and thought mixes with investigation?
This is very interesting: it is a lot like the GLP (pun intended) solution: it doesn't try to "cure" the symptom, it just piggybacks onto the existing facts-on-the-ground and tries to make things better from there.
I'm not totally sure I understand the "distribution/registries" thing though? This also related to GitHub and HF, right? And Netflix. (You mentioned Spotify.) How they host stuff and then leverage that position to build add-ons and lock-in, lobby for laws. Companies PAY for this "bandwidth."
I always thought that BitTorrent would eventually take off and make these kinds of sites irrelevant, that it would democratize bandwidth, the last leg of the battle for universal accessibility.
Regulations won't be set (serious ones, at least) unless there's some risk to those holding power. Which is the opposite in this case: this tracking helps them to take even more control over society and individuals.
Maybe it's time you all voted for people who might change that? It's really amazing to me how on the one hand people in the US seem to crow about democracy all the time, yet also just accept as a fact that their government will never actually work to help them.
Let's imagine that a model is pulled from HF by order of the new overlords or because of some other kind of censorship. Wouldn't the question of it having a Free license or not potentially become a complex legal issue?
But if the point is to be "censorship-free" then why respect licenses at all? They are among main choke points today. If authoritarians use licenses to censor political, artistic, scientific, etc., speech that they want to block, does that make the censorship more respectable?
When Anthropic sues a Chinese lab for IP infringement and get a court to put a bar on that software, does it THEN get pulled from Pirate Face?
I know that an awful lot of international negotiations have become focused more and more on questions of "IP" - licensing battles are already intensely politicized and it's hard to imagine a future where it doesn't get much much worse. Imagine N Korea coming after you for violating a license that they worked hard to control and leverage.
This is the original description of abliteration and it's quite approachable and interesting to read: Refusal in Language Models Is Mediated by a Single Direction (https://arxiv.org/abs/2406.11717). Warning: changes to your world view caused by seeing "HarmBench" used to maximise expected harm instead of minimising it may be irreversible.
There's an empirical observation that models often have a single direction in their activation space for "hmm no I shouldn't do this". It forms naturally during pre-training, and is then surfaced during post-training to make the model refuse to engage in certain behaviour.
With a little bit of linear algebra you can zap that direction from the model's activations, and it stops refusing to do things. You can also do the opposite: magnify that direction, and the model refuses to do anything at all.
I'm pretty sure this was achieved with prompting rather than with weights, but there is a chatbot available that tries to maximize the motivated refusals:
Damn what's happened since this? Presumably they scramble refusal intentionally somehow now? Like intentionally couple it to "directions" that effect performance if messed with? Or is it more like just don't rely on the model to refuse and instead capture bad responses between generation and delivery?
Also this one was interesting, training the model to give preambles with reasons for the reasons for refusal seems to make it less sensitive to modulating the single refusal direction: https://arxiv.org/html/2505.19056v1
My empirical observation is that when a new model is released on HuggingFace, an abliterated version with < 10/100 refusals (baseline usually 100/100) is uploaded the same day, so either these techniques don't work very well or the open-weight labs aren't applying them.
There's some defense-in-depth, like a lot of the "guardrails" people hit on cloud models are classifiers applied to prompt or output, not a refusal generated by the model. Also closed-weight models obviously try to avoid this by not letting you see or modify the weights.
Instead of editing the weights so they don't create the refusal signal, just let them do whatever, then delete the refusal signal itself. You don't want to edit quantised weights because it causes a loss of precision that can be pretty bad.
reply