I think the big lesson of this that hasn't yet been learned: it literally doesn't matter how moral Amodei or anyone else is at Anthropic. When push comes to shove, the US government can step in and take it away.
They will either play ball with the US government on the US government's terms or they will be replaced or destroyed.
It is a misconception for them to believe they can dictate the terms of this technology.
That's a bit like asking how the defendant in a legal case is an interested party.
Even if you think someone is guilty, it does make sense to allow them to at least submit their defense. And if they choose to use that time to advocate for their own promotion, let them.
Your logic doesn't hold up well to simple escalation logic.
Company A founds itself on doing 0 harm to Area X. Competitor B shows up and starts finding success doing 10 harm to Area X, so Company A makes a "moral" decision: If we do 9 harm to Area X, we are preventing 1 entire harm. Isn't that real value? then Company C shows up and starts finding success doing 100 harm to Area X, so Company A changes it's moral stance to "unless we do 99 harm to Area X ..."
I know an old lady who swallowed a fly kind of logic going on here.
> then Company C shows up and starts finding success doing 100 harm to Area X, so Company A changes it's moral stance to "unless we do 99 harm to Area X ..."
I mean yes this is technically possible. But I think in many cases, especially "winner-take-all" markets like online search engines, social networks, etc., you don't get this large number of repeated threats. Fending off a competitor or two might be enough. And just as it's possible for there to be some advantage that opens from doing 99 harm to Area X, it's also possible that it never happens.
But also, let's pretend the hypothetical you say _did_ happen?
What should occur? Should the company just NOT do 99 harm to Area X and instead allow 100? If so, why? Unless you break the hypothetical by adding some alternative option C, as much as we don't like the preventative-99 option, it's still better than the allowing-100 option.
> Unless you break the hypothetical by adding some alternative option C
That is kind of the point, isn't it? That my hypothetical scenario isn't realistic.
Let's imagine two worlds. A world where individuals refuse the false dichotomy and search for option C. And the world where someone accepts the false dichotomy and justifies evil.
I would argue that anyone that advocates for the justification of evil is actually using motivated reasoning. It breaks my original premise "Company A founds itself on doing 0 harm to Area X". Clearly they didn't and their embracing of evil shows that their principles mean nothing.
As a moral test, ask yourself: If I said "you must kill 99 people otherwise I will kill 100", would you feel morally justified to kill those 99 people? If your answer is "yes", then you are manipulable by those who want you to commit evil on their behalf. They don't have to commit any murders, just convince you that you have no other choice.
I mean, your proposed logic seems to be quite consistent from a basic game theory perspective. Defecting in a prisoners dilemma and races to the bottom are both well observed phenomena.
We have 10,000+ years of human civilization at this point. There must be some other active ethical maxim operating other than "choose the lesser of two evils" to explain why there is so much cooperation amongst humans. Evidence is not on the side of the preeminence of races to the bottom.
You should investigate the repeated prisoners dilemma.
> You should investigate the repeated prisoners dilemma.
Well aware. Obviously, the entirety of human civilization is a bit more complicated than a prisoners dilemma, iterated or not. Yet prisoners dilemma's and races to the bottom still exist, and it makes no sense to argue against them in the abstract.
I think we are very disconnected on the topic of conversation here. Somehow you've confabulated my point with an attack on the prisoners dilemma or races to the bottom?
The person I was responding to made the point that if you want to minimize evil in the world, sometimes you have to add evil to a lesser degree. As in my example, if I do 9 points of evil but prevent 10 points of evil then according to OP I've added value to the world in the form of the 1 point of evil I have reduced.
I responded that this can lead to an escalation trap. This assumes that we would all prefer less evil in the world, right? So how do we get out of the escalation trap? Repeated application of the maxim "always do a bit less evil than the worst possible competitor" will not lead to a minimization of evil overall, only a creeping increase in the total amount of evil in the world.
How are you equating this to me arguing against the existence of races to the bottom?
It is very difficult to see this move as anything other than Anthropic pulling the ladder up behind itself. They can dress it up in "safety" all they want, I find it hard to interpret this in a charitable way.
This reminds me of how dark-pattern common wisdom in Web 1.0 website development was to ban external links. Then how social apps prevented the export of data and actively worked to nerf significant interoperability through APIs.
But this is a tool, not just a data moat. Like a knife that degrades your ability to create knives. Or like a text editor that prevents you from implementing a text editor.
It's also hard to imagine them not doing this with any of the products they're building. "You can't use Claude to build an agent because that competes with Claude Code, you can't use Claude to build a design tool because that competes with Claude Design, you can't use Claude to build an email tool because that competes with Cowork."
Only the priest is allowed into the sanctum is a rule that is as old as society. It is created for one reason but gets violated for another. The human mind is made of layers to handle predictions over different time horizons. Due to unpredictability in the universe contradictions between layers will keep arising. We make up stories to cope. So there is Control and there is Illusion of Control.
It's becoming extremely important to support open-source AI, especially legally. Anthropic is willing to go totalitarian this quickly; imagine how much worse they'd be willing to do with government-granted monopolies that ban open-source competition (like they've repeatedly pursued).
It's a little shocking and gruesome how quickly they're willing to tip their hand. They want to replace all software engineering with their own product, and then silently kill anyone making competing software. What other products will they launch in the future? Better hope you aren't in a space they want into: they'll cut your legs out from under you.
Oh, and training on your data from the internet? Ha ha. Terms of service apply to other people, not them. Parasites.
Open source doesnt matter if you still need to make 100k year to have your own mediocre model.
There is no magic compression.
There is no magic post training.
Your phone or laptop will never do what you think its going to be able to.
There are limits to what consumer hardware will ever be able to run, in its current form. Open source isn't going to save us if they gatekeep access to hardware, which idk if you've been paying attention. They dont plan on making consumer grade hardware more powerful, they want to rent that power to you.
Technological serfdom is coming if they get their way.
You don't need to be able to self-host it. It's fine to pay someone else for it. If it's open-source, competition will ensure inference providers support it well enough, and if an open-source provider is dumb enough to nerf their model for (useful) coding tasks, there's plenty of incentive for inference companies to do some lightweight finetuning to restore the capability.
I disagree, I think being able to self-host it to some extent is very important.
Personal computing democratized the means of (software) production and enabled real upward class mobility for a lot of people.
The efforts happening now are threatening to completely lock up the ability to compute locally, seizing the means of production from us. That must not happen.
your parent said "You don't need to be able to self-host it", you countered with "I disagree, I think being able to self-host it to some extent"
Bro, I don't know what you're disagreeing with, the two statements can and should be true at the same time. It's not only unnecessary but also impossible for everyone to self-host, for the vast majority this isn't a necessity and it shouldn't be. Actually being stuck on self-hosting for all is mighty silly from economics standpoint, pushing on it can ruin the entire enterprise.
But being able to self host? Sure why not, if you insist and are ready to suffer... knock yourself out, but that's a socially insignificant act which doesn't scale, good only as a backup option.
Because "you don't need to be able to self-host it" is a constraint, I'm arguing that you DO need to be able to self host it, not that difficult. Every thing being rented out instead of available for ownership is nothing but neofeudalism, which we are rapidly spiraling into.
> pushing on it can ruin the entire enterprise.
I'm supposed to feel sorry for the trillion dollar corporations that hoovered up all of human knowledge, for profit, and are now the direct reason why 32GB of RAM is now $500 instead of $90, all while renting compute back out to us, making it more and more expensive to actually own hardware, a fundamental privilege that enabled all of this technology in the first place?
Let the "enterprise" be ruined. It'll be for the better.
> Let the "enterprise" be ruined. It'll be for the better.
Maybe my choice of words was a bit confusing, I actually had in mind the "enterprise" of making sure people have access to capable and uncensored models. As far as the enterprises you dislike, I don't use them, I do use hosted models but not theirs.
> I'm supposed to feel sorry for the trillion dollar corporations that hoovered up all of human knowledge, for profit, and are now the direct reason why 32GB of RAM is now $500 instead of $90, all while renting compute back out to us, making it more and more expensive to actually own hardware, a fundamental privilege that enabled all of this technology in the first place?
No disagreement here, I've been writing about it for months now. There's a lot to say about it but it's a long discussion that will have to be focused on economics and politics, something HN isn't fond of.
All I can say, is that you're right, the goal is to have abundant and cheap hardware and a lot of other things too. But in order to get there we will have to learn to pick, choose and support hosted models that care about our freedom to know things.
> Technological serfdom is coming if they get their way.
I'm deeply concerned about this. We're seeing all these moves towards remote attestation, identity verification. Now we're being literally priced out of hardware...
Many, many, many public policy positions; for a clear-cut example, they eventually supported SB 1047 [1] which would have banned open-sourcing any model trained with over 10^26 FLOPS (i.e. what Anthropic reportedly used to train Mythos). Their "Responsible Scaling Policy" [2] — a set of policy proposals that includes recommendations for government regulation — specifically calls out requiring "third-party controls" on model weights to prevent access; for developers to prevent "modification of models" such as fine-tuning (obviously impossible for open-source or open-weight models); prevent usage of model weights in "Automated R&D in key domains" which they specifically call out AI development as a key domain (again, obviously impossible for open-source); etc etc.
They want to ban open-source AI and are not shy about it.
The nuance is not what they propose, but why, even according to them, they propose this. Honestly the proposals are appalling, but biosafety arguments are not immediately dismissible. Ultimate cyber threats we can handle by rewinding society 50 years back. We can’t undo a novel genetically engineered virus.
I mean arguably, we _could_ create conditions in which it is much less likely for people to start developing such a novel genetically engineered virus.
If you think about the factors that lead to people wanting to do such a thing, they're almost always tied to (perceived) inequality, (perceived) injustice or similar in some way.
I do believe that we could greatly reduce a whole bunch of such risks by just stopping to squeeze people as hard as we do right now.
But that would require a major refactoring I guess.
Yes but ethical responsibilities are entirely irrelevant, because that crazy person would still bioweapon you, and you kinda do not want that.
So regardless of if they're evil/bad/crazy or not, you still want to have that not happen. Hence the systemic perspective instead of the focus on the individual.
America has somehow managed to hang on to the right to encryption, despite plenty of well-heeled opponents, so it's possible to hang on to the right to open-source models. But it'll take a lot of vocal support, since there's strong incentive for Anthropic to try to cajole the government into banning competition (and they've already crossed that particular Rubicon, whereas OpenAI to my knowledge hasn't and at least still releases some open-source models like gpt-oss-120b).
> It is very difficult to see this move as anything other than Anthropic pulling the ladder up behind itself.
It's worse than that, it also exempts from examination and competition some areas of science and technology while sterilizing others and emptying them from human participation. None of this is good for anyone except a very narrow circle of people.
Then, it creates a precedent where private entities decide who will be allowed access to what knowledge. Instead of government regulation, private corps will be "fighting crime" by dumbing down and spying on the people they don't like.
I don't think this Soylent Green strategy is a coincidence, it's been predicted and depicted, the social forces leading there are plainly visible to anyone capable of independent thought.
Open science can't come soon enough, unsubscribing is the best option until then.
They believe they're going to eventually develop AI that's capable of recursive self improvement into world-redefining super-intelligence. I wouldn't expect someone in that position to risk giving away their lead. I expect we're going to see more of the top labs selectively holding back their best stuff.
I accept your point, in the sense that I wouldn't suggest that they have any obligation to share their own research.
What seems to be different here, is that they are saying they won't let you use their tool to do your own research.
It is a subtle but important difference. They aren't saying "we have secret sauce we won't share", they seem to be saying "we will prevent the tool you are paying for from independently creating a competing idea".
It's the inevitable end game. If the models ever become practically useful in a closed loop, there's no other choice except to keep the model private and use it to compete directly with their current enterprise customers.
I don't see it as a ladder at all, unless you claim Anthropic built their own models by training off of other closed frontier models, violating those models' ToS
to be clear, I'm not saying what they did in scraping to learn was ethical. It wasn't. But I just don't see it as pulling the ladder. The ladder is still there.
"You can't take code produced by our service to make competing services, but we can take code you produced to compete with your service (i.e. software engineering)" is pulling up the ladder IMO. If they can from-scratch train a model without using human-produced code, I think they're within their rights to stop humans from using their model to compete with them. But if they're training on GitHub/Hugging Face/arXiv/Common Crawl/etc, which certainly includes many open-source repos whose licenses they're violating, I don't think they should be legally allowed to prevent people from using their model to produce code that merely competes with them. They themselves have taken other people's code in order to compete with software engineers.
I hope they get nationalized and either the models are open-sourced or the profits are owned by the public.
I don’t know if you’ve tried to scrape or programmatically download a lot of websites recently! It’s not possible to repeat their data collection process anymore.
maybe i'm just pedantic. it's possible you could only build models like these from scratch until a few years ago for that reason, but isn't that an (illegal,unethical) early mover advantage?
to me ladder pulling would be:
- web scraping for model training becomes illegal, with heavy punitive penalties
- training models above a certain compute threshold requires government licensing
- expensive third-party audits are required before deploying models above a capability threshold
Imagine you are in the shower one day and you come up with the sketch of a possible innovation on model architecture, however there are some fine details and tricky implementations that you need to do in order to test it out. So you fire up Claude Code and describe your idea and ask it to provide some reflection on the idea and work out some proof-of-concept code.
In this scenario, this is your idea. You aren't "training off of other closed frontier models" in a distillation sense. This is your insight, your idea, possibly gained from reading a lot of papers and built on your own experience.
How do you feel if the model refuses? Do you consider the scenario I described a violation of someone else's rights?
This is an interesting project and in some ways similar to an idea I had. My idea was actually just to aggregate primary texts (whatever public domain versions are available) for a wide range of philosophical and spiritual work and provide an easy way to include it as context in straight-forward LLM calls.
I've skimmed this announcement, your github repo and your site and it isn't clear to me, are these custom models? Are they fine-tuned from some base model? e.g. do you have 30 separate models?
We have no custom models and no fine tuning. It is the base model, Qwen3 235B for the free tier and we recommend Qwen3.6 27B for the local mode. So the figures are data, not weights. We have instruction files for every figure and additional voice profiles for the councils. The work was put into the iterations to improve the instructions. It is possible to fetch them from the CDN, for example: https://media.agoracosmica.org/instructions/jung/free_conver...
I like your primary texts idea. For our case we tried to keep the instructions lean to have them around 4k tokens, so that it also works in local mode for users with limited context.
Totally understand on managing context length. But in a sense, the work is just providing the library of primary texts (and given the public domain nature of these kinds of works, that seems both ethical and legal) and some mechanism to include them into the context as desired by the user.
As an example user story, maybe I want to get Plato's reaction to Buddha. It might be convenient to have a library of sutra's that I could grab extracts from in order to send to the instructed model for further reflection. That puts the context management into the users hands. From a UI perspective you would need a library interface, the ability to select extracts, some indication of context available vs. context used, etc.
If xAI is a datacenter REIT, it is a special kind that has a promise that no other datacenter provider could dream of: LEO datacenters. As far-fetched as that may sound, the biggest profit center for SpaceX in my understanding was Starlink. xAI already has extremely high-bandwidth connections from Earth to LEO available. Connecting that to solar powered orbital datacenters seems doable in realistic timeframes, especially once Starship comes online and gives them a significant boost in launch capacity.
If that ends up being viable and profitable, there is no realistic competition for decades. In this view, xAI earning a reputation as a reliable AI hyperscaler is just another tactic in that strategy.
Cost per pound? Then $/Watts/TFLOPS minus cost per pound?
Out of curiosity (since I basically never saw $/lb mentioned in any replies anywhere on this, which is hilarious; like talking about having your grain mill at 10,000ft/in the mountains since sunlight is better there): Have you ever tried a forSpace Program?
(And not only 100mi or more above, they're 17,500 mph faster--Mach 22 Datacenters in an oxygen-free, higher-radiation, insulated environment with absolutely no resources)
I have not done any calculation on the capex but I'm guessing that SpaceX has. We're also just guessing on what these "datacenters" will look like. It is silly to think of them in the same way we see the football-field-sized ultra secure facilities on land. They can be highly distributed in a way that just wouldn't make sense on land. Perhaps even incrementally built out in the same way that Starlink capacity was.
Regardless, if Google is spending just shy of 1 billion USD per month, that suggests that there is a pretty high ceiling on capex available.
Consider the PR of massive datacenters here on Earth. People complain about noise, water usage. It doesn't even matter if those concerns are valid, the PR is bad enough. That might attract other massive corps that want to outsource instead of deal with the headache of building local.
You realize that not long ago companies were exploring building nuclear power sites next to their data centers to handle the expected power needs?
I'm not saying it will work. I'm saying if it does, SpaceX will own the market for a good while.
Sure, but considering the size of the challenge it makes sense to figure out the parts that can be studied on the surface first. Challenges like procuring, challenges like setting up relationships with potential customers. You probably want to figure out everything you can so that when you move on to the hard part you aren't distracted by the rest.
Consider the alternative. SpaceX figures out how to build the datacenter in space thing but fails at the rest. That would be an expensive mistake.
I have an understanding/interpretation of the philosopher Kierkegaard's metaphor of the Knight of Infinite Resignation from his work Fear and Trembling [1] that relates to this.
He tells the story of a Knight that falls in love with a princess. In the olden days princesses were married off by their parents for political reasons. There is no way his love, even if it is returned, can ever be fulfilled.
So the Knight resigns himself and marries the butchers widow. After all, she is pretty enough, she has inherited a profitable business from her late husband. And she will be elevated socially by a marriage to a Knight, so she is very keen.
But the Knight has to resign himself constantly, like in the dead of night while lying in bed and dreaming about what might have been. He must avoid falling into resentment and maintain the strength of his will.
This is a central concept to Kierkegaard, started in Either/Or and continued in Fear and Trembling.
I handle this by not dwelling too much on the past. Things undone do haunt me, but they are not and what is, is. The ephemeral is as useless as the emotions I have about them, and I treat it as such. Anything else is inviting agony and regret. I have enough of those and I don't need more of them.
A lot of this emotional anguish can be quashed with what might be considered "rude" pragmatism. Remove what is useless, ignore what intrudes and cannot be changed, get on with what you have and don't dwell too much on what you do not have.
Another way to frame this is from a Biblical perspective: be thankful for what good things you have and do not yearn for what you have not been given. Everything in this life is a gift. Every penny you earn, every breath you draw is more than you deserve. In that light, the life I have is wonderful, and full of joy and satisfaction. I have so much that I do not deserve, so many good things and many beyond the limit of my awareness. So I choose thankfulness before the pain of seeking that which is not, that which is unattainable, or that which is long past.
I think that Kierkegaard was considering that this attitude has to be permanent. You can't praise yourself for having resigned yourself last year, last month or last week. You have to resign yourself each and every time the urge to "dwell on the past" arises. And that urge finds a person when they are at their weakest.
There is some subtlety in how one chooses to look at the past. For example, you say "I choose thankfulness" and you recognize "the life I have is wonderful, and full of joy and satisfaction" which sound healthy. However, this describes your attitude to the present while leaving out your attitude toward the past.
He specifically calls out when a person trying to move on from the past degrades the lost alternative. In the metaphor of the princess, perhaps the Knight forms a negative opinion about her, or about aristocracy in general. He may choose to lie to himself like the fox from Aesop's fables, who when it cannot reach the grapes decides they must be sour. He may get angry with the girl, angry with himself, or angry with the world.
For Kierkegaard the ethical life requires staying true to your desires, even in the face of depravation. One ought not deny that they still want the thing they can't have. They will not rely on self-deception or ignore their true desire. They will realize that these desires will resurface and they cannot spend their life running from them.
All of this is easy to say but hard to live. It ends up forming the basis of what Kierkegaard suggests is a true faith. Fear and Trembling goes on to describe a Knight of Faith that lives beyond the Knight of Infinite Resignation but that is another thing entirely.
This is a contrarian view and I am a biased AI-maximalist. But I actually think these kinds of results are genuinely important.
There is a lot of frustration and even anger over CEOs pushing AI onto employees and some schadenfreude when it goes wrong. But there is some element of "fail fast" happening here.
I am glad wealthy corporations are footing the bill by stretching this technology to its limit. The fact of the matter is, we don't know how effective the best-of-the-best models are at scale.
There is a feeling that once we figure out how to leverage these agents, we'll see explosive growth. It's just going to cost a lot of money figuring it out.
It seems that for now, handing over 100% of code writing to LLMs is going to be too expensive. Cost per token for equivalent code is too high.
I have a feeling it's not going to be magic and will obey the laws of Supply and Demand like all other tech products; further that it's hugely over valued and is going to crash like a meteor before it's over. But we'll all find out together, right?
Yes, right now all we have are vibes/feelings. My point was that one benefit of the hype and the "CEO psychosis" is that we'll find out together fast. Uber, and companies like it, have the money to take the kind of risk that accelerates learning.
And the first data point is in your favor, kind of. I mean, Uber engineers were sufficiently incentivized to use the tokens they were given. It isn't easy to determine what the exact motivation was. What might result from this latest round of CEO backtracking is either relief (don't have to pretend to use AI anymore) or frustration (upset at a useful tool being taken away).
There are two possible stories here. One, they forced everyone to use AI and didn't get enough benefit to justify the cost. Two, they gave the opportunity to their employees to use unlimited AI and those employees jumped at the chance with a vigor that management didn't expect.
All we really know is that value per token must have been low enough to cause this change.
My own anecdote related to this idea was on a playthrough of Skyrim.
I was between objectives and wandering through the map. I came across one of the ubiquitous caves which I decided to enter. I was attacked by some generic low-level bandits and I cleared the cave.
After dispatching the enemies I was looting through the cave and came across some letters. They detailed a tale of a family that came on hard times in a nearby town and were forced homeless by circumstance, how they were trying to rebuild their lives, etc. I looked around the cave and could tell the individuals mentioned in the letter were accounted for in the cave. I mean, they were generic bandit models but the designer had matched them to the narrative.
I thought about the situation. I was this extremely high level wizardy kind of build trekking though the wilderness and I came across an encampment. When I barged into their makeshift home they rightly were like "get out". And then I slaughtered them all with no reason and was now deciding if the clutter was worth packing and re-selling.
I had a similar experience playing, of all things Operation Flashpoint: Dragon Rising. You play as a variety of US Marines helping Russia defend some island from a Chinese invasion so as to control some recently discovered oil deposits.
And at one point while playing this already somewhat implausible situation, I thought "that random PLA soldier I just mowed down would not have, in reality, signed up to be the invading force in a petro-conflict". It kind of killed first-person shooters for me for a while, because I accidentally humanized a poorly-programmed bot in a ridiculous, made-up scenario into making me feel bad for them.
I knew a Buddhist monk in Berkeley in the 1990s who was very interested in "games" to teach Dharma.. Games in those days did not look like they do in the last ten years.. Obviously what is now called AAA games are very much the opposite of Dharma talks..
I read your comment to my partner who pointed out that what your Skyrim character was doing was exactly what the billionaires do - treat us all as NPC’s whose stories don’t matter. I’m glad you had your epiphany.
They will either play ball with the US government on the US government's terms or they will be replaced or destroyed.
It is a misconception for them to believe they can dictate the terms of this technology.
reply