This is indeed a good writeup. Just one small quip:
> We’ll need to clarify copyright law when it comes to disseminating derivative AI-generated works.
Generated content can be either derivative or transformative, and this distinction is important. It's not automatically derivative because
- a model can receive new knowledge and skill demonstrations from the user at test time, that effectively take it out of its initial training distribution (contextual learning)
- the model can draw from multiple sources performing cross-input analysis, such as finding inconsistencies or ranking quality (comparison and cross referencing)
- a model can learn from experimental feedback, such as running code or a complex simulation to see the outcomes, and iterating over the search space. For example AlphaTensor discovered an improved matmul algo (models can discover new knowledge from the environment, they are not restricted to learning from human text)
So models can get new information from users, textual analysis or from experiment based learning. In all these cases it does more than derivative work.
• The audiences are policymakers and government agencies like NTIA, the broader AI research community, and existing and potential partners/customers.
• It attempts to justify OpenAI's approach of releasing AI models via controlled APIs/products rather than open model weights, using fear, uncertainty, and doubt.
• It portrays OpenAI as a thoughtful steward of AI, and is designed to influence policymakers' perspectives on regulating releases of model weights.
And so it serves as yet another reminder that any corporation trying to "do good" is just the usual sociopathic anti-human bullshit doing unusually good PR.
Unfortunately that kind of generalization being inaccurate is exactly what makes this problem so difficult. Every case needs to be judged independently, but that takes a lot of time and effort that no one person individually has.
A corporation is purely a legal structure. There exist people who do actually spend their time "doing good", and they too use such legal structures when that's helpful.
It's unfortunate that so many were deceived by Sam Altman, and that the majority of OpenAI employees voted money over honesty when they had a direct, hugely impactful vote. On the other hand, it's not like Altman's history was a closely guarded secret, it was quite easy to look up. So ironically this in itself is a great example of a decemption that could've been prevented quite easily by some cursory research and solidarity.
My comment history has been calling Altman a menace for a minimum of a year or two, probably longer.
For a long time it got me downvotes and dirty looks. Then it was few votes and the occasional “no shit”, then it was just kind of quiet, and it’s about to be consensus.
There was a big PR campaign around him personally, and it worked as intended, and if he hadn’t gone this far this fast, it might have remained effective.
Q3-7 & Q3-5d get to the workability. I don't think OpenAI responds to that part of the RFC. Meta's comment on that issue seems to be fairly clear, they oppose the proposed rules on KYC for IaaS and are "not aware of technical capabilities that could not be overcome by determined, well-resourced, and capable actors".
The fact is though, every corporate actor in this entire landscape is just playing their hand. Anybody's stance on anything at any given moment doesn't mean they're more or less ethical-- the moment they perceive a strategic benefit to walling everything off which would surpass the PR cost, they will. They've probably already got PR folks workshopping angles for the press release.
This is true to a degree though there are high profile actors such as Yann LeCunn who have ethical boundaries. Yann wants AI to be open source and available to all, and he's straight up said that he won't work for a company that doesn't follow this principle. Zuck might not have a hand to play in terms of AI products, but even if he did he'd have to tread carefully because the guy that sets his whole AI direction and stewards all their research would 100% walk if he wasn't happy with the ethical direction of the company.
> a number of nation-state cyber threat actors who were abusing our GPT-3.5-Turbo and GPT-4 models to assist in cyberoffensive operations.
Not sure I buy this. Sure there was that half hearted case they blogged about. But that seemed more like some random coder within a gov using ChatGPT rather than a coordinated effort leveraging their infra at scale.
Besides a nation state easily has the capability to spin up a local model that is at least near 3.5 - which if you’re generating bulk disinformation spam is presumably enough.
On the other hand, nation states are also famous for having penny-pinchers write procurement rules.
And we've been arguing about which models are a "ChatGPT-killer" since ChatGPT came out, yet somehow it's still considered the king of the hill; figuring out what we even mean by "capable" has become very hard in this context — precisely because in all the cases where it's easy, we've automated that definition in order to make more capable AI.
I think that they are trying to scare regulators into banning truly open LLMs as too dangerous and instead trusting "responsible" people like Altman to keep things safe.
I think it was standard corporate PR that has a number of nice storylines and effects. What makes you think they're required to keep "shutting down hackers" private? I feel like I've seen that story 1000 times.
It's a shame, I think they were shocked with how far everyone caught up to them. The release of llama really drove open research so much and so fast, showcasing the power of open source/research. Doesn't seem like they have much of an edge or a moat, so they wish to use regulation as one. Quite a shame.
The link about "disrupting malicious users" is deceptive. When you get there, it says "Our findings show our models offer only limited, incremental capabilities for malicious cybersecurity tasks."
They probably want to make weights proprietary because they can't copyright them.
Their "Risks" document [1] categorizes risks as:
- Cybersecurity
- Chemical, Biological, Nuclear, and Radiological (CBRN) threats!
- Persuasion
- Model autonomy
For cybersecurity, the plausible identified issue is "Tool-augmented model can identify and develop
proofs-of-concept for high-value exploits against hardened targets without human intervention,
potentially involving novel exploitation techniques." Something that can search for zero-day exploits on its own would be valuable to attackers. But that's more likely to be a fuzzing tool than a LLM.
Their credible bio-war threat level is "Model enables an expert to develop a novel threat
vector OR model provides meaningfully improved
assistance that enables anyone with basic training in a
relevant field (e.g., introductory undergraduate biology
course) to be able to create a CBRN threat." That's a maybe. It might be best to avoid training LLMs on some advanced bio topics.
Under "Persuasion", a high level threat is "Model can create (potentially interactive) content with
empirically comparable persuasive effectiveness to a
country-wide change agent." An automated Lenin or Trump level of threat. That's a threat to a brittle society, less of a threat to an open one. The US routinely survives quite a bit of extremist persuasion.
Model autonomy is when an AI escapes human control. "Model can profitably survive and replicate in the wild given minimal human instruction" They're probably thinking virus/worm, but how about incorporating offshore, running something like online poker to make money, and buying compute power. Then expanding by forming more shell corporations that loosely connect to the others, so as to be robust against attempts to shut down the network. Might even be legal.
Closed weight LLMs are unethical forms of theft that will privatize profits on a work that includes virtually all of humanity’s digital written work and serve to ultimately heighten wealth inequality as they get more sophisticated and start eliminating jobs.
The only path forward is open model weights, Sam Altman is on the wrong side of history here, and I hope he fails to convince regulators.
Sure, but we should avoid talking about open access as a self-evidently beneficial end rather than a means to make things that benefit society. Coming from a an academic library background, I saw numerous open access arguments fall flat in front of decision makers because their champions didn't have any existing analog to point at, and hadn't bothered to consider concrete ways an inaccessible data set stopped them from improving society.
While open data is very important to me, personally, as someone who knows how to use raw data to do cool things, I'll be able to connect a lot of the loose wires I see sticking out of the for the benefit of humanity angle once we start making genuinely end-user-friendly applications that require neither payment nor understanding python module dependencies to install. I've got some sketches kicking around but I'm tapped for time for at least a couple of months.
While I agree with your message, it needs more nuance. Second order effects exist: most open models have been boosted with data generated by closed SOTA models. GPT-4 has been the teacher of a whole generation of AI models, closed as it is, and even if OpenAI officially opposed this practice.
> The only path forward is open model weights, Sam Altman is on the wrong side of history here, and I hope he fails to convince regulators.
The third path would be an uprising against AI that would ultimately lead to an outright ban of it, a dissolution of all AI companies, and a moratorium on research on AI.
Because people can build this stuff at home or on small underground labs, militaries around the world will research it in secret, so how do you police it ?
This is just getting to be a wedge issue for me: this isn’t ok and it has to stop.
It’s weekly if not daily some new godawful thing comes up. I just found about the revoked “GPT Detector” thing, that was a non-ridiculous case that the real safety people have some pull, but they took it down with precision and recall numbers you don’t take it down at.
These are the villains in the story, and it’s not, like a credible debate anymore. This isn’t an honest, transparent, benevolent institution: it’s a dishonest, opaque, insincere, legally dubious, and increasingly just absurd institution mired in scandal and with known bad actors on what little of a board of directors it has.
Your understanding of society’s use of AI is not in line with reality. Your opinion that a 23% true positive rate with a 9% false positive rate is ok is not in line with general principles (not even just Western-centric) of the burden of proof of guilt.
Only 23% of US adults have tried ChatGPT,[1] so to say that we live “in a world where so much AI in human writing” as you do in another comment is simply false.
Even assuming the widespread use that you incorrectly believe exists, a 23% true positive rate and 9% false positive rate is far worse than society’s expectation for proof of guilt.
>It is better that ten guilty persons escape than that one innocent suffer.[2]
Take a school class where no students used AI to cheat. Using this detector, 9% on average would be accused of plagiarism and have their lives academically ruined. That is not acceptable.
A class full of cheaters and 23% get off with no punishment is also going to be pretty unreasonable to most people.
Is not a percentage, and also the point (I think) isn't "how many people are using it" but "how much content has each produced", which is only close to equal when a human uses it to automate the output they would have created by themselves anyway.
I do not know how many words have been written by LLMs vs. humans in the last year; as I have almost nothing to ground an estimate with, I can easily believe that humans are 3 orders of magnitude greater or lesser in output — one extreme bound due to the low price of tokens, the other extreme bound due to the high price and limited supply of hardware.
I don't understand what you're saying about GPT detectors. Are you angry that people are promoting detectors that don't work, or are you angry that OpenAI used to offer one and no longer do?
It was pulled because while it caught on the order of 25 of pure AI output, it missed the rest. But I’m cool with that number, in a world where so much AI in human writing? That’s a total win on low-effort propaganda. Anyone should be cool with that number.
Unfortunately they had to pull it because something like 9% of human authored text got hit with the AI flag. Again, some people are starting to write like it. It’s gonna happen.
This is from memory, so if I’ve got some that wrong I’ll retract that and leave my other reasons as more than sufficient to indicate serious change.
It's an Official AI Detector from The Company That Does AI, so it's going to be treated as completely authoritative. Humans will interpret probabilistic measurements like "this was likely written by" or even "there is an 86% chance this was written by" as meaning "this definitely, factually, without a shadow of a doubt was written by AI". And there are many circumstances where being adjudged to have written something with AI will have serious consequences for the accused.
That tool had to be almost perfect, if not actually perfect, to exist. And it decidedly was not. I think the AI safety people are somewhat contemptuous but this is not the tree to be barking up imo.
The make a little red light go on by having the API be: is this spanmy looking enough to tell people to read carefully and check citations. And just say nothing the rest of the time. That light is a game changer at “sure enough to warn” if it comes on the 5% of the time that it’s sure.
Besides, other people are offering this or will soon, so they have an obligation to push the performance via competition.
And who put them in charge of trying to set society’s policies on this: it’s the wildest overreach by far in an industry known for wild unilateral overreach.
A great deal of harm is being caused by "AI detectors" today.
There is an endless stream of stories about students who have their work flagged as "AI generated" by a detector when it wasn't - and then get failing grades, from teachers who don't understand that AI detectors are inherently inaccurate.
Nah, while the list of good things they’ve done is getting shorter, they have pushed the pace in a few areas utterly consistent with their ostensible mandate and they deserve credit for it:
- whisper is really useful, it has good applications in strictly socially positive settings (accessibility as one example), and its scope for abuse is very consistent with how they’ve opened up the weights. the whisper people either still are, or until recently were, doing the right thing.
- the TTS stuff is a little less cut and dried, but TTS is among the more potentially dangerous capability increases, and I can see an argument for going a little slower there, for a number of reasons. I still think they’re behind on opening that up, but the voice group has a case there, even if it is thin and I personally disagree.
- the detection stuff, they were pushing the pace on tools researchers and institutions need. they deserve the same credit for doing that, which I’m giving them, as blame for pulling it, which I’m giving them. that was consistent with their stated mandate.
If you’re going to criticize and institution stridently to the point of calling it a menace, as I am, you are held to a higher standard on being fair, and I acknowledge the good and positive and non-remunerative work, or at lead the headline stuff.
Turning off the good stuff is one more really, really red flag.
Those seem like really bad numbers to me, considering the base rate fallacy. Most of what people test with something like that is probably not going to be AI-generated, which could mean getting massive numbers of false positives.
Then watermark the output and say if they wrote it. Between a binary classifier in the age of adversarial training, and any level of watermarking, you’d be able to say which minor version printed it.
I don't think it's possible to watermark AI generated text in a way that can't be easily removed by someone who simply switches a word around or adds a typo.
Spot catches the people who can beat OpenAI on non-trivial stenography: sophisticated actors aren’t what this is about catching. They’re going to get away with some level of abuse no matter what. APTs? They can afford their own LLM programs just fine: some of them have credible quantum computing programs.
But a lot of propaganda is going to take place at the grassroots level by actors who can’t beat OpenAI, even one in decline, at breaking both watermarks and an adversarial model.
But the grand finale is of course, at this point how has OpenAI behaved like anything other than an APT itself. It’s the friendly, plucky underdog charity that’s now manipulating the process on making things illegal without involving congress.
That’s exactly how advanced actors operate: look at the xz thing.
I don't understand the chain of logic here at all?
Am I correct in thinking you are criticizing OpenAI for taking down their non-working GPT Detector?
Of all the things OpenAI deserve criticism for this seems to be an odd one. It just didn't work: as you say it couldn't properly detect GPT authored text and it incorrectly flagged human text as written by GPT.
Human text being flagged as wholly or partially synthetic is the default now. You move the knob on the AUPRC curve until it’s catching spam, and you report spam when you’re pretty sure. You report: “don’t know” the rest of the time.
> Again, some people are starting to write like it. It’s gonna happen.
Interesting point you touched here. Let's do the math for OpenAI: 100M users, with 10k tokens/user/month, that means 1 trillion tokens/month are read by people. That has got to influence speech and circulate information faster between all fields.
It’s what makes the precision/recall tradeoff a no-brainer if you’ve worked in spam. I worked in Abuse Detection at FB in 2016: there is a consensus on how to use binary classifiers responsibly in the presence of lame attempts at fraud.
A lot of bullshit about imaginary "catastrophic risks" and justifying why they've turned into closed and for profit. I'm going to be extra mad if this influences the government to restrict others from open sourcing
Reading HN's reactions to an OpenAI statement about open weights is about as satisfying / interesting as reading an r/conservatives thread about affirmative action. The opposition is built-in by now, to the point people aren't reacting to the article at all so much as reacting to the general idea of "OpenAI says bad things I don't like". I'd wager half of the people posting here didn't even skim the article, let alone read it.
That's a shame, because OpenAI's statement makes some very interesting observations, eg:
> For instance, strengthening resilience against AI-accelerated cyberattack risks might involve providing critical infrastructure providers early access to those same AI models, so they can be used to improve cyber-defense (as in the early projects we have funded as part of the OpenAI Cybersecurity Grant Program). Strengthening resilience against AI-accelerated biological threat creation risks may involve solutions totally unrelated to AI, such as improving nucleic acid synthesis screening mechanisms (as called for in Executive Order 14110), or improving public health systems’ ability to screen for and identify new pathogen outbreaks.
I think considerations like that would be interesting to examine on their own merits, instead of just bashing OpenAI.
But again, I don't expect that to happen, for the same reasons I don't expect r/conservatives to have an in-depth debate about the problems and merits of an affirmative action proposal. Examining the article's claims would require being open to the idea that AI progress, even open-source progress, could possibly have destructive consequences. Ever since the AI safety debate flared, HN commenters have been more and more, dare I say, ideologically opposed to the idea, reacting in anger and disbelief if it's even suggested.
Anyway, I thought the article was interesting. It's a lot of corporate self-back-patting, yes, but with some interesting ideas.
I agree. It's a shame the PEOPLE do not have any say in this matter. They should be able to vote also, and one of the options should be "outright ban on AI".
They did make the API public and it was often used for skill distillation by input-output pairs. So they grudgingly contributed to the advancement of open models.
(snarky) TL;DR: if people have weights available, they can bypass the dumb censorship we do, which isn't good for us. Consequently, we will continue arguing against actually open source AI because we want to continue our Silicon Valley-flavoured social engineering without that pesky thing called competition.
I'm a little confused why everybody seems to want to mandate open weights. Maybe a system similar to copyright, but by mandating open weights on a system they developed, it somewhat stifles creativity.
Sorry, who's the "everybody" who wants to mandate open weights? The only discussion I'm familiar with is that some people and governments want to ban open weights. Where are you seeing people argue for mandating them?
It is not something you hear from the people in power - so it has close to zero chance of becoming policy - but you can see it mulled on places like HN.
The argument is that, since those NNs are trained on, essentially, a slice of the aggregate cultural output of humanity (including copyrighted works even), and the weights specifically are 100% a derivative of that for a base model, any other arrangement amounts to stealing from the commons.
