I would rather my reports tell me to fuck off than to generate something telling me to fuck off in polite but insincere terms full of emojis and em dashes. Honesty is valuable.
I too would much rather that, because then I get to know them as unhelpful whining complainers and fire them. That's not a great outcome for that person though.
One solution I haven’t seen recommended much is to have a Claude instruction/skill that explicitly audits the diff of every upgrade, and force this manual audit as part of your upgrade workflow. This seems like it would work pretty reliably.
This is what many AI supply-chain security startups (like the one that posted the article) are already doing with all NPM packages, so save yourself the Claude tokens. All of these compromises were detected within minutes, but it takes some time (<1 hour) for NPM to unpublish all of the affected packages.
Sorry for my ignorance, but then couldn't we build this into NPM itself? So before a package is publicly available it would be quaranteened and checked.
Super dumb question as someone who has been using some form of AI for dev since 2023:
How does having an AI audit external code help? Can they not be prompt injected to ignore a malicious change?
I guess I am sort of concerned that they are a pretty thin layer and even if you put "DO NOT ALLOW PROMPT INJECTION", it's a bit like saying "make no mistakes". There _is_ a priority between `system` and `user` level messages as I had recalled, so a specifically made tool that has its own system prompt should prevent injection while asking Claude CLI could still allow for prompt injection.
There are prompt guard classifiers that can detect prompt injections, but they are not perfect (false positives, obfuscation) and should be only a part of the defense.
The concern is real and unsolved. I think security researchers have an advantage here because they still can fall back to manual audits if their automated analysis (or scores thereof) is off.
The $200/mo Claude plan is not available for every employee. You can buy the $100/mo plan for up to 150 people, and then you have to switch to API billing.
Yes, token use can be tracked the same way, just have to MITM everything. The ToS is a non-issue as it's not a legal issue, unless you plan to do business with Anthropic, not really an issue as you can always go to API later-on, in which case, Anthropic can't supposedly "ban you" as they are saying they don't record prompts.
This is probably the fact that Anthropic's website serves different ToS based on your source IP. European ToS for the subscriptions forbids commercial use.
Those plans are going the way of the dinosaur, ai provider loses money on them. Most enterprise offerings are already there, Anthropic changed theirs to $20/seat plus token usage a couple weeks back
You're welcome to have opinions on that, but the answer to the person's question is objectively compliance. A corp can't get enterprise features like ZDR without switching to token based billing. That's why they aren't using subs.
This isn't some kind of new thing. There's always been an enterprise tax, like SSO.
Today's tech echoes 1960-1970 mainframe era: very centralized around a handful of companies controlling "massive cloud compute" in bespoke mainframe-like topology.
All of that will all be legacy in a couple of years. Today's B200 clusters are tomorrow's e-waste. Decentralization might happen gradually or abruptly. But to me it's obvious that we'll be thinking of high-tech tensor processors and GPUs the way we thought of individual transistors and tube amplifiers in the 1980s.
If AI turns out to be the revolution it purports to be, than the underlying hardware will change much more rapidly than it did with ICs and microprocessors in the late 1970s. Today's hot is tomorrow's junk.
Hardware depreciation timescales are actually getting longer, not shorter, because frontier hardware like B200 clusters is highly bottlenecked. It's not just a RAMpocalypse out there, we're seeing early signs of production bottlenecks with GPUs and maybe even CPUs.
One thing that is potentially different this time is that Moore's Law has stopped scaling. Computers aren't getting smaller exponentially. They're getting bigger with multiple chips glued together to make up for Moore's Law.
It's basically converted sand. Most of that conversion happens in Taiwan at the moment. Which is considered, by China, to be one of their provinces and as a protectorate by the usa. Hence the interest in that region....
West has been incentivized to build their own fabs for years but still fumbles that effort. All the billions spent hardening the south china sea and taiwans chip manufacturing from the future chinese invasion would have probably paid for a lot of manufacturing capacity stateside.
Mainland China is growing its own RAM manufacturing capacity. They are too tiny to make a real dent into the RAMpocalypse yet but this can potentially change.
> Every week, somewhere between 1.2 and 3 million ChatGPT users, roughly the population of a small country, show signals of psychosis, mania, suicidal planning, or unhealthy emotional dependence on the model.
> Why is mental-health crisis not a gating category, the kind where the conversation stops, full stop, and the user is routed to a human?
Well, obviously “routing to a human” is not feasible at that scale. And cold exiting the conversation is probably worse for the user than answering carefully.
I want to use an analogy here. The same arguments are often made about cleaning up environmental damage. So either make the companies doing the polluting pay for the costs themselves or if we care so much about them being profitable then we subsidize them by paying for those cleanup efforts out of taxes. Doing nothing is a worse form of subsidy as it not only costs more (in literal dollars) but shoulders that costs onto the people with the least ability to pay for it. The problem is you're treating "doing nothing" as having no cost. It has a high cost, but the cost is also highly distributed.
So if it is not scalable, then why subsidize them? This is literally a tragedy of the commons situation. Personally, I'm in favor of making the people who make a mess clean up that mess. I really don't understand why this is such a contentious opinion.
We keep letting em get away with the same old excuse: "The company can't fix X problem because it operates 'at scale' and you'd need millions of humans to perform corrective action Y at that same scale!"
We keep letting people get away with a lot of thought terminating cliches. I've learned that people are desperate for an answer, but incurious to the answer.
I guess to be fair there's no "the" answer as reality is incredibly complex, but that's a whole other conversation
Tech companies will pull trillions of dollars out of their asses when the problem is boosting ad revenue or automating people out of a job. But when asked to deal with the crisis they invented and dumped on society the answer is “that’s impossible, doesn’t scale”
Figure a "mental health crisis" human conversation takes 30 minutes. Three million incidents per week would require 37,500 qualified mental health counselors on the phones working a 40 hour shift that week. Figure they make $75k/year each. You're now spending $3 billion per year on crisis response, and you're employing like 10% of all of the health counselors in the US. And all you're providing is 30 minute chats.
> You're now spending $3 billion per year on crisis response
Honestly? That's really affordable[0]. That would be cheap if these were just for the US but it looks like these are global numbers. We spend $2bn/yr alone on "BREASTFEEDING PEER COUNSELORS AND BONUSES"[1]. I mean let's be serious, even in the article that OpenAI published says that it is a small portion of their users. So it doesn't "need to scale" as the scale is relatively small. But just because it is small doesn't mean it is unimportant.
$3bn/yr is a lot of people money, but it is nothing for government money.
Edit: Last round of OpenAI funding was $122bn[2] and in the same article they are saying that they are generating $2bn in revenue per month. While that's not profit, it is worth mentioning that what you are saying "doesn't scale" is about 12% of the revenue of something that does scale. A single company. And mind you if we implemented what you're proposing it would be available to all the AI companies and more. Making it only a smaller drop in the bucket, not larger.
[0] Not to mention that better mental health care services will result in savings elsewhere. It's always way more expensive to fix a broken pipe that's flooding your house than it is to fix a pipe with a small crack. "Don't fix what ain't broken" is used too broadly. Maintenance is always cheaper than repair, but people just can't seem to understand this.
Mark Zuckerberg can spend $80B on the failed metaverse experiment, but can't spare some relative pocket change on solving the psychosis issue his products caused.
Well, Metaverse could have been a profitable business if they had delivered a product or service (other than Oculus, which is a decent though not $80B product) which regular people wanted to use and were willing to pay money for.
That underinvestment is the entire reason their stock prices are so high. This is effectively pollution of our information economy and environment, and the costs are offloaded to society.
The fact that we have the first generation with lower education attainment is not a problem for their stock prices or operational profit.
Tech has ungodly profit margins, because they are all about scaling without having to bring people in. Sadly there is no such thing as a free lunch, and if firms are made to clean up their mess?
Oil spills affect Oil firms more than Tech fallout affects Tech firms.
> This is effectively pollution of our information economy and environment, and the costs are offloaded to society
Tech firms are remarkably successful at offloading costs as externalities and avoiding any sort of liability. '
What other products do you buy that require mandatory click-through "agreements" absolving the manufacturer of any kind of responsibility? Why is that legal?
Mapping and photographing every road on the planet? Easy.
Not manipulating our chatbot users into psychosis and suicide or worse? No way can't be done.
So let's summarize: these "people" are allowed to induce mental illness in the general population with no repercussions. Let's speculate about who might benefit from a population with a significant amount of mental illness? Hmm, I wonder who would benefit? Could it be the people at the top that like to treat the bulk of humanity as cattle?
Many of the problems with LLMs may be structural and intrinsic due to the way they work (probabilistic text generation) and their training data (often human-generated text that incorporates many features of human discourse that are undesirable in machine-generated output.)
The continual failures of "guardrails" show that it's incredibly difficult to get these systems to behave in reliable and predictable ways; unsupervised interactions with them are intrinsically unsafe, and should be treated as such.
Presumably Meta and others are trying to detect and prevent bad output and pathological interactions, but that detection is unlikely to be 100% accurate, and we've seen what the failure modes can look like.
I'm not talking about the edge cases where it goes off the rails, I'm talking about the way it normally conversates. The way it was trained to do so through RL.
Oh, are you saying that incentivizing engagement (usually to increase ad views and revenue) also implicitly increases bad behaviors and outcomes, and that is why it won't be fixed? That sounds plausible. Even before LLMs the engagement/attention economy had strong negative effects that companies didn't want to address.
I don't think it's obvious that routing to a human is infeasible. I'm sure many local authorities, health agencies, and non-profits would be okay being routed to. Additionally, I'm sure many of the users are the same week over week, so giving them long term care would reduce the total volume. Finally, there is a long gap between psychosis and emotional dependence, so there could be some triage to make sure those most in need have human intervention.
None of them are resourced enough (globally) to do this.
Safety is my area, and I interact with help lines and safety networks. Most of the time they are getting crushed and are underfunded. Offloading the work to them is hard and it requires investment in staffing, people, and organization.
It’s currently cheaper to do some amount of donation and support to such orgs, and bury the issue, than it is to actually deliver / invest in the degree of support needed.
These are also long tail problems, so solutions for a case can take years. For example if you are a woman in Pakistan who has been a victim of revenge porn, you are going to be spending a good chunk of your life trying to get those images/videos taken down from sites that are not based in Pakistan.
This is only an example of the types of problems that these helplines will have to triage. There will definitely be cases that can be resolved with a single call.
There isn’t any money in it, and it is seen as support work.
Step 2: 90% of users stop sharing their negative thoughts because "talking to a machine, not a human" was the entire selling point, giving them a sense of privacy and safety
Step 2: engage ongoing trauma, grief, stress, paranoia, or reality-breaking episodes haphazardly with no clinical insights or boundaries or pre-screening, provoking new and occasionally catastrophic reactions, while holding full liability
Step 3: get mercy-murdered in the middle of the night by corporate’s lawyers swinging batteries in socks
Well, then maybe you can't scale it as a free service with self-serve signups. Maybe you need to gate who you allow to use it and pace how intensely they can engage. Or maybe you need to look for other solutions.
Yielding to "not feasible at scale" is exactly how we ended up with a lot of today's most pressing and almost intractible problems, from social media's ills to person and society straight through to enshittification and non-repairability.
> ...straight through to enshittification and non-repairability.
funny as "enshittification" was the topic of a 99% Invisible pod just a few days ago and I also was listening to the new Stewart Brand book that Stripe published. i fixed a Norwegian desk I bought a decade ago on Valencia. happily not feasible at scale but neither was how i broke it :)
If causing problems at scale is possible, being held accountable for said problems is also possible. Not attempting to deal with the crisis they've created should, in my personal view, result in corporate forfeiture and the immediate sell off of assets and destruction of whatever led to said crisis, and a charge of gross criminal negligence filed against the C-suite and board of said companies.
They're in a tough spot. They can train out the pretending to be human, sycophantic, lying, all-knowing aspects of the model, but this is how they got all the investors and CEOs on board the hype train. Psychosis is the product.
This conflict is popping up everywhere. There is a push by a lot of companies to allow agentic use of their services (and new companies explicitly offering "X for agents"), ignoring the fact that "agent" means the same thing as "bot" which we've spent the last couple of decades actively filtering out. Will be interesting to see how it plays out.
The future is the internet will be entirely bot activity and humans will ether be strapped in to the metaverse reels ai slop feed or they will be outside interacting with people in person again. Both of these seem like likely futures and probably both at the same time.
This reality also crystalized for me earlier this week when I saw a post about unchecked AI slop videos about WWE being posted to YouTube. Many of the videos suffer from the LLM stroking out (for lack of a better term) and devolving into mumbling, screaming and white noise. Yet, the comments are replete with obvious bot content which doesn't mention this at all and talks past the larger, flimsy narrative on display (i.e. AI-generated), anyways. We're exhausting our natural resources and reducing quality of life for a great number of real, live people so bots can talk past each other on YouTube.
So, if you're looking for me, I'll be hiking while it's still legal.
You better mean “hiking” as in through the metaverse forest strapped into your corporate-sponsored VR headset, because outside time is for citizens only, friend.
My wildest dream: we make a superintelligence, which destroys humanity to free up resources for it to make and consume an endless stream of impossibly cute kitten videos.
And before anybody replies: no, I don't mean "and puppy." They're just not as cute.
fyi you probably mean “albeit”.
reply