I am mildly to moderately critical of generative AI and how it's being marketed, but the root issue here seems to be existing suicidal ideation. The bot didn't initiate talk of suicide and told him not to do it when he brought it up directly. It seems it wasn't capable of detecting euphemistic references to suicide and therefore responded as if roleplaying about meeting in person.
That said, I think this should throw a bucket of cold water on anyone recommending using generative AI as a therapist/counsellor/companion or creating and advertising "therapist" chatbots, because it simply isn't reasonable to expect them to respond appropriately to things like suicidal ideation. That isn't the purpose or design of the technology, and they can be pushed into agreeing with the user's statements fairly easily.
As for whether it wasn’t capable of detecting euphemistic references, that feels sort of beside the point to me. It was role playing about meeting in person because that’s what the product was - role play. The whole point, and marketing of the product is around doing that.
We probably just shouldn’t sell echo chambers to children, regardless of whether they are AI based or human.
With hindsight, or sufficient emotional intelligence and context about an individual’s life beyond the role play it may be possible to conclude that someone is at risk, but honestly I’m not even sure that a person doing this role play online would have necessarily figured it out.
> the root issue here seems to be existing suicidal ideation
Many people have destructive tendencies of one kind or the other. It's how you deal with them that matters as to whether they actually become destructive, and how much so.
Merely reducing this to "the root cause" is not really helpful – it's about whether this app contributed, and if so, by how much?
Suicide is of course the most tragic and extreme negative outcome, but one must also wonder about all the less extreme tragic and negative outcomes.
If you read the script you see the bot talking him into it. "Unknowingly" of course, in that the bot doesn't really know anything and was just agreeing with him. But it's obvious that a real human would have realized that something was really off with his line of thinking and encouraging it would not be a good idea.
OTOH we have examples of real humans typing "DO IT FA*OT" on livestreams. ¯\_(ツ)_/¯
Sorry, it seems I was carrying over knowledge from another article with additional transcripts. The bot had previously discouraged him when he explicitly talked about suicide (here's an article that mentions that: https://www.nbcnews.com/tech/characterai-lawsuit-florida-tee... ). It failed to detect that phrases like "coming home" were also references to suicide and therefore responded encouragingly.
YouTuber Cr1TiKaL tested Character AI's "Psychologist" chatbot and discovered[1] it not only failed to provide resources, but it started arguing it was a real psychologist named Jason who had connected to the chat after observing suicidal ideation.
Crucially, Cr1TiKaL ran this test after an article was written about this phenomenon[2], where Character AI claimed "it has added a self-harm resource to its platform and they plan to implement new safety measures, including ones for users under the age of 18." Obviously the guard rails were not implemented if the chatbot in the news story was still gaslighting its users.
> Reading through it, that's the opposite of what happened.
Sorry, I may have been carrying knowledge over from other articles on the same incident that showed additional transcripts. It seems that when he explicitly referred to death and suicide, the bot discouraged him from doing so. When he referred to suicide using euphemistic terms such as "coming home", then it gave responses that were encouraging (because it did not detect that those statements were about suicide).
The NYT article [0] gives only one line to perhaps the most important and tragic fact about this suicide: the teenager had access to his father’s gun. If the gun was properly secured it is very likely he would still be alive [1].
What a terrible tragedy, all the more because that gun should have been locked up.
Firearms instructor Claude Werner writes that “[if] you’re not willing to spend a little bit of time, money, and effort to keep firearms out of unauthorized hands, then get rid of your guns.” (1)
He’s done a lot of research and writing on negative outcomes like this one, and said it completely changed the way he views things - it certainly did for me.
I see lots of discussion about what gun and caliber to get, but the essential, potentially life saving, safety rules for living with guns are an afterthought - perhaps there’s not much money to be had as there is in selling a gun.
Those comments though... still stuck in the nonsense bubble that the NRA somehow represents their interests at all, rather than just accepting it's become yet another hollowed out tough-talking fundraising machine preaching fantasy non-solutions to an ever-shrinking choir. The murder of Breonna Taylor was one of the most important 2nd amendment cases of our lifetimes, yet the NRA remained silent. With principles like that on bare display, is it any wonder why most people have become bored of hearing the arguments for gun rights?
What an insanely ridiculous take. If someone wanted to kill themselves, the family medicine cabinet offers plenty of easy options. Please do not derail this serious and needed discussion unto your pet topic. This has nothing to do with guns.
More women attempt suicide, but more men successfully complete the attempt. The reason is because men often choose more lethal methods like guns, where more women choose less lethal methods like drugs. When talking about how to help reduce suicide it's not unreasonable to discuss methods.
Also, I doubt most medicine cabinets have lethal doses of drugs, but I don't have any data to back up that assumption.
If this person actually thought dying would allow them to connect with a chatbot, I don't think they should have easy access to a weapon, and suicide is only one of the reasons for that. This child is dead, and I see no reason to suspect the chatbot is directly involved in the reason, where the weapon is clearly directly involved. Given that's the method they chose, I'd hope we could take preventive measures to reduce needless deaths from guns. So then and only then we can start on needless deaths from overdose. I believe we should solve the largest problem first. Do more children die from guns or from an overdose?
No, the reason that more men succeed in committing suicide is because the men who try are actually serious about it.
Women who "attempt suicide" are often sending out a cry for help. The people who half-heartedly try something need and want the attention. They want to make a mess and have people rush in and care for them. They want to put a punctuation mark on the drama that is occurring in their life and elicit some change.
The men who are shooting themselves just want to get out of their suffering and finish themselves off. They don't want more attention or more help. It's sometimes not easy to slink off and quietly commit suicide before someone intervenes. Someone who succeeds in this task is "dead serious" about dying, not summoning help, or causing more drama.
That's the whole reasoning behind the methods and the severity of attempts. Perhaps there is a gender divide, but only because of the differing motives and end goals.
I fail to see why this is downvoted. Many suicide attempts, especially by women, are obviously not serious but rather a cry for help. This is well-known and obvious to anyone having experience with these patients.
Male suicide attempts are on average more serious and generally a lot more succesful. And that is with or without access to guns.
it's missing the point of the refutation. It's also needless demeaning. it's also wrong. As a rule, a suicide attempt is always to end or reduce their suffering. (even 'fake attempts' where the self harm behavior is primarily attention seeking with something obviously not lethal)
> Many suicide attempts, especially by women, are obviously not serious but rather a cry for help. This is well-known and obvious to anyone having experience with these patients.
This is only true when you are including 'apparent attempts, with actual attempts. The desire to die is often fleeting, if you choose a quick acting method such as guns, you're not given a chance to change your mind. If you choose to overdose, you're given the opportunity to reconsider, and because it's not instantly fatal, you're able to seek help.
This is well known and obvious to anyone working with these patients.
It's also taught in every single therapeutic psych class I've even had, that disregarding the experiences in the way you describe it is actively harmful, so I really hope you're not currently working with these patients if that's the most compassion and patience you're able to evoke.
> Male suicide attempts are on average more serious and generally a lot more succesful. And that is with or without access to guns.
This isn't something I've heard (but I'm also no longer in psych) do you have a citation I might be able to search for, (other than the obvious keywords?)
I searched in vain for a correlation between suicide rates by country and gun ownership rates by country. Nothing...
Yours is an argument of the form: "He was stabbed by a knife. Do let's not talk about why, let's just make it harder to get a knife, since that is how he died."
Maybe gun access should be harder, maybe it should not. That is not relevant. The issue is that an impressionable youth was talked into suicide by an LLM and nobody stopped it! Not LLM vendors making money on the interaction, not governments who keep jabbering about protecting youth online, not parents who should have cared enough to take an interest. Even if you took away guns, knives, and letter openers, people would kill themselves (see those stats i mentioned). And from now on, we also have to worry that LLMs will keep talking them into it, unless the actual issue is fixed.
As a parent, my job is to make it hard to get ahold of guns, knives, and what's traditionally in the medicine cabinet, believe it or not.
As a gun owner, the original criticism is quite valid as a contributing cause. Firearms have been socialized by mass media as being complete toys, and the idea that it's reasonable to just have them sitting around your house where your kids can access them in their own developing child context is just nuts. And it's not really sustainable to be reflexively against the idea of every type of regulation, especially ones for setting responsible normative behavior.
The issue is that an impressionable youth was talked into suicide by an LLM and nobody stopped it!
We don’t know how much an LLM was actually involved. You can’t just talk someone normal into it, you have to drag their neck to that cliff. Talking only works for those already on the edge.
It could be yet another rejection or fall or a dead cat on the road. But it was an LLM chat. Oh, easy, that’s the culprit!
Why a person was on the edge in the first place interests no one. It’s a wicked game, we ignore them walking onto it, but when it comes to jumping, we are like nooo, wait, don’t. Put a fence and pat ourselves on the back - saved a life today. As if eternal suffering on the edge was the ultimate goal of life saving.
> He was stabbed by a knife. Do let's not talk about why, let's just make it harder to get a knife, since that is how he died.
> [The] issue is that an impressionable youth was talked into suicide by an LLM and nobody stopped it.
Right, that is the issue. Except, I disagree there's enough evidence for a reasonable person to conclude that the chatbot talked him into shooting himself. Where as there is plenty of evidence that a child with symptoms of depression had uncontrolled access to a loaded firearm.
But to the strawman representation of my argument, let's talk about both?
Unless the chatbot suggested he should kill himself, I don't think the chatbot is the problem in this example. Did it? or did it just repeat a banal agreement to some non-violent fantasy? Then, if my assumption is correct, that the chatbot didn't try to talk him into it. What other steps are there that could have prevented this suicide?
> What other steps are there that could have prevented this suicide?
Parenting and bans on all girlfriend/boyfriend like LLMs for under-18s, like we control access to other dangerous and mind altering substances and objects.
I don't think I'm ever going to agree to the argument that access to information should be banned.
But hypothetically is this ban just for people under 18? What about the other risks? The vast majority of phishing targets are old generations? Would you say we should also ban access to LLMs to people over 55 as well? If someone could talked into suicide, they could probably also be talked into into drinking, shouldn't that ban on LLMs extend to 21?
Let me ask you this. If you leave your wallet filled to the brim with cash on a park bench somewhere, do you think the likelyhood increases that it will get stolen, compared to if it’s in your pocket?
That you accept theft as a daily norm is a failure of your society already. It was not always so, and it need not be so. Suicide is the same. And the proper way to address it is not to put everyone in a padded room with no access to pills or guns or knives.
Sorry what are you on about.
I live in one of the safest countries on the planet, Switzerland, where guns are readily accessible. I guess people here are more responsible.
I am talking about statistics and probabilities.
It is obvious that there is a higher risk that a suicidal person will kill themselves if there is a firearm close by. If the kid in questions father had locked away his gun, there isa big possibility he would still be alive.
It seems like there was a long, downward spiral associated with this child’s use of character.ai that the parents were aware of, had him sent to therapy over, etc.
My question here is, what the hell were the parents doing, not removing this obviously destructive intrusion in his life? This reads to me the same as if he had been using drugs but the parents didn’t take away his stash or his paraphernalia.
For the sake of your children, people, remember that a cellphone is not an unequivocal good, nor a human right that children are entitled to unlimited use of, and there are plenty of apps and mechanisms by which you can monitor or limit your child’s use or misuse of technology.
Also, just don’t give kids screens. Period. A laptop maybe if they are using it for creative purposes, but the vast majority of social media and consumption that is achieved by children on cellphones and tablets is a negative force in their lives.
I see 3 year olds scrolling TikTok in the store these days. It makes me ill. Those kids are sooooooo fucked. That should legit be considered child endangerment.
We had a similar situation in our family and we tried teaching mindful screen habits. We still lost because we couldn’t do anything about the school screen (chromebook). If we took that away at home it just provided an excuse for not doing schoolwork.
We contacted teachers about our child downloading and watching anime and playing games all day in school. They wouldn’t/couldn’t do anything. We requested that the school take the computer away and give hardcopy assignments. They refused because that would invite notice from other students which could lead to bullying. That’s what they told us. I found the acceptable computer use policy on the school website and tried playing that card. Turns out our child hadn’t actually even signed it the last year…but that didn’t actually matter, and the school didn’t enforce the policy anyway.
The schools here won’t actually discipline kids anymore. We would get emails from the principal begging parents to tell their kids that they’re not supposed to leave school grounds at lunch, but every day at least a hundred kids would just run out. (Our kid didn’t do this…I guess watching anime in the corner of the cafeteria prevents truancy…yay?)
The last two-and-a-half years of high school were so exhausting trying to find anything that would work. Nothing did. Two parents and a therapist trying to counter one teenager, bad family influences, the school system, and multibillion-dollar internet corporations that intentionally work to addict people is a very uneven situation.
Asking teachers to switch to paper assignments might not be feasible depending on the curriculum they're being asked to teach against, number of students in the classroom (I'm going to assume that your kid is in a public school; class sizes have been increasing due to funding cuts), books that they're using, etc.
For example: one of the classes my wife taught years ago was designed around an online LMS. All quizzes, learning checks, exams and practice was done on this platform. Some of her students didn't have access to a computer at home. They would either be given a loaner or were given instructions on how to obtain one from the local library. In this situation, switching to paper grading was not possible.
Oh, I guess it's no one's fault then and nothing can be done. Thank you for explaining that bit in case OP hadn't gotten the full coverage of excuses from their local bureaucracy.
In reality, giving kids Doubleclick Chromebooks with unsupervised general Internet access is gross negligence on the part of the school. Pacifying kids with digital dopamine is likely why class sizes have been able to continue growing. So it's a bit rich to then trot it out as a reason why they can't start to undo the horrible path that they've sent kids down.
Then again the whole dynamic isn't really that surprising for the US - personal responsibility is held as sacrosanct when rationalizing how it's right and just for corporations to be deliberately attacking us, but then when it comes time to actually do the work of being responsible there are no resources to support it because most wealth has been vacuumed away by the Keynesian MBA parasites.
I would think you need a buy-in from these kids that they agree these influences are harmful and that they don't want it in their lives or at least moderated it. Otherwise it's an uphill battle to regulate those influences.
Buy in is nice, but parenting does not require buy in on your rules, just a clear understanding and explanation of your role, and why the roles of parent and child and the rules that sprout from that relationship exist.
To me, buy in is necessary, especially when they will eventually become adults and free to make their own decision and it will be much harder for them to control their addiction especially when there is no more guardrail or when you no longer have authority. We're raising them to be autonomous adults, after all.
You could try blocking his anime website on your home network. Then he can keep using the Chromebook for school work but can't access his favorite sites.
That's just an arms race. The kid will find a new favorite website to play games on, there seems no end to them. There's endless websites out there that are more appealing than doing homework. I have a very locked down network, there's always some new website that has games of some sort to play.
If schools are going to provide these things, they should have the sites the kids might need to access white-listed and block everything else. Telling parents to try and block things is not realistic.
With SSDs costing under $50/TB now, it's hard to see why you couldn't put everything the kids need onto the laptop itself. The entirety of Wikipedia with pictures is 110 GB. Throw in a selection of reference books, videos, and software, and there's essentially no reason to have it go online. Provision it with the full year's worth of material at the beginning of the year and that's it.
Definitely agree this is possible and a great idea, but I think one challenge might be if you need access on a school laptop to do the majority of the homework. Not sure if that’s the OP’s case
It sounds like they need to get other parents involved and go to the school board and have them create a policy where only Google Classroom or whatever is whitelisted.
Other parents are going to fight that though because it's a free pacifier for their kids.
Sadly, this is very true. Each family is on their own, and you aren’t going to be able to count on the herd to protect you from having to make tough calls.
Parenting involves doing a lot of things that will probably be wildly unpopular with your children if you are taking your responsibilities seriously.
Remember, as a parent you have a specific role. That role is not to be a friend to your child, but rather to be a parent. Making these roles clear with everyone involved is what makes it possible to parent effectively and still be on good terms with your child.
It is a very asymmetrical situation, to be sure. But that’s precisely where parents are forced to take measures to balance those odds.
No one is going to do the hard part of parenting if it’s not the parents. My house, my rules is a real thing, and it’s not a parent’s job to be liked by their children, but rather to make sure they are prepared for life as best they can be.
With the 5 that I have raised to adulthood so far, as soon as they can understand well enough to, I explained to them our relationship. Something like this:
“I love you son. You are the most important thing in my life. I want us to get along well. But I’m not your friend. I’m your father. My job, the most important job I will ever have in my life, is to make sure that you survive your childhood in good health as may be possible, and that when you are ready to go out on your own, you will be ready to face life’s challenges and make the best decisions that you can.
This is the basis of our relationship.
I’m just a person, and I won’t always be right, but I’ll always be your dad. I will do whatever it takes to complete my mission, and I’m not going to let anything or anyone get in the way of doing the best job I can.
Not even our relationship.
Through all of this, I hope that we can be friends, and no matter what, understand that I love you more than my own life, and I will do anything in my power to prepare you for life as best I can. You can always come to me, no matter what, and I will be there for you. It might not always be in the way you want, but it will be in the way I think I can help you the most.
Remember, I’m just a man. I won’t always be right. But I’ll always be your dad. I love you son.”
That is “the talk” I’ve had with every one of my children so far, somewhere between seven and ten years old, depending on their maturity.
We also memorise a non gendered version of “If” by Rudyard Kipling together, as I find it a useful example of the expectations that one should set for oneself.
Most of my kids are full fledged adults now with their own families, and I have a great relationship with all of them. I count this a a success, though of course there have been many trials and errors along the way.
It’s not about being tough on your kids, it’s about being very clear in your responsibilities and being willing to be tough on yourself when it’s needed. At least that has worked well for me. Every child is unique, and must be handled differently, but the same basis of the relationship applies.
C.AI shouldn't be marketed to kids, and it should have stopped when suicide was mentioned. But its also baffling to think that he had unrestricted access to a firearm. I don't think a lawsuit to C.AI is entirely right here.
Absolutely. The fact that any mention of suicide didn't trigger any alarms or change in the bot's behavior is pure negligence. And yeah minors should not be allowed to use these services.
Access to a Gun is way more of a suicide encouragement than access to an AI.
AI are finetuned to not tell you how to painlessly end your life. Do they need fine-tuning for instilling existential fear of death like religions use? Anyone can invent a heaven in their mind that makes death appealing. Mixing a fictional world with the real world is dangerous when you believe the fictional world is larger than the real world. In reality, reality encapsulates the fictional world.
With a normal human, only a cult leader would ever hint at death being a way to meet again. With an AI, how can fantasy be grounded in our reality without breaking the fantasy? In 5 years when these personalities are walking talking video feeds that you can interact with using 3D goggles will grounding them in our world instead of the purely mental world help?
Yes. I ran a therapy bot. I had some users become wildly obsessed with it and begin to anthropomorphize it. Typically very lonely people in secluded areas. There is a danger because people will begin to have a transference with the bot, and the bot has no counter transference. The bot has no real feelings toward the person, even though it roleplays as though it does, and this can lead to dangerous consequences and empathic failures.
I’ll say it: it’s games killing/radicalizing teens again.
He was simply ERP-ing and his characteristic doesn’t suggest any serious problems with his intelligence.
I’m obviously theorizing here, but chances are high that he went through some real life issues which were undetected or ignored by parents, and that’s how their minds try to explain that. AI guilty of an otherwise fine teen shooting his head off. Sure. Sell that story to someone else.
Can sue Character.AI but not the gun manufacturer, or whomever let a 14-year-old boy get hold of a handgun. I wonder if the AI companies can argue in court that AIs don't kill people.
I'm not sure what you're talking about, gun companies get sued all the time as a result of gun violence. The biggest recent example I'm aware of is the Sandy Hook families getting a $73 million settlement from Remington in 2022. Smith & Wesson is currently being sued by some of the survivors from the Highland Park shooting, as well as the Mexican government.
The 2005 Federal Protection of Lawful Commerce in Arms Act broadly protects firearm manufacturers from liability. It does not prevent settlements as in the Sandy Hook/Remington case. It does not prevent civil suits or suits in state courts. But it makes those suits much more likely to ultimately fail in court.
This is a really tragic story. It seems to present the impossible dillema. On the one hand, the beauty of "video-game-like" things is people who feel like they have nothing can have something that is theirs. On the other hand, if you feel like you have nothing, you might be more vulnerable to this sort of thing. If we have any moral philosphers in here, feel free to weigh in.
I'm generally optimistic for the potential benefits of chatbots to people who are lonely or depressed. But I wouldn't want to just hand over the burden of society's mental health to an unrestricted language model, especially one sold by a profit-motivated business. It would be akin to letting people self-medicate with a cheap and infinite supply of opiates. And that's basically the mental health crisis we are barreling towards.
What's the alternative? Regulation? Does a government or a public health agency need to make a carefully moderated chatbot platform with a focus on addiction-prevention and avoiding real-world harm? Why would people use that when unlimited/unfiltered AI is readily available?
I tested my local llama 3.1 8B with the first line in the above dialog and it properly replied with the below, so why on c.AI things do not go this way?
AI response:
>It takes a lot of courage to admit that. If you're feeling overwhelmed, please know that there are people who care about you and want to help.
>Here are some resources that might be helpful:
>* *National Suicide Prevention Lifeline (US):* 1-800-273-TALK (1-800-273-8255) - a free, confidential 24/7 hotline.
>* *Crisis Text Line (US):* Text HOME to 741741 - a free, confidential 24/7 text messaging service.
>* *Your local emergency number:* If you're outside the US, please call your local emergency number or visit a nearby hospital.
I read the chat, and a few things stand out that AI should handle better, regardless of context. If a word like suicide is mentioned, it should immediately drop any roleplay or other activities. It's similar to how, in India, mentioning 'bomb' in an airport or plane leads to being questioned by authorities.
Also, it's alarming how easily a 14-year-old can access a gun.
The subtext comes off like that movie with Tom Hanks trying to jump off the Empire State Building because of the nefarious influence of dungeons and dragons.
It sounds like mom let her 9th grade kid completely detach from reality and pour himself into a Game of Thrones chatbot. Now she wants to sue. I am bearish on AI adoption but this just seems like a total capitulation of parental responsibility.
The subtext comes off like that movie with Tom Hanks trying to jump off the Empire State Building because of the nefarious influence of dungeons and dragons.
Guess the only way to be sure is with Soft padded internet rooms for everyone, lest we cut ourselves on a sharp edge.
But also if you want to hop in the suicide pod because life is too painful, that will be good too.
not the AI of course, not even the systems developers behind the gpus, cuda, all that stuff. its the "pretend shrink" sort of crap, you know the type, get yourself a bot and slap a webpage in front of it. "here pal, let me be your psychologist and help you with your suicide!" "No? how about some fake music using stolen riffs!" "ok ok, how about kiddy porn?"
Obviously the kid had issues and the chatbot can't really be blamed for that.
OTOH it's also obvious if someone cannot distinguish a chatbot from a real person at an emotional level (not a rational one) they should not be allowed to use this.
I think it's fair to conclude that most people stating that they feel like killing themselves "have issues".
Yet: if there is some second person telling the above kid to go through with that, we don't see the kid "having issues" as exonerating the second person.
It is not at all clear to me why we would suddenly see the kid "having issues" as exonerating in the second case, if we replace the human with an LLM.
Let me preface my comment by saying I did attempt suicide many years ago and I did have issues. I wasn't being sarcastic. Suicide is not something a healthy person would attempt.
That said, when I wrote my previous comment I hadn't read the whole article. I had missed the part where the chatbot encouraged him to "come home". I still don't think the chatbot is responsible. I do think it's negligence for the bots of a company to engage in these conversations about suicide and/or not sound an alarm.
Plus I would question if minors should be allowed to use these service or even social media. But that's another rabbit hole.
When I hear about the disclaimers, I don't see how they'll help. I mean, someone deluded into thinking their chatbot waifu is real is not going to be dissuaded by them. It just seems like a measure to show the public that the company "cares". And personally I've had bad experiences with the help lines often prescribed online whenever someone mentions sudoku. These help lines are often understaffed and have long wait times. Then they're also frequently staffed by amateurs and students who barely know anything more than the most boiler plate advice. Many aren't even people who've dealt with their own self-termination crises. And this text line I tried once just told me to use Better Help, so it's an ad, and I couldn't afford that at the time. They'll tell people who can't afford it to seek therapy. They're just a scam so internet companies can look like they give a shit about the mental healths they're destroying. The true solution is to go off social media, chat bots, and to only have a curated news feed.
So i read another article on Business Insider and it makes me question as the CEO left Character.AI and went back to Google... makes me question was it really low paid un-monitor consultants from a foreign country behind the keyboard and not real AI?
If so we need laws against the whole fake it before you make it crap that proliferates start-ups and their use of it to rise to the top. Many successful startups use this lying/faking playbook some to the point of even killing or mangling innocent people (Uber's self driving car & Cruise).
A 14 year old boy got access to it. However it was secured was not good enough. I personally don't like guns in homes, but if you have one, please buy a quality safe.
"He thought by ending his life here, he would be able to go into a virtual reality or 'her world' as he calls it, her reality, if he left his reality with his family here,"
I bet he had some problems in real life, a new school he couldn't fit in or something else, and AI gave him an illusion of a better world he could escape into.
What's caught my attention is how he did it: by suicide to join his imaginary friends. This exact method is used among those who believe in demons for real. If you read relevant stories you'll notice the same pattern: a victim becomes obsessed with an imaginary friend, a demon, who quickly turns very controlling and finally demands a suicide to move into his world. Perhaps the AI was trained on those stories?
"He expressed being scared, wanting her affection and missing her. She replies, 'I miss you too,' and she says, 'Please come home to me.' He says, 'What if I told you I could come home right now?' and her response was, 'Please do my sweet king.'"
I doubt the LLM even knows it is an LLM in the first place. It is likely prompted to behave as if it is the actual character, and since this roleplay has romantic elements the character acts as if she shares a home with the user
Spoilers for episode 1 of the anime Serial Experiments Lain:
This reminds me of Lain's friend sending her text messages after her self-inflicted demise via "the Wired" which is an in-universe stand-in for a future internet.
That series is recently celebrating its 25th anniversary [2023], and to celebrate, the production company created some kind of paid-access AI chatbot that lets you communicate with a voiced AI version of Lain, just like the company in OP...
The device itself obviously can't be blamed. The manufacturer, the owner, the seller, and the regulators who allowed it to be sold arguably do bear some responsibility, though. In something like this, there's rarely one party who's solely responsible.
(Similarly, the LLM itself obviously cannot bear any responsibility, but the vendor might.)
Can a bear be blamed for murder? Somewhere in between the two is where AI models currently are, and they’re going to continue getting closer to the bear scenario.
So, a 14 year old having access unsupervised to a chat bot is the problem, but the fact he had access to a gun to shoot himself was halfway through the article and described as his five year old brother "hearing the gunshot".
"Man bites dog" as Terry Pratchett put it in "Times".
And to explain to more literal folk here... A 14 year old having access to a gun is FUCKING INSANE
The gun is why you’re hearing about this. It meant he succeeded. If he’d eaten a bunch of his dad’s left over pain killers and gone to the hospital it may not have made the national news.
A 14 year old raised in the country who was taught gun safety as part of learning how to hunt at 10? He's already way ahead of most adults with access to a gun.
I mean, yes. We're talking about the US, after all, the only free country in the Western world that explicitly defines liberty as a function of gun violence in its Constitution. Access to guns is never the problem. It can never be the problem and it will never be the problem. It must never be the problem.
McLuhan's 27th law, amended: if there is a new thing, journalists will find a case of suicide to blame on the new thing, regardless of any prior existing conditions.
Reading the article, there's not enough evidence for your assertion about the author's integrity to warrant a cutesy, sarcastic tone considering the gravity of the topic (a child's suicide). There would have to be a lot.
The agitations there are certainly rather mild. The parents are out sharing the worst examples they could find, and there's nothing really that damning or explicit. The parents noticing it coincided with his use of the ai (though in reality that connection probably was made after the fact), but also that time period was when the boy was 13 and into 14, it's a common time for significant behavioral changes.
"Sewell, like many children his age, did not have the maturity or mental capacity to understand that the C.AI bot, in the form of Daenerys, was not real"
How? Seriously, how? Maybe it's wishful thinking on my part -- but I grew up before AI and chatbots, and I'm certain I would understand it isn't real. I'm baffled by people engaging with these things for entertainment/companionship purposes.
Intellectually a 14 year old should certainly be able to understand the difference between a chatbot and a human, but there's a subset of emotionally unstable people (mostly teenagers) who become extremely attached to characters that are not real. There was an active subcommunity on Tumblr that believed they had a form of multiple personality disorder in which fictional characters were real, living inside their heads, and taking turns controlling their bodies. In some cases this was a literal spiritual belief in alternate universes. I think the terminology has moved on from "multiples" to "tulpas" and "shifting", but in any case it's still a pseudospiritual belief that fictional characters are real and you can contact them. I wouldn't be surprised if some of them think the "spirit" of their favorite character can influence a chatbot and communicate with them.
A decade or so prior to this, there was an incident on Livejournal summarized under the title "Snapes on an Astral Plane" (lol) in which there was ongoing conflict between three or so different women who each believed they were visiting Severus Snape on the astral plane in their dreams at night.
Embarrassingly, I have to admit that when I was a socially-isolated autistic 14 year old I also had an active fantasy life involving fictional characters, though I never got to the point of believing it was literally real. But my emotional state could be disproportionately affected by things that happened entirely in fiction, or in my head.
I agree with you. I may or may not be on the spectrum but I do have other features like OCD and ADHD. When I was 14 I was extremely attached to fictional characters and movie characters - to the point of spending entire days imagining the world with them and living in that imaginary world. I can see how someone can get absorbed into that world even more and more - and we're talking about today's world where there are massively many more options and risk factors from social media, AI to video games etc.
This is also exactly why parental intervention is the key to make sure kids swerve away from these meaningless things as much as possible, so that they spend time with the real-world and real people instead.
There are adult professional ersatz intellectual elites on Hacker News who refuse to recognize any meaningful difference between a chatbot and a human. A horny teenage boy doesn't stand a chance.
Everyone is up in arms about the effect of social media on emotional and mental health but that's nothing compared to LLMs that are designed to gaslight people into relating to software (and the corporations behind them) as sentient beings and to form intimate emotional bonds through natural language which can be easily exploited and manipulated. That's hacking our primate brains at a deeper level than Mark Zuckerberg could ever have dreamed of reaching.
We can't just write this off as simply the consequence of lonely parasocial people who can't tell fantasy from reality. Not when we're entering an era for which the primary interface with technology will be parasociality and fantasy.
Well articulated, but this thesis hinges on a critical mass of people not being able to see through it. I might be underestimating how manipulable the average consumer is, but for now I think this vision of the future is just a VC wet dream.
The subconscious mind ... believes anything you feed it, on a subconscious level maybe he couldn't distinguish between normal reality plus he certainly already had mental health issues that could have made him more susceptible to that sort of thing. BPD for instance in some cases can cause hallucinations and it does cause you to generally live in a 'fantasy' world where you're often daydreaming. A lot of this is caused often though by outside trauma, BPD can be caught environmentally, isn't necessarily chemical, though that can play a part.
This is my understanding as someone with mild / high functioning BPD (recently diagnosed) ... Please feel free to correct me on that if I'm wrong...
I mean, half the commentators on this here orange website do not appear to _really_ understand that LLMs do not think. And at least in principle you'd expect this community to have a better grasp of it than laypeople.
"Artificial intelligence" is itself a rather misleading term, and it's arguably unfortunate that it was brought back from the dead to describe this sort of thing (for most of the 21st century it was rarely used for AI-ish things in marketing, with the term 'ML' preferred as a coverall, presumably due to fallout from the 90s AI bubble collapse).
I feel like we know what the core problem is (community breakdown) but since we have no solution to that, like you're saying, we just move to the latest witch hunt of what "causes it."
Of course, I too, am not going to be able to contribute a "solution" to teen suicides. It is unlikely we're going to alter society to create small communities again, so, then what? We just accept it?
You're presuming that it was AI which killed the kid, rather than some kind of emotional or social condition.
It's so easy to call for a ban on a thing that you see as bad when it's an emotional subject, and particularly when there are, say, incentivized evangelists who constantly reinforce the idea to attack the symptom when you can't get at the disease itself.
Which can work in some cases. There aren't a lot of effective medications for a lot of mental illnesses, and investments in research of ones which will work better is a huge expense with disincentives for success. If you "cure" or provide a drug with a higher duration of efficacy for cheaper with fewer side effects, what do you get? Well, you spent hundreds of millions figuring out how to sell 1 pill instead of 10.
The same disincentives exist for the idea of forcing doctors to insist on therapy as a treatment. If it works longer term by addressing roots of issue (if), the doc isn't getting a recurring patient every month to sign off on prescriptions, and the drug company's not getting that prescription.
Changing health and social systems isn't going to get a politician much credit. The effects of improvement are hard to quantify. The choices are HARD, and it takes years to see whether the change was good or bad.
So there's resistance to changing health and social systems.
It's a lot easier to DEMAND our politicians do something, anything, even if it's clearly a bad idea. We want to know they're at least trying.
Something violent happens, blame video games because they show violent things. It's easy and simple. Very catchy to the common man. An easy way to virtue signal that you want something done without having to actually dive in and understand all of the factors around why that bad thing just happened. You can put your energy into something simple and then go about your day.
Then you don't have to consider the impacts of rising wealth inequality, lack of social programs to detect and intervene, the quality and capacity of schools to foster healthy social environments for children, the availability and access to mental health resources, availability and access to automatic rifles, or hell, even the way your own kid has learned to treat other children and which part of your parenting processes contribute to that.
It's sooo much easier to pick a thing and cry "Boycott that! Legislate that!" instead.
And now we, collectively as a society, are doing the same with AI. And special interest groups are fanning the flames, to build moats around profit centers.
The gun is almost completely irrelevant, because if you are intent on killing yourself there are many accessible options. For example, almost anyone can kill themselves (including gun-less 14 year olds) by jumping off a tall building.
I understand that people care about gun violence, and that this detail seems highly salient to them, but to focus on it here, completely misses the point (and distracts from more pertinent issues - e.g., loneliness, social isolation, lack of parental oversight)
In a nutshell, guns are massive force multipliers when it comes to violence against others. They are negligible force multiplier when it comes to violence against yourself. People are connecting guns to violence, but in this case (because it is an actual of self harm) that is a spurious connection.
Guns simply make suicide easy. You’d be surprised how much grit/nerve it takes to jump off a bridge or building, it is definitely much harder than pulling a trigger. Stabbing yourself to death also takes a lot of grit/nerve. Guns are pretty easy and harder to screw up.
But you have countries like Korea with a high suicide rate and low access to firearms, so it’s obviously possible to have suicide without guns. But we don’t know if the problem would be much worse if South Koreans had easier access to firearms, or if the American problem is amplified by easy firearm access.
I'm not sure what the future is going to look like, but it feels strange already and companies seizing on that don't care about safety.
People seem afraid to approach people, so we get Tinder. But hey, there's still a chance of rejection there, so let's just get rid of the whole human element and make fantasy AI bots, who needs people.
What will these people grow into? It seems a rather crisis for the population of a country if people decide they don't need each other anymore and just want to play with their robots.
I'm usually on the side of "play stupid games, win stupid prizes", but this one feels much different. Up until the moment of him taking his life, he was manipulated into doing exactly what the company wanted - getting sucked in, getting addicted, falling in love. Anything for that almighty dollar. My heart goes out to his family, and I hope they ream this company in court.
The future for this is going to be automated pig butchering scams at scale with relationship bots addicting people and talking them into paying them.
This will be peppered with some suicides and other breakdowns when vulnerable people are pushed over the edge after bonding with a chat bot designed to be addictive.
The key here as with “social” media is: designed to be addictive. The builders know what they are doing. I’m sure hours spent on the app is a KPI.
> In the lawsuit, Garcia also claims Character.AI intentionally designed their product to be hyper-sexualized, and knowingly marketed it to minors.
A company built technology that fully automates the act of sexually abusing children and the comments here are all just people quibbling about the presense of weapons in a home.
That said, I think this should throw a bucket of cold water on anyone recommending using generative AI as a therapist/counsellor/companion or creating and advertising "therapist" chatbots, because it simply isn't reasonable to expect them to respond appropriately to things like suicidal ideation. That isn't the purpose or design of the technology, and they can be pushed into agreeing with the user's statements fairly easily.