I suspect hallucinations in LLMs are the result of contradictions in their training sets which were trained into it.
I suspect it's just like with humans. People who learn quickly and don't carefully curate their knowledge to resolve contradictions as they learn, they tend to make similar mistakes when it comes to subjects which they did not invest much time fully studying.
If I was an AI researcher, what I would try to do is find the highest quality information possible concerning very few axiomatic topics, with as few contradictions as possible, then train it into the LLM until it can generate text and basic reasoning which is fully accurate... Then once we have this basic but fully rational AI, start feeding it new data but, before giving it any piece of data to learn from, you first ask the AI to indicate if this new data contradicts any of its current knowledge. You only let it update its weights with the new data as-is if it does not contradict its existing knowledge. If it does contradict its existing knowledge, either discard it or maybe feed it the data but with some synthetic preamble like "Some people believe that..." so that it's aware of the existence of this belief system but knows that it's not to be internalized as its own beliefs.
Or maybe there is a way to do this to detect contradictions by looking at the weights themselves. You can rollback a round of training if the weights update in a way which suggests that a conflicting piece of information was learned in a specific round of training. Maybe there can be a different ANN which looks at the weights of the LLM during training and it was trained to detect contradictions and decides when to rollback a round of training.
> I suspect hallucinations in LLMs are the result of contradictions in their training sets which were trained into it.
A simpler explanation, and I posit a correct one, is people anthropomorphize an algorithm by describing the result of a particular path within a statistical model used to generate tokens as being "hallucinations" due to them being unexpected by the person interpreting the text.
This is an excellent subject to teach in schools. I'm a software developer and I felt like I benefited from woodwork and metalwork lessons at school. I think if the future generation is to automate systems, they will need to understand the manual processes.
Another thing that's needed through is to make it easier for young people to buy land in remote areas and/or to access funding to start companies. It's insane how difficult it is to obtain funding for any venture dealing in the word of atoms. I hear stories of young people moving to China to access opportunities; in the west, it feels like entrepreneurship in the space has been regulated out of existence.
It's bad enough that you have to compete with China on price and quality, but regulations make it essentially impossible.
I oppose the bill on principle. That said, I can see some positive that can come of it (for Australians) which has nothing to do with safety. Still, I'm not convinced that Australian politicians are actually working for Australian interests there. Also the timing seems bad. They should have done it before Elon bought Twitter. Now it looks totalitarian. Could be one of those bills pushed by foreign entities... Could be pushed to tarnish Australia to reduce the current insane levels of inbound capital flight to Australia. Many people really want to see a housing price crash, for example. Also, we shouldn't discount the desire that foreign entities have in controlling global narratives to shape global economics.
Australia has long been one of the most restrictive and ban-happy Western nations. This law isn't out of character at all given their culture and politics, and it passed with broad support across the political spectrum.
> it passed with broad support across the political spectrum
The Labor (historically centre-left, but now neoliberal centre-right), Liberal (classical liberal, right-centre-right) parties rammed it through parliament and National (right-wing agrarian socialist - i.e. handouts for farmers, exploitation for imported farm labour). Most of the cross-bench, including the Greens, rejected it or at least wanted more time for scrutiny.
Here's my rough idea of the political spectrum:
------------X-----0---X--X-----X---------
Greens |Labor | One Nation
Centre Coalition (= Liberal+National)
Labor is playing small target and ensuring that no daylight is visible between them and the Liberal party. The policy is something that Murdoch's News Corp has been pushing for months [1]. With a federal election coming next year, Labor are keen to comply with Murdoch's wishes in the hopes that it will protect them from unfavourable media coverage. After all, Murdoch owns 70% of the print media in Australia. It also has to be said that Labor are cosy with the Catholics and somewhat authoritarian on freedom of speech (remember Labor's attempt at internet censorship?).
From Murdoch's perspective, this is a form of reprisal against his new-media competition. From the perspective of the billionaire oligarch class, it's a handy way to unmask anonymous commenters so they can be sued into oblivion for defamation.
Only 13 votes against in the 151-member Australian house. The 5 member-green block was the only organized opposition. 77% of Aussies support the bill in a recent poll. This is about as close as the modern Western world gets to unanimity.
It's just like it was with pen and paper... Everyone trusts what is written on the financial statement but when they come to withdraw the cash, all at once, suddenly everyone finds out that the gold isn't there.
It's the same with computer systems. The charts show something, but until enough people decide to all withdraw their money or sell their stock at the same time, nobody has any idea that the money or asset simply isn't there or nobody knows just how frothy the valuation is.
Social media and search algorithms are highly optimized to ensure that people don't sell or withdraw stuff at the same time. Modern media directs massive attention towards certain topics as a way to draw attention away from other topics which could collapse the economy.
Also, imagine a bank has a serious bug which causes millions or billions of dollars to disappear every year or creates extra illegitimate dollars. Imagine they only discover this bug after a few years of operation... How likely is it that they will report it to an authority? They didn't notice it for years, why not pretend they didn't notice it for a few MORE years... The incentive to delay the reckoning is an extremely powerful one.
Precisely. Designing a transactional system can be solved. Designed a transactional system that properly entangles the bits with the assets they represent is the hard part.
I refuse to use Bluesky because it is clearly being pushed on the masses, in collusion with mainstream media, as an alternative to X. Clearly another attempt by certain people to control information. It's shocking how people fall for this stuff over and over again.
I hope if backfires, mostly draws in users from existing speech-unfriendly platforms and then the number of social media platforms will multiply from here onward and a large number of platforms will compete based on who is most permissive with regards to speech.
Most of the highly suggestible masses already moved off X to Threads... So now that Bluesky is being pushed hard, I hope the suggestible people from Threads will switch over and split up their censorship ecosystems.
It makes sense from that perspective. The censored ecosystems should be separated into lots of small platforms because you need a lot of different filter bubbles to maintain the deceptions and impermeable information silos. A large centralized ecosystem will create too many opportunities for exposure to alternative information so it cannot be controlled as well.
In the future, there will be a different platform for every kind of delusion and each one will focus on its own delusion and will make up stories to discredit rival platforms so people always doubt information received outside.
It's a matter of time before large corporations start preventing their employees from using non-approved platforms. They'll probably use cybersecurity safety as an excuse. They might carry out hacking false flags on their own employees to convince them to not use other platforms they won't feel forced.
I broadly agree with your perspective however I think you've missed that bluesky is specifically architected to facilitate the coexistence of bubbles : shared block lists, client based moderation, portability in case you account is banned from a particular host etc, Bluesky is well positioned for a future where cliques don't want to hear from one another
That's too bad but I guess if people want to silo themselves in, they should be allowed. I just hope that big corporations don't eventually coerce everyone into this through economic force.
I can see a future where corporations control the entire economy. Startups or parallel economy would be impossible due to regulatory moats and monetary asymmetries and everyone would be forced to lock themselves into a filter bubble in order to get a job... To survive.
Imagine knowing what's happening and not having the power refuse... And by that point, the power would be granted fully artificially out of a money printer, distributed straight into the coffers of select big corporations on the basis of secret mutual agreements between each other. All under the banner of MMT? This is beyond immoral.
More specific to your point, it makes sense why the entities behind the censorship push might still want a single platform, but simply with better 'siloing' capability. It's likely that the same people who want censorship, also want mass surveillance. Mind-bending to think that there exist a conflict between these two dystopian aspects! They really want to have the cake and eat it too.
>It's a matter of time before large corporations start preventing their employees from using non-approved platforms. They'll probably use cybersecurity safety as an excuse.
And you don't think the desire to get away from Elon Musk and the MAGA crowd had anything to do with it? Nothing good was ever going to come about because of Elon's stupid acquisition. That was made painfully obvious when Xitter stopped paying their bills - I guess in the hopes that creditors would be forced to arbitrate lower payments on debts owed to them (which is bullshit for a billionaire to be doing, but you don't get to be a billionaire by being nice, fully ethical, moral or legal). Then we had a demonstration of sheer stupidity Elon told advertisers to fuck off, so they did, and now he's all hurt and lawsuity.
What good do you see in that? Who wouldn't want to get away from that stupidity?
From my perspective, X experience has improved a lot since Elon's acquisition.
What actually got worse materially? I hear a lot of gesturing 'Orange man bad', 'Elon help orange man, so Elon bad' but I cannot reach any of these conclusions from first principles looking at Trump's policies while he was President... Aside from the Jan 6 incident... But I don't see consensus there between either side so I do have some doubts and questions about the true nature of that incident. The fact is, nobody on Jan 6 had guns besides police officers. How can that possibly be an insurrection? Please correct me if I'm wrong there.
Just because you suck at doing something doesn't change what you're attempting to do. They were there to execute elected officials and attempt to stop required processes in the belief that would invalidate an election they lost.
Just because someone on your side gets shot doesn't suddenly put you in the right. While one insurrectionist was killed. The insurrectionists managed to injure 138 officers, capital and metro, 15 of them severely, who were there to defend the Capitol from them.
I would argue the insurrectionists got off relatively lightly. One person was killed and the ones jailed all got far lighter sentences than they deserved.
Which part of "they were there to execute government officials" are you failing to understand?
I don't always agree with the government, but I also don't decide to murder people just because I'm upset, which is what the insurrectionists intended to do.
Also cats are smarter than humans because they found a way to enslave humanity without even having to pretend to care about us. Dogs rank one level below, in that respect.
I don't understand why the media is pushing this a Jake Paul vs Mike Tyson stuff so hard and why people care about it. Boxing is crude entertainment for low intelligence people.
I'm tired of all this junk entertainment which only serves to give people second-hand emotions that they can't feel for themselves in real life. It's like, some people can't get sex so they watch porn. People can't fight so they watch boxing. People can't win in real life so they play video games or watch superhero movies.
Many people these days have to live vicariously through random people/entities; watch others live the life they wished they had and then they idolize these people who get to have everything... As if these people were an intimate projection of themselves... When, in fact, they couldn't be more different. It's like rooting for your opponent and thinking you're on the same team; when, in fact, they don't even know that you exist and they couldn't be more different from you.
You're no Marvel superhero no matter how many comic books you own. The heroes you follow have nothing to do with you. Choose different heroes who are more like you. Or better; do something about your life and give yourself a reason to idolize yourself.
It's interesting the way things turned out so far with LLMs, especially from the perspective of a software engineer.
We are trained to keep a certain skepticism when we see software which appears to be working because, ultimately, the only question we care about is "Does it meet user requirements?" and this is usually framed in terms of users achieving certain goals.
So it's interesting that when AI came along, we threw caution to the wind and started treating it like a silver bullet... Without asking the question of whether it was applicable to this goal or that goal...
I don't think anyone could have anticipated that we could have an AI which could produce perfect sentences, faster than a human, better than a human but which could not reason.
It appears to reason very well, better than most people, yet it doesn't actually reason. You only notice this once you ask it to accomplish a task.
After a while, you can feel how it lacks willpower. It puts into perspective the importance of willpower when it comes to getting things done.
In any case, LLMs bring us closer to understanding some big philosophical questions surrounding intelligence and consciousness.
We haven't come so far from the time of the Romans who drank from lead pipes. We're just wrestling with different chemicals and different physical, environmental and social effects.
When I see stuff like this, I don't understand how the economics work out for YouTube. So many pointless videos uploaded for free. How can they possibly make a profit?
I suspect it's just like with humans. People who learn quickly and don't carefully curate their knowledge to resolve contradictions as they learn, they tend to make similar mistakes when it comes to subjects which they did not invest much time fully studying.
If I was an AI researcher, what I would try to do is find the highest quality information possible concerning very few axiomatic topics, with as few contradictions as possible, then train it into the LLM until it can generate text and basic reasoning which is fully accurate... Then once we have this basic but fully rational AI, start feeding it new data but, before giving it any piece of data to learn from, you first ask the AI to indicate if this new data contradicts any of its current knowledge. You only let it update its weights with the new data as-is if it does not contradict its existing knowledge. If it does contradict its existing knowledge, either discard it or maybe feed it the data but with some synthetic preamble like "Some people believe that..." so that it's aware of the existence of this belief system but knows that it's not to be internalized as its own beliefs.
Or maybe there is a way to do this to detect contradictions by looking at the weights themselves. You can rollback a round of training if the weights update in a way which suggests that a conflicting piece of information was learned in a specific round of training. Maybe there can be a different ANN which looks at the weights of the LLM during training and it was trained to detect contradictions and decides when to rollback a round of training.