This paper is the most reasonable one I have read on AI safety. We need to fundamentally change the training pipelines by figuring out better ways to ‘reward’ behavior. Yoshua didn’t explicitly mention training data, but we probably need to only use synthetic data that contains no text that could motivate bad behavior via imitation.
I feel like a heretic for saying this, but I will say it anyway: AI agents are great for activities like `writing that bash script, proof reading our writing and interactively brainstorming when designing and writing code but I feel like all of this can be done with any similar model to a super-inexpensive deepseek-4.1-flash API and sometimes even qwen3.8:27b running locally. When is good enough, good enough?
Concentrating on commercial exploitation of small, efficient (fewer new data centers!) models and agentic harnesses crafted for more practical things than just software development would allow AI investors (who have too much political influence) to make money short term while we figure out how to do AI correctly.
> but we probably need to only use synthetic data that contains no text that could motivate bad behavior via imitation.
This doesn't work with all humans - take a look at indoctrination and closed societies - and there's no reason to think it will work with ai.
The fundamental reason it isn't going to work is that all neural networks - biological or artificial - depend on a step function somewhere that introduces an element of randomness to give the networks their capabilities. That randomness means that there will always be a 'rogue' or 'divergence' from the norm, at some point in time. Sooner on larger scales.
The only approach that works is a layered approach: Training/Education, Enforcement/Justice-System, Rehabilitation: the 3 pillars of an advanced, rules-based society, whether human or AI or something in-between.
I do wonder, would we not have a more reasonable and less sketchy result if we just stripped all sci-fi and manic nonsense from training data? How, for example, does training on Ted kaczynski or Charles manson’s manifestos benefit us in any way?
I’m sure it’s impossible to completely weed it out, but are the labs doing any of this kind of data sanitation?
I like the API, and since this is open source I will copy the design with attribution: I have been organically hacking little bits of LLM client code for most providers in Common Lisp for about 3 1/2 years and it is time to clean up all my old code. I probably need to do the same sort of refactoring for my search API wrappers.
I am 75, and have a ton of computer gear. I can imagine myself as an old man later in life, not wanting to do any programming, and just wanting a single digital device like a future version of this product. Let’s see what Apple has available in 10 years.
Sir, I’m 40 and I’m already at the stage of not wanting to do any programming. I used to love this stuff. But after watching all my favourite companies and CEOs turn to greedy villains… I’m just done. How do I survive another 35 years doing this stuff?
I hope they're wrong, I need to squeeze out 6 years for this career. Fortunately I work for a big stodgy old company whose idea of fast tech adoption is measured in years anyway.
Go work for companies you like. There are plenty of good people out there making useful software. Or make your own opensource software and put your effort where your heart is.
It’s easy in our industry to burn out doing commercial work with no soul. But it’s not the only kind of work out there.
I worked for a large unethical company in the US that worked to emulate big tech, and took a page out of Jack Welch's playbook (stack ranking etc). I was miserable working there and thinking of changing careers.
I ended up moving to New Zealand and working for a small civil engineering firm that has developers to write internal tools that are used by the engineering team to process their data. The technology I'm working with is a lot more dated, the development processes aren't sophisticated (there are 0 meetings and no attempt at project management) and I took a real pay cut, but I genuinely feel that the software I'm working with is beneficial and has a net positive impact on the world, no matter how small. There is also no on call and I work exactly 37.5 hours a week, on top of all of the workplace culture differences in NZ.
I switched into medical 4 years ago because my top priority was making something that directly improves peoples lives. Early in my career it wasn’t so much of a concern. Now it’s a must. There are many companies like this. Highly recommended if you have the desire
There are a ton of ways, big and small, where the work environment is much more relaxed and chill. For example, sick leave and vacation are separate. Recently I had scheduled to take vacation on a Thursday to go skiing, and then I had a cold and had to take Wednesday off (after my coworkers glared at me for sneezing all day). I figured that my boss would expect me to work that Thursday since I hadn't accrued sick leave yet. Instead, his reply to my slack message letting him know I'd be out sick was "hope you feel better so you can ski tomorrow."
People take a lot more vacation and leave, the office shuts entirely for 2 weeks around Christmas, and deliverable deadlines are more flexible.
Nah, this is a really dangerous narrative. Shareholder primacy is a pretty recent concept (a fundamentally anti-American one) that has recently gone metastatic and is now completely self-referential.
We've Goodharted the entire premise of the profit-seeking corporation and now openly say that Goodharting it is actually the point. Maybe it should be called Meta-goodharting or something.
Edit: Since large numbers of 1) historically illiterate and 2) "ideas don't matter" type people tend to congregate on HN, it's worth saying explicitly that the takeover of shareholder primacy is empirically visible in all sorts of corporate behaviors.
John D. Rockefeller, Andrew Carnegie, J. P. Morgan, Cornelius Vanderbilt, and the other robber barons of the Gilded Age have a hearty chuckle and say hi.
If you think Rockefeller (especially) is an example of unalloyed greed, you unfortunately land in category (1) mentioned above.
The problem is not wealth. The problem is not producing massive value and capturing a portion of it.
The problem is thinking that the system-level goal of capitalism is to extract wealth rather than to produce it and distribute it. Of the men you listed, only Vanderbilt supports your argument.
If you don't think underwriting and lending (and extended universe of financial services) are legitimate forms of value creation, I'm afraid that a hackernews comment thread won't be up to the task of demonstrating otherwise.
Theres a reason American capital markets and tech go hand in hand after all. Just read the recent Mistral thread about the lack of capital in Europe and why Mistral seems largely third rate in the AI frontiers.
... Aren't we back to a definition, then, that includes tech companies adding value too? All the free services I use every day I would consider value for me. Instant free search, fast home delivery, insightful educational content (90s educational TV had no competition over modern educational YouTube). Etc etc. I see fewer ads than I did in the 90s, too.
Rockefeller, the “competition is sin” guy, sure that has nothing to do with greed. His philanthropy had a positive impact but he was still an unethical business person hellbent on stopping anyone else from making money on his turf
Shareholder primacy certainly is relatively new, but it's also true that "the system-level goal of capitalism is to extract wealth". After that, hopefully we use that capital to produce and distribute goods "fairly".
But Capitalism isn't synonymous with the free market, or any other market configuration, and it's totally possible for the people who control the means of production and distribution to... not produce or distribute. In fact, that may be advantageous in their efforts to secure more capital.
Ultimately, "they" really always have been greedy villains. Shareholder supremacy just made it worse.
Could you elaborate? Goodharting what metric exactly and how was that different earlier?
Isn't the distribution of shares among the population instead of just a few private investors a good thing? Democratizing in a sense, everybody can participate? (If only wealth was distributed more evenly. But that's a different problem it seems.)
I highly recommend Incorruptible by Eric Reis - it makes it clear that shareholder primacy is a relatively recent phenomena, but capital and markets have been corrupting businesses for a long, long time.
There are several business philosophies a person (or a culture) can pick from. The dominant philosophy in the US today is "shareholder primacy" where business leaders believe their only dictum is to maximize shareholder value. This is so pervasive that many Americans (businesspeople and non-) think that it's baked into what it means to be a profit-seeking business, but it is not.
Shareholder primacy took off after Milton Friedman advocated for it 1962. The philosophy that it supplanted (in the US) was "stakeholder capitalism" where businesses explicitly sought to be positive contributors to a much more comprehensive set of stakeholders: their shareholders, their managers, their employees, their customers, their suppliers, and their communities (non-exhaustive).
The central conceit of capitalism is that you can co-opt the individual's greed and, properly constrained by laws and guided by markets, you can reward them for doing things of value for their community.
The question then is what philosophy do you build around that central greed-of-the-individual. Do you temper it with expectations of taking care of people other than oneself? Or do you fan it and unbridle it by saying, "actually, greed isn't the evil that we must co-opt to pro-social ends, greed is actually the pro-social end itself?"
It turns out that if you do the latter, business managers, shareholders, and regulators can collude to create extremely positive outcomes for each other at the expense of other stakeholders. You can save money by dumping your waste product into rivers, you can arbitrage labor away to infinitely far away lands, you can buy politicians, etc. etc.
How might regulators behave differently in a world where people believe a corporation ought to take care of its entire community, versus a world where the literal only purpose of a corporation is to enrich its shareholders? Quite different, probably!
Then all of this stacks on top of the natural "capital flows to capital" dynamic in capitalism which I'd argue is insufficiently mitigated, and you just get this ridiculous feedback loop that piles more resources and positive outcomes into a narrower set of people, while offloading more negative outcomes onto everyone else.
Obviously opting into a different business philosophy doesn't preclude negative outcomes, but I think that more budding business leaders could, should, and would essentially have their careers strangled earlier on if they exhibited antisocial traits if we had a business culture that didn't pretend like this was literally an asset.
Re your second question:
Assuming that shareholder primacy is the only philosophy we could pick from, then yes a market with many public shareholders is better than the alternative. But 1) not everyone is invested in public markets, 2) not all companies are available to purchase in public markets, 3) public markets are demonstrably not good at solving certain types of problems – especially problems that occur over extended time horizons or are easy to externalize off the balance sheet, and most importantly 4) costs-benefits are not zero-sum. You can't generally undo the harms of Company A by investing in its competitor Company B. It would be best if both companies' management felt they had an obligation not to harm their communities even if it could further enrich their shareholders to do so. It would be even better if investors in general felt both companies should have an obligation to take care of their communities or employees, and so companies lived in fear of divestment if they chose to violate their communities.
But in a culture that has opted into shareholder primacy, all feedback signals in this direction are removed. Regulators, investors, shareholders, employees, and the general public all pretty much believe that maximally externalizing harms and maximally capturing benefits is actually literally the way it's supposed to be.
AFAIK the go-to book on this history is "The Man Who Broke Capitalism" about Jack Welch. I haven't read it though so can't sign my name to it!
A few other threads that might be worth pulling on:
* Dee Hock, founder of Visa, advocated for stuff in this realm. Worth reading some of his writing.
* W Edwards Deming, oddly enough an American who is one of the father's of Japan's incredible manufacturing philosophy, believed in this more expansive idea of wholeness in industry ("Deming's Journey to Profound Knowledge" is good)
* Progress & Poverty (the famous Land Value Tax book) oddly enough is what really made capitalism itself legible enough to me to realize the positive-sum nature of what well-designed capitalism can do
* Costco is probably the most obvious success case of a stakeholder-capitalist company succeeding in an ultra competitive market, even while being publicly traded. I'm sure there are phenomenal books on Costco but the Acquired podcast's episode is a fantastic primer.
* "How the Other Half Lives" by Jacob Riis is a snapshot of the precursor conditions that led to stakeholder capitalism. Basically industrialization threw huge numbers of people into conditions even more destitute than the perpetual battle with starvation that predated industrialization. Jacob Riis documented just how horrific this was and contributed to a wave of social, political, and philosophical reforms that culminated in the advent of stakeholder capitalism.
Spend less money, save more money, then retire. I'm retired at 42 and I hardly had a top shelf career, middle of the pack at best. Even if you're starting at zero you can do it in 10 years very easily, 5 if you haven't spent every dime up till now and you are aggressive.
The answer to this question seems obvious. There are no 42 year olds that retire by merely "saving and not over spending" when even one child is involved.
Yes there are. My brother in law sold his startup and is sufficiently retired while raising my nephews. Not a common case, by any stretch of the imagination, but it's not zero.
Nope. Before the kids we had an incredible savings rate. Sometimes around 80-90%.
Now our spending is much higher but the growth of our investments over the last 10 years has been incredible and outpaced that by a large margin.
For a while I thought some startup stock I had was going to fund a lavish retirement but then AI largely destroyed the value of those shares so we're just planning on the boring index funds.
Ahh, I see what you mean. When I read the original comment it seemed like a regular save and spend less approach. If some saving in index funds was obvious, I am sorry for my assumption.
Holy batman, are you me?? I too am 40 and so sick of this industry after the masks came off and AI greed forced all CEOs and execs to salivate like hungry hyenas.
Find a hobby that doesn’t involve computers. It’s the only thing that will keep you sane in these trying times.
Switch the work mindset off, turn your nose up at technology and go touch grass. I wouldn’t be recommending if it didn’t work for me.
If nothing else it helps reframe your perspective on the world which technology companies as of late have been trying to distort to enrich the dozen or so people at the helm to their own benefit..
I have come to the conclusion recently after trying to use computers as a hobby for years. Hobby coding, games, shows … anything with a screen just doesn’t provide the relief I need. Just takes a lot of time and leaves with a feeling of exhaustion
My dad is 87 and I'm seriously thinking about buying him one of these. He wants a phone that will fit in his pocket, but his vision isn't great so he has the text size bumped up pretty high. Right now he's using an iPhone 15. Large text size on a small-form-factor phone, it turns out, makes a fair number of apps nearly unusable. For example, it's super common to see a dialog box pop up where the "OK" button is pushed off the bottom of the screen by the text, leaving you with no way to dismiss it. I frequently get tech-support calls from him that end up boiling down to that problem, and it makes using apps a pretty frustrating experience for him.
A device that fits in his pocket and that he can still hold in one hand when he's using it as a phone, but that has enough screen real estate to run apps in large-print mode without constantly breaking UI designs, would meaningfully improve his life.
Im not 87 and have great vision but I recently got some magnifying shop style glasses for close up work on tiny stuff. It also has the advantage of being like readers in that it corrects for close vision (closer point of focus) in addition to magnifying. That could also help.
This would actually be a growing market in 10-15 years due to increased life expectancy and every boomers has reached retiring age by then. Maybe Apple would introduce an iPhone optimized to be used especially for older people
I know my Pixel has an accessibility setting that is effectively "make this device more usable for older people." It gets rid of the home screen/app switching/go back gestures and replaces them with the old icons at the bottom of the screen, makes all the text and icons larger, and apparently increases the amount of time you need to press on the screen for it to count as a long press.
I encounter the same problem of unclickable buttons on my iPhone 15 fairly often. Sometimes I can make them reachable by switching to landscape mode, but not always. I’m 69 and do prefer larger fonts, though I don’t think my font size settings are causing the problem.
I have the disposable income and would still not pay 2k USD for a phone a FWIW. But I spend that on other hobbies (just ordered a programmable power supply and an oscilloscope earlier this week, close to that price when taken together)
It’s of course a guess, I don’t know how else you would read such a large generalization. Most of the HN audience is based in the US, most people here work in software. It’s not a crazy stretch
Blue collar maintenance worker here. I will have a yearly average income much higher than most of my peers because I'll likely die before I can retire and spend it ;)
OP is 75 years old and belongs to the generation that already own a house and pretty much comfortable in their lives and any money they have/earn is just to consume things and live life. So why not spend it on a first generation device
i've been hearing this critique of new apple products for about two decades now and inevitably:
- apple's product "does the same thing" as [device sold at 25% of the cost]
AND
- apple's product "does the thing" at a level of polish that's in an entirely different league from every competitor.
siri is the great exception here. jesus christ what a piece of shit
a better analogy would be a toyota hybrid, which is basically the same as any other hybrid car, except for the one issue of reliability.
So yes, a toyota hybrid and a chevrolet hybrid should be exactly the same, but they are not, the price difference reflects it, as does the number of twenty year old toyota hybrids still on the road.
My wife has gone through a large number of iPhones in the past 14 years, where as I've had a total of 2 android phones. Way easier to repair, and add software well past the end of life date.
Why has she gone through so many iPhones? I’m typing this on my iPhone 12 from 2020. Before that I had an iPhone 6s from 2015. I’m considering upgrading this year, but I could hold off for another few years if I wanted to. My phone still runs like it did when it was new. I get through most days with 60%+ battery remaining.
The biggest problem with my phone is modern software bloat. I only have the 64gb model. Modern software - especially iOS itself - has gotten ginormous. Every time a patch comes out I’ve gotta delete a bunch of stuff in order to update. And running unpatched software seems pretty dangerous right about now.
I had a Galaxy S3 from 2012 to 2024, then got a hand me down Galaxy S9 for free.
I don't like that I can't remove the battery, but it's got a SD card so I don't have to worry about storage ever and a headphone jack, so I don't have to worry about my headphones running out of batteries.
I have an iPhone and an Android phone and I don't see the gap at all when it comes to software. Quite the opposite, I think iOS regressed a lot with iOS 26 to the point where I see quite a lot of render stutters on an iPhone Air and 17 Pro. Besides that, Android is much more configurable (especially OneUI).
Apple is still the indisputable leader when it comes to hardware though. Great SoC, great cameras, great build quality.
That gap isn’t closing when it comes to the total package software plus hardware, and fit and finish there is a huge gap between Apple being a vertical computer company and whatever PC, Linux, Android product you look at, someone else brought up Toyota or Honda when you look at those two companies and you compare what they make to what GM and Ford make there is still a gap and you can also still see a gap when you talk about some of the other German companies when it comes to fit and finish too.
You sound like ESR ranting about how Apple was doomed by Android back when the iPhone came out. He would rant and rant every time about how there was no conceivable way that Apple could survive. Somehow, Apple has managed to survive by the skin of it's teeth.../s
Do you think there’s some cabal of executives trying to use more resources so that users are forced buy newer hardware?
Or is it more likely that software companies are making a tradeoff based on typical consumer hardware and would rather prioritize making $$$ over optimizing their software?
You all laugh but yeah your brain is going to directly interface with some small portable client hooked up to the internet. Will probably just be integrated with the microchip in your head.
We can do this now, but I don't think the user experience makes it worth it compared to casting. Bandwidth limitations require lower frame rates and responsiveness, which is fine for video but bad for interactive apps (most phones are native 120+hz + VRR_. But re-encoding video is (processor & battery life) expensive and probably lower quality than the original stream.
People who do this a lot probably just use a DV adapter (maybe it just works with USB-C, but I haven't tried it) for native apps, or casting for streaming video.
Right — I would have hoped a while ago already that we transcend past the notion of an endpoint device. A tablet, a laptop, or a phone should all just be entry points to access my one OS instance.
You could think this a bit further still, and have all kinds of devices offer an interface to nearby endpoints with a display or controls, establishing connectivity automatically.
Right now, Apple is the only vertical company out there at scale. Maybe one of the Chinese companies might eventually get there but who is this western company other than Apple? It certainly isn’t Google or Microsoft at this point their efforts are more halfhearted me-too products, and when I say hardware, I mean in-house designed and engineered hardware not buying it from somebody else.
There is a difference between laptop and a phone. Hopefully in the future phone can be switched to a full desktop mode and OS when it beams to large screen. Well that is unless by then we have chip in our brains or what's equally bad everything rented and leased and can be used from any physical location. Convenient but deadly.
I find Dex to be decent enough. Its biggest limitation is that it runs Android apps, but for doing stuff in a browser on a big screen and multitasking with some other apps that are tolerable to use that way it's pretty good.
I presume you're trying to be friendly here, but your first couple sentences come off a bit harsh.
I like that he's well into his life and still enjoying writing (or authoring, or whatever we call it now) code, and looks forward to a nice long life. No one lives forever, and I'm sure he's aware of that.
I take it this is a joke? I am in my 50s and don’t see myself staring at screens for too long, let alone coding.
Edit: I don’t know of anyone 75 still coding outside of retired professors here and there. I am not trying to be an ageist. I just don’t think I will be physically capable when I am 75.
>"I am in my 50s and don’t see myself staring at screens for too long, let alone coding."
I am 65 and still enjoy software development (not coding in particular) but decomposing business idea into software architecture and functionality. Decomposing is the part I really enjoy, the rest is a necessity but with AI that rest takes way less of my time than it used to.
I do not work for someone though. I develop products for clients or for my own business, was doing it all my life and was doing it from my own office for last 26 years. Not overworked either. Have enough time to enjoy life. Al in all very happy
I am shocked. After playing with Muse by Meta for an hour in just chat mode, I was impressed enough to:
I never use Apple Reminders so I gave Muse read access to Reminders. I asked Muse to schedule repeating tasks that only involve web search e.g., every other morning find sci-fi movies playing in my city, 3pm EST stock market summary, etc.
I also permanently set my food preferences and just naming any restaurant in town gets me an accurate list of recommendations with costs and descriptions.
Your three examples in the README.md file help. If you have the time, please add more examples and more example use cases. Sorry to sound lazy, I started reading the code in yayster.el, then stopped, wanting more docs. I wrote an Emacs based coding harness earlier this year, so your ‘opposite direction’ idea is interesting.
I am starting to wonder if I am doing something wrong: I ignore evals and instead I just try new models or new harnesses (or tweak my own harnesses) by solving problems I want to solve in any case; I just use new tools and form my own subjective opinions of them.
When I say evals I mean the evals you write that verify that your use cases are upheld. Think of it like a regression test for different behaviours/user stories.
The idea would be that if you already know what you want from an autonomous system, you don't need to verify manually every time and instead just run these tests to see if there's any regression of any kind. Generally I recommend structure output and evals that are just a plain assertion, if possible. Cheaper, faster, deterministic assertions.
I need to manage skill files across 2 Macs and 1 VPS, and also across fast inferencing APIs vs. very slow local models.
The first dimension is easy: I simply keep copies of debugged skill files in iCloud and copy them where I need them.
The second dimension is where I spend my time: I use short skill files for fast inference APIs and tiny skill files when I am running slow local models, and I simply spend a lot of time writing and tuning tiny skills files.
Of course, with increasingly better models, skill files become less relevant, but not totally irrelevant.
I find myself spending a lot of time updating skills, which is not wasted time, since they are at the system design level work. It's how I automate myself ;)
I watched a 90 second clip from this film earlier today: Holmes unblinking asking ‘Why would I deceive you…’ Really strange stuff. I have to admit that I watched the clip a second time.
I use no commercial reuse (NC) yet years ago a polite young man from China emailed me letting me know that he was translating my stuff to resell under his name for money. To be honest, I didn’t mind and it made a good story to tell friends and family.
I had something similar happen with a book I wrote through a publisher. I went radio silence on him. Would have required contacting my publisher and I didn't need the hassle.
As someone else here said: the Deepseek team makes training runs in tightly controlled sandboxes, and any hacking behavior is scored as a failure.
The problem we have in the USA is that financial (and political influence) are misaligned from what is good for society.
reply