The biggest challenge with making a search engine is to combat adversarial SEO. It's an issue that's very easy to be overlooked when you are small, but at Google scale, your enemies have billions of dollars to make from your visitors.
I bet Google spends at least as much to combat that, and it's extremely hard to deal with while being open-source. It's useless to call for a non-profit search engine without tackling this very core issue.
They haven't given up, they are staying exactly where they need to be.
All the mainstream search engines' priority is to maximize ad clicks/impressions (or collect data to target future impressions), either directly on their own property, or indirectly when linking to websites that embed their ads.
There's no reason why they can't detect ads or analytics and use that as a negative ranking factor (so that all other factors being equal, a non-ad-infested result would rank higher than the ad-infested one), but this would go contrary to their business model.
I think it's possible - red flags (for example blog spam or commercial sites) seem easy enough to catch for a human; probably in an automatic way too, at least as long as you're small enough that they don't specifically target you.
Google's search results are so bad I can't really alledge incompetence here but have to wonder whether there's some different motivation. Maybe it's that low quality search results tend to be plastered with ads, which they get a cut on.
I bet Google spends at least as much to combat that, and it's extremely hard to deal with while being open-source. It's useless to call for a non-profit search engine without tackling this very core issue.