I don't want to disclose the niche because it's so small that disclosing it will likely allow people to figure out what the project was. It's not something that's conventionally thought of as filled with spammers. I was pretty surprised at the total vacuum of professional behavior when I first got plugged into it.

The larger issue here is that Google's algorithms can't identify quality content without something that it already believes to be quality pointing out to it (and in a very specific way that Google detects as a "natural" link, whatever that means). This causes issues with innovative solutions that aren't immediately accepted within their niche, niches that are small and heavily paranoid, and in which incumbents maintain their positions by offensively seeking to harm and/or silence discussion of anyone they dislike, or other sites that deserve good rankings but are unable to get acclaim from either the NYT or the niche-equivalent. You shouldn't need an entity Google trusts to run a story on you to get good rankings. It'd be great if Google fixed that, but I don't really expect them to do so.

We should just make it known among real entrepreneurs that they shouldn't have reservations about SEO tactics (which are almost all somewhat uncomfortable, at least) and that they are required to compete so that no one else with a good, legit business gets quashed by a spammer because Matt Cutts and Google swears up and down that not only do you not need to do anything but "make good content", but that you'll be hurt if you take artificial steps to enhance your rank. It's BS -- those artificial steps are mandatory to control ranking.

Unfortunately, this is likely going to be a problem till strong AI becomes a thing, since there's no real way to judge content quality automatically without it.

Also, popularity may not be tied to quality, but to some degree, it's tied to what people expect to get when they search. Which is its own problem, since someone like Microsoft or Apple could do anything, yet people would be suspicious if they didn't show up for obvious queries. So it's a balancing act between 'rank the sites that might be better but less obscure', or 'rank the ones people expect to find because they already know about them'.

As for having reservations about SEO tactics... it should depend on exactly what they are. Something that hurts communities or users (like spamming forums or social media) is pretty damn sleazy, and is a quick way to commit business suicide. As is outright or borderline criminal behaviour for rankings.

>Unfortunately, this is likely going to be a problem till strong AI becomes a thing, since there's no real way to judge content quality automatically without it.

While this is true in absolute terms, there are ways to improve that I don't really know of Google doing (not that they aren't, as I don't have information on their internal workings). Google's algorithms seem to consider linking supreme and they don't appear to do any of their own research. I think a research/polling program to get users to rate some content would greatly improve search results. Other signals like the average reading level of the page's text could also be used to surmise the type and quality of content.

Overall, we need more humanity in the search results and less blind belief in the academic theory. This is a lapse in judgment frequently made by academics and mathematicians; they refuse to accept the evidence of failure that lies in front of them (since their theory/model/whatever doesn't say it should exist) until the failure reaches catastrophic, impossible-to-ignore thresholds. We need more humanity in a lot of systems, in my estimation.