If you go to the about section of Yep.com, you can see this is powered by the "Ahrefs" bot (not just for sourcing crawler data but also run by the same people).
Ahrefs is a sleazy "SEO tool" ( https://ahrefs.com/pricing ) that basically exists to exploit weaknesses in search engine algorithms and other shady things (and make lots of money doing it -- $99/mo to $999/mo).
It sounds like you never actually used ahrefs, or even know what it does. You just read “SEO” and sent straight to “these are the people spamming Google with content marketing fluff posts when I search for a product review!”
What it does is showing you errors on your page 404, slow urls, pages that link to your site, missing tags, images that are too big and load slowly, keyword research and other things.
I use it even for my small projects (the free version) because it helps me fix broken links or improve the page speed which is all good for users.
Been using ahrefs for years, not sure what is sleazy about it.
It audits your site for issues, shows Google rankings for keywords, estimated volume for keywords, backlinks to specific sites, backlinks to specific pages...etc.
It doesn't do anything for SEO (no content, no links...etc), it simply reports on the state of affairs.
SEO itself is sleazy. Any gaming of search is sleazy, whether active or passive. The NSA is sleazy, even though they simply report on "the state of affairs".
Hi, I have a small SEO consultancy. We work with tech companies (mostly B2B saas), many you’ve heard of.
You’re right that some approaches to SEO are sleazy. They are a bit old fashioned at this point, and usually result in short term results which crash and burn. Google does actually care if users don’t find a good result; they are working on improving the results every day.
But you’re wrong in the sense that SEO done correctly is just a good way to get in front of potential customers who are actively searching for information by proving value to them up front.
Let’s say you run a company and I tell you customers are searching for your product 1000 times a day. What are you going to do? Not invest in SEO?
So, to demystify it, here’s what we do:
1. Identify topics that people are searching for, prioritize by search volume and conversion potential
2. Create content which is going to best satisfy that person’s search intent. We don’t want them clicking the back button and choosing another result, that would hurt our rankings in the long run
3. Try and link to the new content from other pages on our own site
4. Identify any technical issues which would prevent a search engine crawler from being able to access or understand our content
There's another side to the coin that I think you're, understandably, overlooking. I don't think either side alone is the full story. It's a big grey ball of tradeoffs, no matter how it all falls.
Even with just your own description of your activities you can find perverse incentives and negative externalities.
You aren't exactly incentivized to just create content that satisfies common queries. More fully, you're incentivized to create content that makes people think (rightly or wrongly) their query is being answered while also marketing your goods/services to them as much as possible.
Even assuming the best intentions to actually answer the query - be it altruism, discerning searchers, fear of Google, or something else - the content itself is hardly unbiased. This isn't inherently some kind of moral failure on your part, but it's hardly negligible either for the searcher. Everybody is going to have different thresholds here for what crosses the line, but I think it's understandable that some people are just unhappy having to even put in the time and energy to disentangle this aspect in the first place.
And this dovetails into one of the obvious externalities. Who is doing SEO, and who isn't? That totally unaffiliated small-time blogger that just happened to write up a bit of info after having the same problem as you probably isn't doing a whole lot of SEO. At least, they aren't hiring you to help increase their hosting costs. Somebody trying to find less biased views ends up having to wade through all the sites making enough money to pay for SEO first, to actually find what they're looking for. It should be pretty obvious why that does no favors for SEO's reputation.
But as I said at the beginning, you're not wrong either. This stuff goes deep into 'small' questions like profit-motives and competition for limited resources. Obviously and trivially correct answers aren't here.
I really do appreciate that you're approaching this from the much better intentioned side of things. One of the few things that might be obviously correct is that your approach is better than the sleezy way.
But that doesn't make it all rainbows and sunshine either.
I think you can find exceptions in any discussion, but that's not exactly helpful.
Here's the thing: If I don't answer the user's query accurately, another website will. Then they will get all the traffic. So my long term incentive is to satisfy the query.
Within that there is a certain tolerance and understanding among search users that websites can't publish information for free. See Wikipedia constantly begging for money.
So companies (my clients) can justifiably get away with a small amount of self promotion. But if you take that too far you'll just get punished in the SERPs too.
I get this sentiment. "SEO" is basically an euphemism for spamming at this point. It's no longer just "Let's repeat this keyword 3 times" but "let's add 120 footer links and spam 35 blog comment sections per day"
I don’t think it’s how it works because I see that crap daily in footers, especially in the travel industry. Open any hotel listing on TripAdvisor and count the “[something] in [location]” links in the footer.
How do you think Google knows which pages are helpful when someone searches for “[something] in [location]”? Anchor text in links plays a key role, but with search listings on sites like TripAdvisor, it’s tricky to naturally insert those links into content. Footer links is one way to do it.
I’d agree With you if those links were misleading, but they usually aren’t. They really do point to pages with “[something] in [location]”. And they really are helpful for search engines looking to give useful results for that type of query.
They have am aggressive crawler that I block. Why would I let a company use my resources to gather my data just so they can sell it to others for a premium at my expense?
They use a specific UA which is in the spirit of robots.txt, you're able to identify and allow/disallow access.
Trying to masquerade as another agent would be considered bad form, but obviously happens a lot.
There's a similar bot, MJ12Bot that powers Majestic's index which is similar to ahrefs. IIRC they have a user agent but their crawling is distributed, it's impossible to verify whether someone with that UA is them or someone else masquerading.
Good practice by bot owners is having a UA and known IPs they crawl from which can be verified by DNS and reverse DNS lookups.
The massive amount of github/stackoverflow content clones sitting at the top of the results for programming related queries seem to disagree with this sentiment. Ok I guess they are "creating content people want to see", but if just copypasting everyting from another source is enough for that...
Also the large amount of bots creating fake accounts on forums (my experience is with Discourse) with profiles and spam posts to SEO boost some site indicate a whole shady background business going here.
Yeah, reminds me of anecdotes from the Dan Lyons book [0] about Hubspot. There was the shiny fun stuff happening "front of house", but back in the boiler-room it was ugly. And everyone gets shown only what they need to see to make them feel happy/sign/pay, and nothing else.
No-one actually wants to know how the sausage is made.
It really depends on who you ask. The most "successful" companies don't whitehat their way there. Just open any modern social media website and tell me if they are respectful of the users — and you can see that part. SEO is mostly invisible so they really don't have to care about you.
Content that people want to see ... plastered with affiliate links and/or biased to sell those people what you want them to buy. This is about extracting value and that value has to come from somewhere. Searchers are not benefiting here, they are the ones ultimately paying for it.
Is it just that it isn't literally desecrating a corpse?
Do you have an alternative example of "something which a shove is especially useful for and which which could be lucrative, but which is widely regarded as bad" ?
If you go to the about section of Yep.com, you can see this is powered by the "Ahrefs" bot (not just for sourcing crawler data but also run by the same people).
Ahrefs is a sleazy "SEO tool" ( https://ahrefs.com/pricing ) that basically exists to exploit weaknesses in search engine algorithms and other shady things (and make lots of money doing it -- $99/mo to $999/mo).