Hacker News new | past | comments | ask | show | jobs | submit login

This whole yep.com thing feels like an attempt to get everyone to please stop blocking their bot



100% this right here. Ahrefs and Yep is not some altruistic endeavor.


Dumb question but would they not rotate bots in a situation like this? Or do they have a specific crawler


They have am aggressive crawler that I block. Why would I let a company use my resources to gather my data just so they can sell it to others for a premium at my expense?


They use a specific UA which is in the spirit of robots.txt, you're able to identify and allow/disallow access.

Trying to masquerade as another agent would be considered bad form, but obviously happens a lot.

There's a similar bot, MJ12Bot that powers Majestic's index which is similar to ahrefs. IIRC they have a user agent but their crawling is distributed, it's impossible to verify whether someone with that UA is them or someone else masquerading.

Good practice by bot owners is having a UA and known IPs they crawl from which can be verified by DNS and reverse DNS lookups.




Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: