Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I think that's the point, you'd use robots.txt to direct Googlebot/Bingbot/etc away from countermeasures that could potentially mess up your SEO. If other bots ignore the signpost clearly saying not to enter the tarpit, that's their own stupid fault.


Consider applying for YC's Fall 2026 batch! Applications are open till July 27.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: