Hacker News new | past | comments | ask | show | jobs | submit login

What stops a scraper from detecting Anubis and just removing "Mozilla" from the user-agent string?



That would allow you to specifically lock out that bot based on its user-agent string. That's the main problem with AI scrapers, many of them normally use user-agents that cannot be easily blocked, so other means have to be found to keep them off your grounds.


It breaks websites when you do that.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: