Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Anubis works because AI crawlers do very little requests from an ip address to bypass rate-limiting. Last year they could still be blocked by ip range, but now the requests are from so many different networks that doesn't work anymore.

Doing the proof-of-work for every request is apparently too much work for them.

Crawlers using a single ip, or multiple ips from a single range are easily identifiable and rate-limited.



Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: