Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> Cloudflare can detect and block verified AI bots that comply with robots.txt and respect crawl rates, and do not hide their behavior from your website

It's the bots that do hide their behavior -- via residential proxy services -- that are causing most of the burden, for my site anyway. Not these large commercial AI vendors.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: