Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Sure, but responding to the parent, that's why people don't crawl Google, not the Robots.txt.

And incidentally, some of those searches are to find forms that it can stuff links into. My sites are constantly getting hit by botnets searching for Drupal comment forms. Luckily it uses a quirky URL format that's easy for mod_security to block.



Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: