Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Would blocking these bots from your site give bots that don't honor this a competitive advantage? Does that indirectly result in promoting not honoring robots.txt?

I'm considering whether to add it to my own site, but given that the future is already here and, while it's shitty to steal and regurgitate content without attribution at minimum, it's also not a big deal for my hobby site. It may serve my interests better to not include crawling restrictions for ClosedAI specifically



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: