Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
codingminds
on July 17, 2025
|
parent
|
context
|
favorite
| on:
I was wrong about robots.txt
Keep in mind that those LLMs are one of the bigger reasons why we see more and more anti bot behaviour on sites like SO.
That aggressive crawling to train those on everything is insane.
Consider applying for YC's Fall 2026 batch!
Applications
are open till July 27.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
That aggressive crawling to train those on everything is insane.