I don't understand this. You don't have routes your users might need in robots.t... | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

Capricorn2481 5 months ago | parent | context | favorite | on: AI companies cause most of traffic on forums

I don't understand this. You don't have routes your users might need in robots.txt? This article is about bots accessing resources that other might use.

IncreasePosts 5 months ago [–]

It seems better to put fake honeypot urls in robots.txt, and block any up that accesses those.

trod1234 5 months ago | | [–]

Blocking will never work.

You need to impose cost. Set up QoS buckets, slow suspect connections down dramatically (almost to the point of timeout).

Capricorn2481 5 months ago | | [–]

Ah I see

Consider applying for YC's Fall 2025 batch! Applications are open till Aug 4
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact