Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
TeMPOraL
3 months ago
|
parent
|
context
|
favorite
| on:
Nepenthes is a tarpit to catch AI web crawlers
Interesting. Now they seem to claim that not only they follow robots.txt for crawling, but that they also broke under pressure and made the unfortunate decisions to have
user requests
follow robots.txt too.
https://www.perplexity.ai/de/hub/technical-faq/how-does-perp...
Consider applying for YC's Summer 2025 batch! Applications are open till May 13
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
https://www.perplexity.ai/de/hub/technical-faq/how-does-perp...