Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
pabs3
on Oct 20, 2020
|
parent
|
context
|
favorite
| on:
Tom Lehrer releases song lyrics to public domain
IIRC archive.org pay attention to the robots.txt now rather than in 2024.
moonchild
on Oct 20, 2020
[–]
They do not. They have in the past removed public access to pages due to robots.txt changes.
pabs3
on Oct 20, 2020
|
parent
[–]
Hmm, this old blog post sounds like they were planning on ending that practice:
https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
