Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Does Yahoo! not have a robots.txt file?

http://yahoo.com/robots.txt Sorry, the page you requested was not found.



If that page doesn't exist, then according to specification, they don't and you can crawl any page.


Update: Some Yahoo! subdomains have 'em.

Yes:

    http://search.yahoo.com/robots.txt
    http://groups.yahoo.com/robots.txt
    http://realestate.yahoo.com/robots.txt
No:

    http://maps.yahoo.com/robots.txt
    http://omg.yahoo.com/robots.txt




Consider applying for YC's Winter 2026 batch! Applications are open till Nov 10

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: