Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Essentially yes - but "previewing" link and sending that link to Bingbot to crawl and index is another matter.

Imagine you send someone a "private" link to a file...Bing sees that and indexes it for the world to see. Not cool.



Is this actually the Bingbot crawling for the index, or do they just use the same bingbot code to generate link previews?

If I have a well-tested crawler sitting here, I don't see why I'd write a brand new one just to fetch previews in outlook...


“Private links” should be covered by robots.txt. The only case I see this happening is for those “anyone with link” shares and those are easy to cover.


AFAIK, most of those "private" links are actually just "unlisted" but they're still public. I'm sure Bing is indexing those too...


Anything private should ideally be put behind authentication. If that isn't possible, than robots.txt. Search engines are _meant_ to index everything that is publicly accessible and not blacklisted on robots.txt.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: