The URL in the search is this: https://shoprocket.io/email-confirmation/34b35b1.... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		scandinavian on June 27, 2022 \| parent \| context \| favorite \| on: “Magic links” can end up in Bing search results, r... The URL in the search is this: https://shoprocket.io/email-confirmation/34b35b1... I don't see that in the robots.txt https://shoprocket.io/robots.txt User-agent: * Disallow: /cdn-cgi/l/email-protection Disallow: /login Disallow: /register Disallow: /404 Am I missing something?

rsbadger on June 27, 2022 [–]

You may be seeing a stale version, try this: https://shoprocket.io/robots.txt?bypass=1

(I made a lot of changes today when testing all, including "visit as Bingbot" from their webmaster tools with and without the URL blocked by robots.txt)

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact