You’re giving permission to scrape by way of having a page accessible to the gen... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		Karunamon on Sept 10, 2017 \| parent \| context \| favorite \| on: Google will punish sites that use annoying pop-up ... You’re giving permission to scrape by way of having a page accessible to the general public. If googlebot is bothering you so much, install a two line robots.txt and it’ll go away.

jiggunjer on Sept 11, 2017 [–]

robots can ignore your /robots.txt.

Karunamon on Sept 11, 2017 | [–]

Googlebot, and every other search engine robot of note. Putting a page up on the public internet includes the risk that it may be visited.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact