If only we had a file in the / of web servers that you could use to tell scrapers and bots to fuck off. We'd say for instance:
User-Agent: *
Disallow: /
And that would be that. Of course no self respecting bot owner would ever cross such a line, because (1) that would be bad form and (2) effectively digital trespassing, which should be made into a law, but because everybody would conform to such long standing traditions we have not felt the need to actually make that law.
1. People in other countries probably don't give a fuck about your laws, global internet and all.
2. How are you going to define this law in such a manner that isn't going to be a problem for someone, for example, writing a plugin in the browser to manipulate the page for their own personal reasons.... 'scraping' is a very broad term that can easily include viewing.
