I can imagine this being an extension to robots.txt - i.e. do I mind my business being robo-called, or would I rather send an API call to <x>. This is an interesting point in AI technology.