This article claims that these big companies no longer respect robots.txt. That to me is the big problem. Back when I used to work with the Google Search Appliance it was impossible to ignore robots.txt. Since when have big known companies decided to completely ignore robots.txt?