It’s people that don’t care if they ruin things for everyone else.
Crawlers have existed forever in search engine space and mostly behave.
This sort of no rate limit, fake user agent, 100s of IPs approach used by AI teams is obviously intentionally not caring who it fucks over. More malicious than sloppy implementation
it is an ecosystem of social roles, not just "people" .. casting the decision into individual choices is not the right filter to understand this situation..
I'm not sure I'm following what you mean by 'social roles'. Which roles are you referring to here?
I'll disagree that it's not at least individual malicious choice, though. Someone decided that they needed to fake/change user agents (as one example), and implemented it. Most likely it was more than one person- some manager(s)/teams probably also either suggested or agreed to this choice.
I would like to think at some point in this decision making process, someone would have considered 'is it ethical to change user agents to get around bans? Is it ethical to ignore robots.txt?' and decided not to proceed, but apparently that's not happening here...
Crawlers have existed forever in search engine space and mostly behave.
This sort of no rate limit, fake user agent, 100s of IPs approach used by AI teams is obviously intentionally not caring who it fucks over. More malicious than sloppy implementation