they respect robots.txt at least major ones like meta, claude, google, openai, based on my infra observations robots.txt is enough in 90%, 10% is just banning ip ranges for couple of days but those are no AI companies