> My issue is that crawlers aren’t respecting robots.txt

Cloudflare has a toggle switch to automatically block LLM's + scrapers etc:

https://blog.cloudflare.com/declaring-your-aindependence-blo...