Outside of disruptive measures like requiring accounts, captchas, or payment, one possible solution would be to use AI against itself by training machine learning models to monitor, flag, and issue challenges to web requests exhibiting "bot-like" behavior. This way, not all web traffic would be disrupted with challenges until the machine learning models have reason to believe the traffic is coming from a bot.