Yeah, because most custom browsers are malicious. They have the data to prove it. This isn't a side feature, it's a direct feature that is 100% intentional. They maintain a backend whitelist of known "good" user-agents. Curl is on that list and there are a few others outside of the big players.

Most people building custom browsers are doing it to do something Chrome would disallow. One instance would only supporting one, weak-ish cipher forcing TLS to use a predictable cipher instead of choosing the best available encryption for transit. While I agree some people have cool browser projects that would be nice to use, it's a side effect of bad actors abusing the system. Most of the annoying parts of Cloudflare exist because bad actors have abused the system.

Any sufficiently bad actor will already modify their user agent. Who is this really stopping?

Bad actors who are bad at being bad actors, which is actually the bulk of bad actors.

It's maddening, but it's true. I've seen tale of people having to modify resource auto-generators that created URLs with hexadecimal identifiers in them because the sequence "ad" in a URL would trip ad-blocking browser plugins. You might ask yourself "how many ad companies worth their salt have 'ad' in the URL path?" and the answer is "The ones who are worth their salt might not, but the ones who are terrible do, and they're probably terrible at other things too, like letting malware on their network."

I suspect that the reason that bad actors are bad at being bad actors is that the income is rather marginal and can't attract skilled devs away from more legitimate companies.

There's somebody who can build a custom browser but can't figure out how to change the user agent string?

They're called "script kiddies" and the trick is: they don't build the browser, they download a kit someone else built that has a user agent in it and use it for whatever purpose they intend to.

I went to school at a place that had a policy of soft-blocking network access for any machine that a portscan detected had TCP or UDP 12345 opened, because Back Orifice defaults to that port and people who built trojan horses to allow remote access didn't change the default. It caught a reasonable number of owned machines every year.

Don't overestimate criminals; if most were good at being criminals, they could be successful in society without having to break the law. ;)

The intersection of information security and game theory is constantly paradoxical.

Check server logs sometime. You'll be surprised how many malicious requests come from user agents that aren't regular, current browsers.

If you're willing to load up a page when you detect something suspicious, as CloudFlare does with their "browser integrity check" page, you can also try to fingerprint the automated tool. There's often something unusual about the setup like odd browser version, strange global JS symbols, etc.

Completely possible to work around of course, but it does increase the effort level quite a bit.

> Most people building custom browsers are doing it to do something Chrome would disallow.

Chrome is not, and isn't meant to be, DRM. There are DRM extensions for that, but Chrome (and let's extend this statement to any other whitelisted browser) does not try to limit what you can do to a website. The only restriction I can think of is the common ports thing, but if you want to connect to port 25 (typically for SMTP/email), go ahead and change the about:config setting and you can do it.

User-agents are not make sence, because custom browsers can cosplay easily, just set the "good" user-agent. If some custom browsers have evil purpose, why it need show it off? Change user-agent is very easy.

Oddly enough, they simply don't do that. IDK why, but they don't. Also, there is a bit more to browser integrity check than just the user-agent. But, yeah. You'd be surprise how often I saw attacks get mitigated that were using some obviously bad UA. The attack themselves seemed sophisticated enough, but the UA was still a 12 version old IE UA string or "1337 browser 2000" or something dumb like that.

This can't only be based on user agents, otherwise it would be pretty useless. I can set my Firefox's user agent to curl if I feel like it, the same way malicious actors would just set the user agent in their scripts / headless browsers etc.

it's not exclusively UA, but in this post the author does say taking the most up-to-date Chrome UA did resolve his issue I believe.

You would be SHOCKED how many bad actors use an outdated UA or some random string they think is funny. This portion of CFs mitigation isn't meant to be hyper-advanced detection, just bounce out the low hanging fruit. They have other security services that aim to mitigate the more advanced stuff (like the WAF).

Is there a readily accessible process to get on said whitelist?

Because if not, what you're describing is a cartel colluding to keep the market controlled by oligopolies. Regardless of whether there's a good reason for them to do so.

Why doesn’t the bad actor fork Chromium or Firefox?

Fork? Everything you need can typically be done with instrumentation. Colleagues of mine do this sort of thing (on the request of the company they're targeting). The browser is headless, but still a full browser with a common resolution and everything, and is (virtually? completely?) indistinguishable.

> This isn't a side feature, it's a direct feature that is 100% intentional.

So Cloudflare is intentionally breaking the web? Good to know.

No they are doing their job of filtering out garbage from most websites, and it's an option that the site-owner can enable.

Is this such a novel thing to look for outliers in web traffic and offer ways to mitigate risks?

Both what I said and what you said can be true simultaneously. I have been increasingly down on Cloudflare because Cloudflare is cutting me out of an increasingly large portion of the web.