I say this as someone who runs several publicly-facing websites, and also scrapes a few others for use in my own tools: why do people care so much about bots?
WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.
Are you using lambdas or something like that to serve a blog?
Is it basically just OCD? Vulnerability protection? Old habits from a time where crawling really did impact service?
In my case it's my Forgejo instance. My blog is static files with a limited amount of pages to crawl and indeed it's no issue at all. Forgejo is a dynamic service with an infinite amount of pages to discover and it runs git in the background to generate (some of) the pages. Tons of bots can easily overwhelm my small server. I could hide the repositories, but they're open on purpose (it's open source after all).
I have it guarded by a simple cookie check now, which seems to work well enough. Only a small subset of bots pass it. It likely kills search engine discoverability, but that's the tradeoff I have to make.
At home, I can't afford a 40gbit connection and a server to match... getting DDoSed by a handful of malicious google cloud VPSes running nmap and various web vulnerability scans can easily impact performance on low end hardware.
Yes, in the grand scheme of things, waiting a few additional seconds to check my local imap server (because my DMZ gets bogged down) isn't a huge deal, but that doesn't mean I have to like it, or allow it to continue.
I have a personal site on a shared host that was recently suspended for excessive CPU use caused by incessant AI bot crawling. I don't have a problem with crawling per se, there's just so many and they're so bad at it.
I run an ecommerce auction website on wordpress that is highly dynamic and can't be cached. Bots are a waste of cpu. I even turned off "prefetch" requests as I found that (android in particular) browsers will prefetch every auction item on the page as the user scrolls down the summary page looking at items.
Bots routinely overwhelm my public-facing websites. The last few times I've turned on "attack mode" in Cloudflare (which I do when bot swarms get particularly nasty), traffic dropped by at least 20k requests/minute (which is huge, considering my human traffic is around 3/4 of that).
It's mainly that dealing with them is a hassle, time that could be better spent doing other more useful things.
Over the weekend I got round to turning off some old viewvc (CVS and subversion) and hgweb (Mercurial) web interfaces. They'd been running fine for a decade or two with basic security updates etc, mostly only occasionally visited for real purposes, but that's fine!
But getting 2.7 million requests/day from residential proxy IPs was a bit taxing for the old uwsgi/cgi programs (that's average 30/s), making other sites on the server suffer. It was also getting close to 1TB/month which is the VPS quota.
So I ended up turning them off, one more cut towards the centralised internet. Sure I could put a cache in front, but there are probably millions of combinations of dynamic urls in the VCS tree so caching mightn't help much anyway, not worth the effort. I'm sure the web server could be tuned better too, but again it's a waste of time.
30/s is preposterous. Are these loser companies who don’t want to cache on their end or are they getting unique info per request or could it really be that many different origins or…?
For me it's just a fun exercise to see what generalizations I can make about bots that botters can not really avoid or code around such as javascript. Yeah this blog is all precompressed static content in a ram disk. I'm sure it could handle a few hundred thousand requests per second. This is to show methods people could use on their forums, chan sites, chat servers, etc... Every option is optional and tunable. Any option people consider should be tested on a test server. People could just point and laugh if they so desired and that works for me too.
As of now roughly 4661 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers.
I could improve this by moving most of the blocking to a demo site so the article is easier to access. I will do that. I will also update some of the nginx ACL's to what I am currently using that explain to the botters what is out of place.
As of now 6622 real humans made it through and a small number of what look to be bots. Curious if any of these people will try their own experiments on this topic.
i guess the thing i'm most confused about is what is the higher level goal here. 1 in 8 humans on the planet are experiencing the "web" through chatgpt alone. many have migrated to purely agentic workflows.
is the goal for your content to just be invisible to this growing population? is the expectation that all of this is just a fad, which will fade away? what is the end game to the tactics you have outlined? what is the strategy?
I like those ideas. If people are truly operating purely through LLM's then I am fine with being partitioned from them. The LLM operators will provide their reality and their truth. This is all for low trust internet-wide access.
For smaller higher trust communities I don't do any of this. Rather we use basic authentication to keep bots and strangers off the services all together and then regular user accounts on forums and such. That is where strategy comes into play and has been working well for some time.
i guess what i'm curious about is where you would draw the line, and why.
how do you define a user agent? is it desirable or not for users to be able to discover and access resources and communities on the internet using the tools, formats, and workflows that they prefer?
are search bots desirable?
the internet wayback machine?
how do you feel about browser extensions and greasemonkey scripts?
I do not define user-agents, they announce themselves and present a particular behavior. I let them define their behavior and I respond accordingly.
I have no need for search bots personally. If I had a complex site I would build my own search feature. If it was a commercial site I would pay for advertising on popular sites that were of related categories.
Wayback is fun to play with but it's just a toy to me. There is no concept of domain ownership. Most of the snapshots of the domain I am using were from when someone else rented it. They claim to respect robots.txt but that is not entirely true. They crawl and save content even if a site says not to and as soon as the site is offline and robots.txt is no longer accessible they will display all the archived content. This means a person has to park their domain on a server containing a robots.txt that matches their intentions.
Browser extensions are mostly invisible. What is obvious to me is when people are using "reader" applications as web clients. They should be concerned more than me. They are reading random sites run by strangers using apps that may or may not have been battle hardened, reviewed by third party penetration testers and so on. Some of them are vibe coded in unsafe languages.
I wonder what the click through rate is for the source links. Based on what I've been hearing, it's very low. And that's if the particular LLM/interface/answer even bothers to add them.
If the bots are eating your site traffic and users, what's the point of allowing them in?
I am an active user of Claude but I do not live my life through that thing. I try to get it to answer questions that it does not want to answer. The only questions it seems to answer without hesitation are technical in nature. In the entire existence of my account I have started 9 chats.
More interesting to me was that Alexa on Amazon Prime answered all my questions without hesitation or disclaimers but once I have my answers I close that tab.
This is definitely a 2026 type comment, but I'm very surprised that it's so "low." GPT is the most well known brand and it's the one that seemingly 100% of kids use to "assist" on their homework and other exercises from grade school to college. One would think that alone would already take you well over a billion.
Not pointing and laughing, just wondering why (since I see so many posts about this). Sounds like it's just for the thrill of the game, which is completely valid.
One of my issues had to do with pure bandwidth, dozens of bots were requesting the same image over and over again adding random crap to the query. My server was being slow for me during crawling peaks.
> An actual proper static site would likely 10x that.
A lot of the stuff I host isn't static. Any non-static content also gets overloaded pretty quickly. Hosting something like a Git server can turn pretty resource intensive if shitty bots start diffing every commit with every other commit.
I'm generally lazy when it comes to blocking IPs, but when I do run into an issue, I just blanket block IP ranges. When that doesn't work, ASNs get blocked. So far I haven't had to take most measures mentioned here but I probably will once the broken bots start coming from botnets using residential IP addresses.
On a dedicated server? It must be somewhere like Korea, where bandwidth is extremely expensive? Usually, dedicated servers are already expensive enough that they throw in unlimited bandwidth for free. And when not, its usually more like 200TB/m
> WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.
Yes we know - I still had my site knocked out a couple of times by overeager bots. They can be really abusive.
Just a personal blog, and I have to admit it had a misconfiguration that was my fault - nonexistent urls would redirect to the index. Still, dumb scraper bots would 1) not realize there was a redirect loop and kept trying every possible url, and 2) they were doing so aggressively enough that it brought the site down a couple of times. A well-behaved crawler would have slowed down and noticed the loop, never had issues with crawlers for years until the advent of ai.
You need to be careful with blocking based on IP. IP blocks get reassigned every once in awhile, and you can end up blocking the wrong people. I've seen multiple cases where a block of IPs was blocked either because based on geography, or being a data center, then the block changed hands to a residential ISP.
Blocking http 1.1 has a higher risk of blocking real users if they are using old browsers.
> One class of bots always claims to come from the root page of google. Obviously this is a lie.
This isn't just bots. Some browsers won't include the full url for cross-origin requests, so that is the referrer if the request came from google, so you probably don't want to block that unless you don't want any traffic from google search.
For what it's worth I block all of Google's ASN's entirely. Just me personally for my silly hobby and tinkering sites, I have not received any good traffic from them as of late. Even their search has gone to pot in my opinion.
I will keep the blocking of http/1.1 at the risk of blocking ancient browsers and API tools. That's just too easy for people to update these days and they are not missing much by not visiting cranky ol' me. If we were talking about proprietary code running in an old financial system I would totally get it, but the public internet can and should update for their own sake.
Any request coming to me claiming to be from Google I know is a lie as I have been blocking Google for a fairly long time. If anything Google would say I am malicious for blocking them, or at least they used to. I cycle by blog through many random domains to break any associations and related snapshots to disrupt birds of a feather associations. I try to control how people discover my silly ramblings.
I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.
When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains likely to figure out which records to avoid and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP's being all over the internet.
Oof, Germany. That's a tough place to be doing cybercrime from. If you told the police, I guarantee they can get that guy's home address. Whether they'd bother is a different story
No they were specifically trying and failing of course to AXFR from the only domains I had added the zeroconf addresses to and shortly thereafter all the scanning went quiet. If they were not skiddies they could have just updated their tools to ignore private address spaces from all domains all together. Don't worry, I am not reporting them.
There are loads of big TXT records on many domains that can be used for amplification attacks. [1] These are just a few by the way, I transparently filter many TXT records on some DNS servers for this reason.
You are only helping the entrenched browser monopoly and furthering the dystopia if you attempt to block anything but "approved" user-agents. This is what people like RMS were warning us about decades ago.
Block on traffic volume and request frequency if that's causing a problem.
(And yes, I can't access the site either. No, I will not conform. But I bet anyone determined enough will still get through, just like with DRM.)
I can accept that. I've sat with RMS a few times. He's an interesting and very smart person. There are things he and I agree and disagree on. I'm sure I would never hear the end of it on this one. I'm glad he put the cancer into remission and will have many many more years.
On the topic of browsing with any kind of browser, I would be extra careful using things people have vibe coded or more specifically have not been battle hardened to protect against malicious sites. The reader apps people use I think would be especially at risk of malicious servers if they have not gone through extensive third party code reviews by penetration testers.
Or behavior. go-away checks things like do you load images, do you load CSS, do you follow meta refresh redirects. Anubis basically checks you can run JavaScript for a few seconds.
That's why Google introduced this secure key thingy so that you'd have to claim to be Chrome in a way that Google hopes is either legally binding by itself or a copyright infringement of the keying algorithm.
I don't remember what it was called and I don't use Chrome to test it. It sends a header with a long hex string that supposedly proves it's real Chrome.
…and not a bot… hoping not to turn off iCloud Private Relay to read, well, anything. (Maybe Google Scholar if I really needed it, the most mainstream thing I know of with a complete and effective iCloud relay ban.)
*edit: per author’s reply elsewhere, as a test site, rather a good implementation! But other webadmins, please don’t adopt all methods if you can avoid it :)
That makes sense. I block most data-centers and that would include much of Apple's data-centers. What I am doing is of course unorthodox for a blog. This is more of a test site to show what could be done. People can pick and choose which features they like for their particular use cases.
I could some day split it out into a blog and a demo site.
It's crazier than that. I block two entire continents. Don't worry you are not missing anything. There is a link to archive.is that mirrors the page if you are curious.
I’ll tell you why this isn’t a great experience for me. I try to actively participate on HN, and based on a title I click on posts that promise to be educational or thought-provoking. The topic of combatting bots is very much on my mind.
Instead of learning something I get a blank screen, for the crime of being interested and living somewhere you deem block worthy. Cool.
Oh I came around myself. We got a blank screen and then the author was awesome and explained themselves and I'm satisfied having this one link as an extreme blocking test. Especially because they were thoughtful enough to archive the site themselves.
Our situations to be fair aren’t equivalent. If my block were due to my region instead of an optional paid service, that would feel worse. Maybe it wouldn’t have been the first time either. Whole continents written off!
Let’s certainly loudly complain if someone implements all these methods and inadvertently blocks us when they should’ve known better or don’t care about collateral damage. (If they’re apologetic, between a rock and a hard place with receipts, would require further discussion. e.g. is their choice really between serving us and closing shop)
Edit: but title could add “(but don’t)” at the end
Oh, I assumed Archive had an automated* tool in their arsenal to overcome the blocks that would’ve affected them. Didja happen to confirm they were blocked initially?
*I think they do something manually sometimes, like logging into a The Information Pro account and mirroring popular articles. (Unless they can use a visitor’s account somehow, like with some browser extension…) And suspected without evidence maybe The Information did some watermarking to ensure they could quickly ban accounts used like that.
Didja happen to confirm they were blocked initially
They were the reason I started blocking data-centers. I wanted to see if I could block all of their addresses which turned into quite an exercise. Initially I was able to block them by blocking TCP SYN packets with an MSS of 1380 and 1300 but talking about it here was a mistake as their admin is on HN. They fixed that and made their MSS 1460 like a nominal home internet connection.
The next thing I had to block was the Russian Federation. That seems to be where the controlling nodes are, I think. That made me even more curious who they were so I just kept iterating through the blocking process. I ended up blocking a number of ASN's and all of Russia. They are all in the linked archive file in the article but I don't specifically call out the ones that are archive.is.
Most on HN think they operate on a shoe-string budget but I have my doubts. I suspect an asset of a technical arm of an intelligence agency but to what purpose and what end I have no clue. They are very clever whoever they are and certainly earned my respect. Either way I think their site is useful, I just wanted to see if I could block it.
> Initially I was able to block them by blocking TCP SYN packets with an MSS of 1380 and 1300 but talking about it here was a mistake as their admin is on HN. They fixed that and made their MSS 1460 like a nominal home internet connection.
I expect >99% of my web traffic is bots or agents and I was actually considering removing the page counter as it is pretty meaningless and makes my site look far busier than it is. I am reluctant to however do anything about it just in case it accidentally stops a genuine human reading it or downloading my books.
The comments can be misleading. About 2600 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.
Rejects me too. Not a bot and connecting directly with a "normal" old-fashioned up-to-date web browser. How will you evaluate how many humans vs bots you blocked?
I used to log the bots too but there are so many they drown out the humans. I fully expect to block a lot of real humans as many people today use VPN's and most VPN's will egress from a data-center. Maybe it would make sense to break this out into a blog and a demo blocking site some day.
As of now roughly 4385 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers.
the top tier ai scaper and bot defeat companies, they are big business. really they are not grey market at all.
the real humans you are seeing, are bots.
brightdata is one vendor, they have 150m genuine residential ips, aquired through standard business.
actually, since this was kind of an invitation, i scraped it. the volume is so small compared to the ai corpus, this volume does not cost anything. as i suspected it was a 100% success rate. the only page it did not reach was the one needing authentication. the tests, they were fine. of course, ai knows ascii from this type of data but it wouldn't be able to read the page unless you use a screenshot.
the bots are not interested in the headers. they never see them. imagine a normal person at home using their browser, that's the bot. the reason this is the case is because the bot ip is someone's house (at least for premium proxy), and the browser fingerprint is a recording of someone's browser. oversimplified of course.
the 'bots' are the public facing corpo scrapers, and the 'users' are bots run by data vendors.
to test this, there is a ranking on 'foil' bot detection. to actually detect them they need 0.05 usd per check, and they are more interested in fraud via automated interaction and account creation.
grab a free account, you can very easily collect such data.
I just use fail2ban and it seems to do a pretty good job, but you do need to pay attention for the first couple of months, to tune it to your situation. I use one filter that is really just failregex = ^<HOST> - \S+ \[\] ".*?" 40[034], and then if that catches something I add it to another more specific list. So far I have about 80 specific regex's and it has stopped all of the traffic. I haven't had anything fallthrough to the 40[034] filter in a long time.
Of course the problem with fail2ban, and also OPs solution, is that it become a pain if you are handling things through a load balancer / proxy, since you need a way to get the real ip.
load balancer / proxy, since you need a way to get the real ip
If it is at all helpful most layer-7 load balancers have an option to enable a header that contains the real IP address, then one need only reconfigure their web daemon to log that header. It's very similar to how CDN's send the real IP in a header.
Most load balancers will silently drop the real IP address header on the ingress side to avoid exactly what you are describing. That used to be a thing around 2000 - 2001 era. If you find a load balancer with that issue today open a case with their vendor.
Differences in parsing between two systems can allow data to sneak through. For example you might put a space before the header name. The reverse proxy might treat it as a header name starting with a space, so not matching the one it wants to remove, and not removed, while the origin server trims the space.
This is a ubiquitous security concern, like IDOR and memory safety. Every time you have a separate validation step for anything, you have the possibility of a parser differential vulnerability.
If you don't know it yet, look up "HTTP request smuggling" - pretty interesting.
The post content is great. I personally hate the way Cloudfare has been the „default answer“ for the bot problem because Cloudfare has become the most successful MITM attack in history. We need content like this to keep the internet alive.
The added explanations by the author in this comment thread are hilarious. You sir are a good writer.
Thankyou for the kind words. I am not for everyone which is the way I would like to keep it. I agree there need to be more alternatives to a big centralized MitM caching proxy even if they can be a bit rough around the edges like mine. There is always room for improvement if enough of us try to come up with alternatives and options.
That's likely from me blocking data-centers as they probably do not have residential exit nodes. There are a couple nftable rules that could block most VPN's as well but they also block some cellular networks so I left that out.
I can spot them but the feds are working on taking down the biggest residential proxy malware nodes. It's a game of whack-a-mole but I will let them play that game. There are some easy ways to spot some of them but I will save that for another article.
Not all residential proxies are malware. You can get paid crypto to offer up your connection. Prices will go up because of the malware crackdowns but the industry will still exist.
> You can get paid crypto to offer up your connection.
Voluntary malware? Funded malware facilitation?
I'm not against crypto, but it would sound more legit if you removed the word crypto from your sentence. If it's exclusively the ability to be paid in crypto, as opposed to local currency or USD, then I'd guess it leans pretty heavily to shady-ness.
I pretty much think all residential proxy traffic is undesirable - otherwise it wouldn't need to use a residential proxy (they're used to get around erstwhile blocking, which someone has actively chosen to put in place - rightly or wrongly).
We've already established that you wish to block basically the entire internet including two actual continents, no wonder someone who was really interested in your site would try to view it with a residential proxy.
How about phonecalling someone with an unblocked connection and having them send pictures of the pages. Would you block that if you could?
I did not and could not downvote you as you replied to me. Only others can. I have not downvoted any comments on this thread. I appreciate all feedback.
visibly ravaged by the more shady denizens of the global internet
That is both funny, sad and oddly appropriate. Indeed, decades of running forums, chan and chat servers. I came here to take a break from all of that. It's been a nice vacation so to speak, letting someone else do all the management of a user-content driven site. Now it's all just tinkering and experimenting for me.
I would be interested in some examples of valid traffic that they would block. For the purposes of the document I do not consider anything from a data-center to be valid traffic. People can of course skip any or all steps, experiment with one at a time on a test server as they should.
This may block schools and libraries that use content blockers. Often the internal client is left to make abuse tracking easier (or because the overworked admin didn't know they could turn it off).
Oh, well that is ok for me I suppose. I add RTA/adult headers that hopefully they also look for and block using parental controls as adult content should not be viewed in a school or library. I could add a note suggesting to skip that step if one wishes schools and libraries that may be using a proxy to view.
I think it's a great article to be fair. We need more of this cheap and quick bot blocking. The fact the solution to unwanted traffic is often "use Cloudflare" is _not_ great for the internet, and nobody really actually likes deploying or managing ModSecurity. Its a nice middleground.
VPN exits often "come from datacenters". And there are entire countries where the web can be unusable without.
If you don't have LaLiga in your country, that doesn't mean everyone is so lucky. Blanket IP range bans, in this day and age? Basically a proclamation of incompetence.
I understand your frustration. Normally a blog site would not have any or many of these measures enabled. This is more of a test site to show what could be done on any other type of site. People can pick and choose which methods to use. Some may wish to block VPNs and proxies especially on forums.
I should make an article that shows additional ways to detect VPN's, including residential.
There's a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz's, media, etc.), basically anything i might need to wget on a server.
A guy gets sent to hell. The demons guide him into the lobby. Satan jumps out and lets out a big evil roar to no effect. Satan then glances down and sees the spot on the mans finger where there was a ring. "Oh... well you've been through worse. The break room is down the hallway to the left, mail room is upstairs to the right..." Satan just walks away.
Every step is optional of course. There are ways to make curl work on my site but I choose to add friction as some botters abuse libcurl. I doubt anyone else will do what I do. They could spoof the user-agent but most botters seem to not know how to do that despite the myth that they all do.
For what it's worth if I had a site specific to sharing code artifacts or archives I would not block curl and I would also enable native rsyncd.
There's this weird entitlement people have in which they think that the author of a site is not allowed to choose the method by which they protect their own site because it may be inconvenient to their specific, niche, use case.
Although @Bender's unique answer is better than my obvious observation.
So long as a some half-decent explanation of reasoning is provided.
Plenty of people would say that I shouldn't block residential proxies, but I would 100% block residential proxies if I could.
I think most people responsible for blocking traffic would actively manage their rules if it was costing them something valuable. If they don't notice, then, well no harm no foul.
Am I the only one that exclusively gets attacks with spoofed user agents and rotating TLS signatures? I feel like every post I see about not needing a CDN has tips that could be overcome in under an hour of scripting.
But they don't script it. Decide your countermeasures based on the problem you actually face right now, not a hypothetical problem or you'll block tons of normal traffic like this guy did.
WordPress with a cache can handle on the order of 1k requests per second on the cheapest VPS money can buy. An actual proper static site would likely 10x that.
Are you using lambdas or something like that to serve a blog?
Is it basically just OCD? Vulnerability protection? Old habits from a time where crawling really did impact service?
reply