It seems like these kind of forums are not exactly your area of interest, if they're community focused. Information gets snuffed out all the time, with every death of a person we lose large piece of information, but hoarding, and especially expecting others to hoard or assist hoarding is not the correct approach to what is essentially a _you_ problem, so grab that terabyte disk and make a mirror yourself if you are so inclined. Nobody is or should be required to let corpo behemoths in for your convenience and to comply with your questionable opinions.
I'm part of several digital archivism projects. My personal disk array is 54TB of data. That's without even getting into 1PB+ of data on LTO carts.
Last time I checked, Archive.org et al weren't a "corpo behemoth", but consuming server resources is exactly what a normal user does.
Site owners should get with the times and serve up cached static pages to users who aren't logged in. Even then, they should be serving up cached static pages and rebuilding cache for relevant pages when someone posts new content when it comes to forums. Not being able to handle a few crawlers is an administration problem. Why should the community/public suffer for someone's inability to configure a server appropriately?
We live in primarily free societies where individual has the right to decide upon their actions. Telling people that there is only one "correct" way of doing things is obnoxious and toxic and reflects upon your inability to see your opinion for what it is, an opinion.
My opinion is that "screw crawlers and scrapers" is a valid opinion. If i'm hosting a playground, it's my playground and my rules. If you want to play elsewhere, please do. If you want to preserve data, please do, but not at my expense.
Disagree with that? feel free to, but don't think that you are somehow in the right, because if you go with this shit to court, you will be laughed out of the door.
