"It turns out that Ma.gnolia was pretty much a one-man operation, running on two Mac OS X servers and four Mac minis."
This should be a lesson: entrust your systems administration to people that have experience in this stuff. I'm not saying you have to find an expert and pay him or her expert prices, instead, you probably have a developer or admin friend who knows what it takes to implement a basic backup strategy that will at the least keep you from losing all of your data.
I watched a little bit of the podcast and he said he was just syncing the db files... well if it's innodb, that's not going to work. It says so in the MySQL docs. You know, in the section about backups.
The lesson here could be this: if you have a great idea, get it developed and out and the door -- awesome. Now talk to someone with experience in systems planning. Don't just throw a bunch of mac minis in a cabinet, bust out an rsync script and call it a day.
I don't want to throw rocks, because I used to work for Larry (in fact, I wrote the original codebase for Ma.gnolia.com) and I like him.
But I distinctly remember talking about backups and how a raw sync of the innodb files wasn't really what we needed (it could sort of work in the original deploy of ma.gnolia, but would not scale as the site scaled). I'm kind of surprised that the community has been so tolerant of this failure. Personally, I'd be furious if I found out that a site I trusted lost all my data because they hadn't even tested their backup system.
> I'd be furious if I found out that a site I trusted lost all my data because they hadn't even tested their backup system
Exactly - first rule of backups - test them. For a database, it can be as simple as using your nightly backup to create your dev or test database or whatever. I guess it isn't feasible with a half terrabyte database every day, but to not have at least a several week old backup set is somewhat unforgivable.
On the other hand, I bet he will never ever make the same mistake again, and it will make a lot of people on here think long and hard about their own backup strategies ...
Well, half a gig is trivial to backup. Half a terabyte is not. Assuming you meant to say 500 gigs, then yes, there is difficulty, but it's not uncharted territory. He had two xserves -- perfect for replication.
I think it doesn't matter what your infrastructure looks like, the principles are the same. You should not rely on a single service to store your data and/or do backups properly for you.
For example, I am building an application on top of EC2 (IaaS) and utilizing the nice EBS system with snapshots to S3.
This is still a do-it-yourself style in many ways, you still need to understand the database system, read those docs and you still need to test restores etc. It's not all that different from the Ma.gnolia situation. The main difference is that you get some nice properties from EBS and S3 redundancy.
But even with those nice properties, like hell am I going to hang the crux of my business on S3 snapshots only. Besides technical failures (there have been S3 data inconsistencies reported), what if they cancel our account? Etc. etc.
Another thing people label "cloud computing" is something like Google's appengine (PaaS). Here, you trust Google entirely to store the data safely in the "datastore" but its only integrity that they try to supply you, not backups. I have never heard of a way to get Google to take snapshots (which are important when you or an attacker delete something that shouldn't have been, etc.).
But you should do something like that anyhow is my real point.
Cloud computing is still a single point of failure because it's a "single company of failure". It may address your SPOF problems with buying and running hardware but I'm seeing people let that throw caution to the wind. Instead of "hard disks will fail, know that will need contingency plans" the situation if you choose to go the cloud computing route would perhaps be "trust the redundancy system they try and supply but be sure to have a contingency plan because there is always some possibility you will be hosed."
If you're building with something higher level than Google AppEngine, or Engine Yard, etc., the principle still applies. Even up to the consumer level, look at those people "building their bookmarks" at the higher level, they lost a lot of their hard work because they trusted a single company/system.
As a business, I don't ever want to put someone in that position where they realize they should not have trusted one system. That deteriorates any trust they had in our brand and is just shitty all around. And the only way to begin on that path is to not do that yourself.
Only in the sense that "Someone robbed an unlocked house" is a good case for sniper teams. I mean, sure, it will work -- but as an interim measure, locking the door is a bit easier, almost as effective, and far less messy.
This should be a lesson: entrust your systems administration to people that have experience in this stuff. I'm not saying you have to find an expert and pay him or her expert prices, instead, you probably have a developer or admin friend who knows what it takes to implement a basic backup strategy that will at the least keep you from losing all of your data.
I watched a little bit of the podcast and he said he was just syncing the db files... well if it's innodb, that's not going to work. It says so in the MySQL docs. You know, in the section about backups.
The lesson here could be this: if you have a great idea, get it developed and out and the door -- awesome. Now talk to someone with experience in systems planning. Don't just throw a bunch of mac minis in a cabinet, bust out an rsync script and call it a day.