Please don't take this personally, but I'd like to propose a new rule of etiquette on HN: it's unacceptably lazy to say that some business "needs to do better" without explaining exactly what they should do better, or how.
GitHub is doing better by proactively upgrading their network. During the upgrade, they have run in to some technical difficulties. There are maybe, what, a thousand people in the U.S. that have worked on network administration for something GitHub's size? And far fewer who could have predicted this kind of trouble?
If we're going to shit on someone, we should at least make it nutrient-rich: include recommendations based on experience from dealing with problems of that nature and magnitude.
Actually you are right it is a bit lazy. Ok so the first question is, are they multi-homed (as in multiple datacenters)? If not, why not? Because that should be the first step to take. We could talk about all the ways in which you can build redundancy and automated recovery into a cluster of machines in a single datacenter but let me tell you the obvious point that the datacenter is a single point of failure. Rackspace are an extremely solid hosting provider, we used them in my old company for 5 years but even they can suffer fiber cuts, power cuts, UPS's dying, generators not working, someone performing network maintenance and crapping out the routing for the entire dc.
My first recommendation would be setting up a warm-standby or even a hot-standby if possible in another datacenter. It all comes down to state, meaning the data/repos. They are currently using drbd for block level replication of disks in their file servers, this can be done over wan using something like drbd proxy. Another option is to write something in house which replicates write requests to git repos hosted elsewhere. They wrote a collection of services to access git repos on the file servers via rpc, its entirely plausible to extend this. If they are using mysql anywhere thats a pretty simple task for master-slave replication or semi-sync replication in mysql 5.5. Redis also supports replication.
I'm sort of hinting at high availability with read/write but starting out with a read-only solution hosted in another datacenter on the order of 60-120 secs out of sync is not impossible. I think we would all benefit from that kind of continuity right?
Replicate your data to a secondary site, keep it in read-only mode, set low ttl's in dns. Take it a step further and start looking at globally distributed file systems if you are feeling adventurous.
That wouldn't have really helped them here though?
If the closely connected storage nodes were unable to detect whether state was consistent, how would have the storage node in another data center known?
I guess what your saying is in an error making the site entirely read only from another data center. That could be of some use but I would imagine that would require a complete rearchitecture to separate out all writes and also have a method of recombining stuff like logging which will still be written later on.
On the git level though it would be useful for a major use case of github which is to grab code.
If its a fileserver pair that go down and they are serving a subset of repos then what do you do? Well you could keep serving all the other repos and start a recovery process, whatever that may be. Perhaps redirect the requests for those repos to another datacenter where there's another copy. Or make the decision of failing over the entire site from primary to secondary datacenter.
My point was really from the standpoint of a system and network wide outage for which it will take multiple hours to recover. Its about business continuity. I've been making comments from the standpoint of a user and the open source community but what about from the other side. I guess its a subscription like service for paid for customers so it has no immediate consequence on revenue but in places I've worked downtime is money lost and customers would actually require us to have SLAs and disaster recovery procedures in place before they would sign contracts with us. Anyway, yea the step from single server to multi server, to multiple datacenters does normally involve some re-architecting at the state layer, everything else is load balancing.
I understand they have a lot of nerd/hacker cred in and around the tech hotspots of the country, but GitHub is by no measure "big". The most recent number I could find puts them at 33 servers total. I've worked for non-tech companies that have that many as hot standbys.
One big recommendation I'd offer is to have a secondary network with as few moving pieces as possible (think a dumb unmanaged Netgear switch or two) to run things like heartbeats and DRBD over. That sort of thing should not be living on the same segment as your production traffic.
All of the major vendors MLAG implementations are limited to two agg switches, assuming you have a pair of 24 ports and a 48 port rack level switch hanging off each, that still puts total servers at less than 1000.
By that simplistic logic then Arista 7500-series [1] would give you a 384-port aggregation (per device) with 48-port rack switches hung off it for something like 8000+ servers.
What stops someone from having multiple zones within a datacenter interconnected with suitably large LAGs thereby avoiding being limited by the port capacity of any one device?
Why is the exact number of servers important to the topic, or is it just to satisfy your curiosity? :)
I'm specifically addressing the original parents straw man.
> There are maybe, what, a thousand people in the U.S. that have worked on network administration for something GitHub's size?
I didn't intend to deflate any nerd egos, just putting things in realistic terms. GitHub is cool. It's the biggest in its industry. It's definitely an interesting problem. But it's still hosted within Rackspace, which in itself isn't that big.
GitHub is doing better by proactively upgrading their network. During the upgrade, they have run in to some technical difficulties. There are maybe, what, a thousand people in the U.S. that have worked on network administration for something GitHub's size? And far fewer who could have predicted this kind of trouble?
If we're going to shit on someone, we should at least make it nutrient-rich: include recommendations based on experience from dealing with problems of that nature and magnitude.