I am not trying to start a debate of sql vs nosql, but i have an honest question. What if foursquare was using a standard RDBMS like Oracle? Would they have run into the same problem?
If your working set (the data that you are touching regularly) grows past available RAM, you very rapidly switch from a system where disk access plays no part in most queries, to a system where it is the limiting factor.
Because disks are so much slower, and having queries take only ten times as long can cause them to rapidly queue up, you end up with an unresponsive system. The appropriate response is almost always to turn off features to get the size of your working set back down, which buys you time to either alter the app to need less data, or get more RAM into the box.
