>For example em-activerecord just didn't work so well under huge load (dropping SQL-connections), so I had to write my own mini ORM for the workers.
That's interesting - could you elaborate (here or a blog post) on your findings. what was different between your ORM and em-activerecord that made it more performant.
The thing is - my first thought would have been to leave the ORM alone and focus on the DB side (like more sophisticated connection pooling/pgbouncer, etc. ). Which is why I'm interested in what went wrong in the ORM that made it screw up when used in a non-blocking kind of a setup.
> That's interesting - could you elaborate (here or a blog post) on your findings. what was different between your ORM and em-activerecord that made it more performant.
I will do a blog post about this when I finally have some time (christmas holidays, maybe). Em-activerecord worked very nice first, but when I hit it with thousands of concurrent jobs, some workers just dropped dead saying MySQL couldn't answer. This was annoying, because the workers didn't really fail in Resque, just failed to do their job and if this was in production it would've cost us thousands.
So, my own ORM is just a database superclass with em-connection-pool, configuration, openstruct and couple of class and instance methods (insert, update, find, query). And now this thing is really really fast, using lot less of sql connections (delayed job had 300 connections, now we'll need only ~40 connections) and scales really well.
The thing here is, that when I'm not using Rails at all, I don't know why I should use evented Ruby instead of Node.
This is why theres people who say you shouldnt use an orm at all. I have to admit intially i liked the django orm however after a while i found that i prefered or wanted to make the called directly myself. There was just oo much mystery going on behind the scenes that i couldnt find out about.
You can probably use both node and ruby. From what ive seen sites that are using node use it more for the backend to serve ajax then for serving the web pages. Although i have seen few site with very ajax based interfaces that apparently use just node. It really depends on your ui i guess. Good luck with whatever you choose though!
forgive me for digging deeper - but could you narrow down the problem _why_ your architecture better. My gut feel is that you are leveraging the connection pool better - because thats the only thing that can explain DB connection issues. The other thing could be that your em-activerecord connection objects might be too heavy, so you're hitting some kind of ulimit/open-file-descriptors problem (basically OS issues).
This seems like exactly what would hit anybody who is trying to build a worker model for processing data... hell even for sending emails. It would be interesting to see where you are pushing the limits - it could even be something deep like
I tried to dig deeper to em-activerecord, but it really was a better idea to build a very simple orm layer with specific sql queries instead of a more complex solution.
This problem is noted by the developers of em-activerecord and em-synchrony a several months ago already. There seems to be no progress, which might lead to a harder problem or that people are not using these libraries in bigger services.
But my point was, that Node.js seems to be much much more finished product compared to EventMachine for example. With EM you have to live without most of the Rails and for example testing the EM code with unit tests is not so nice job to do...
What I liked the most with EM are the Ruby fibers. The best solution for hiding the callbacks and writing nice and readable code by far. Too bad the fibers are a bit of a ghetto still, like somebody said earlier.
That's interesting - could you elaborate (here or a blog post) on your findings. what was different between your ORM and em-activerecord that made it more performant.
The thing is - my first thought would have been to leave the ORM alone and focus on the DB side (like more sophisticated connection pooling/pgbouncer, etc. ). Which is why I'm interested in what went wrong in the ORM that made it screw up when used in a non-blocking kind of a setup.