We are currently building some core infrastructure with MongoDB. I looked at Riak, but it was capped collections and ad-hoc querying that really nailed it for us. Riak is a great product, I loved it's replication mechanism where any node was writable, but we didn't require that, so we went with mongo.
We had a 3 node system across a wan, where local clients would read from a local node, and if they needed to write, would write to the master across the wan. The issue we had was there was a significant amount of data being written to the master that one or two of the nodes weren't interested in 100% of the time, but it was still being replicated across the wan (and at cost)
so I investigated how the replication mechanism worked - to see if I could gain greater control over what data was replicated. As it turns out, there isn't, but you can emulate the replication yourself if you're interested. Well this is how we did it:
- master node has a capped collection (we call db.messages)
- slave nodes have a bit of mongo console javascript that execute tailed cursor querying for messages being inserted
- when a message that matches the query inputs is inserted into the messages collection, it gets inserted into a local db.files collection, which clients then read from
The added bonus is that occasionally we do need to replication additional data to particular nodes, so we just craft up a tailed cursor query that finds any messages we can, and pulls them across the wire
we make a fair bit of use of the adhoc querying in mongodb, so that was a massive selling point for us
speed was also a major issue - we have a average write/very high read requirement, and it's really really fast.
finally the platform was a consideration. We're mostly a windows shop, but we'll use linux where needed, and we were prepared to have riak running on linux if thats what we thought was bets, but it just didn't really fit for us.
the only downside was we use Delphi, and the existing Delphi drivers were.. not good, so we wrote our own, which I'm trying to negotiate with my boss so we can push onto github.
He probably never used Delphi and thinks it's slow.
I've worked with Delphi for many years, and its not far from the speed you can achieve with C++. Delphi is one hell of an underrated language for native programming.
Also, Delphi's standard library, the VCL, is great, and you can learn so much reading the source code, which comes with the IDE. Or does it? Delphi 7 was the last version I used.
Actually, I have used Delphi, wrote tens, maybe hundreds of thousands of lines of it early on in my programming days, many years ago.
Yes, Delphi, the language, which uses a static, strongly typed language, reference counting for memory management, and native ahead-of-time compilation can achieve good performance. That's a shocker. The problem is this isn't what it's intended for, and the standard kit is hardly optimized for it.
Reference counting? Except for some basic types, like the string, you have to manually manage memory.
And the other point you are trying to make, I still don't get it. You imply that the guy is doing it wrong for using Delphi where speed is important, and you come here and say that Delphi is, indeed, a fast language.
What if speed is not what the language is intended for? If the language is intended for other purpose, and speed comes as a bonus, I don't know how that could be a bad thing.
Yeah unfortunately the actual Delphi compiler is stuck in stasis, and has received no love in terms of targeting newer cpu instructions like SSE etc, which is why C++ can achieve better performance these days, back in the day Delphi was as fast as C++ for general performance, and Delphi absolutely killed it in terms of productivity.
We started out as a TP shop, so there is code dating back to the 1980s that is still used today (=\)
The rumor is Borland/CodeGear/Embarcadero lost the source code to the delphi compiler, which is the reason there had been no additions to the compiler, but who knows if that is legitimate, or should be an entry on snopes
interfaces are reference counted (mostly for doing com stuff, however I've seen some examples of people trying to use interfaces as a cheap/dirty "i don't have to think about it" memory management tool, badly)
everything else is managed by yours truly
please enlighten us as to what you think Delphi is intended for, and what the standard kit is actually optimised for
Reference counted memory management always involves some level of manual memory management. That's why garbage collection exists, because it's not really possible to get full coverage with ref-counting. Thanks for the education lesson though.
We had a 3 node system across a wan, where local clients would read from a local node, and if they needed to write, would write to the master across the wan. The issue we had was there was a significant amount of data being written to the master that one or two of the nodes weren't interested in 100% of the time, but it was still being replicated across the wan (and at cost)
so I investigated how the replication mechanism worked - to see if I could gain greater control over what data was replicated. As it turns out, there isn't, but you can emulate the replication yourself if you're interested. Well this is how we did it:
- master node has a capped collection (we call db.messages) - slave nodes have a bit of mongo console javascript that execute tailed cursor querying for messages being inserted - when a message that matches the query inputs is inserted into the messages collection, it gets inserted into a local db.files collection, which clients then read from
The added bonus is that occasionally we do need to replication additional data to particular nodes, so we just craft up a tailed cursor query that finds any messages we can, and pulls them across the wire
we make a fair bit of use of the adhoc querying in mongodb, so that was a massive selling point for us
speed was also a major issue - we have a average write/very high read requirement, and it's really really fast.
finally the platform was a consideration. We're mostly a windows shop, but we'll use linux where needed, and we were prepared to have riak running on linux if thats what we thought was bets, but it just didn't really fit for us.
the only downside was we use Delphi, and the existing Delphi drivers were.. not good, so we wrote our own, which I'm trying to negotiate with my boss so we can push onto github.
good luck!