I'd like to add an independent voice too. I tried VoltDB out around a year ago. Ran it on 3 servers, with k=1 (all data replicated on 2 nodes). Servers were Dell PowerEdge 860 - only 8GB RAM, and a Q6600 quad-core processor. They cost maybe $500 or so. Even with a low-end buildout, I was able to hit around 110k TPS. This wasn't one of VoltDB's sample apps; it was a small proof-of-concept for a telecom application. They don't seem to be riding the hype train.
As far as open source: I was using the .NET client, and fixed a few issues with it. The folks at VoltDB were really responsive about working with me and accepting my patches. They also took time to explain the inner workings, sorta give me a launching point to investigating the core code. Pretty good experience for a commercial open source project.
Yes, very basic realtime billing/CDR storage. With traditional DBs, I have to resort to batching balance: Read N messages from queue, compute overall balance updates, then apply at once. With VoltDB, I can just fire off records as they come in.
TITLE DISCLAIMER: 695k using 20 fairly large machines
I was, at first glance, pretty impressed with the 695k TPS title claim. Upon reading the article, it became apparent that this was based on running 8 8-core EC2 m2.4xlarge instances for node.js in addition to 12 8-core EC2 m2.4xlarge instances for VoltDB. In terms of server architecture, this is quite a large setup.
Just for an idea of how big of a machine we're talking for each of these instances:
High-Memory Quadruple Extra Large Instance
68.4 GB of memory
26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each)
1690 GB of instance storage
64-bit platform
I/O Performance: High
API name: m2.4xlarge
$1.800 an hour running Linux on demand
$864.00 a day baseline for 20 of these
This isn't meant to discredit the author or work, just to make sure it's apparent that this solution is for serious businesses only with deep pockets and boatloads of traffic. You'd be spending $26,784 for the servers alone over a 31 day period if using them on demand.
The results are definitely impressive for the right target audience.
> On the database side, I configured VoltDB to run at K=0,
> which means the database was running with HA switched off.
> Most applications would not run that way in a production
> environment, but my goal was to test client-side throughput
> and scaling; I wasn’t really focused the database side of
> the tests.
Testament to the author for mentioning it. It makes me interested in voltdb, whereas before it was unknown to me. But the numbers should indeed be taken with a grain of salt I think.
The benchmark was very much about whether you CAN scale if you need it. You'd probably not rent this size hardware at Amazon if you needed it on a daily basis. So $26,784 is probably not realistic.
At any rate, as I mention in my longer report on the benchmark [1] the high total translates into very LOW hardware
requirements for lesser needs. The number to go by may rather be "10k+ TPS per core for linear scaling ACID transactions." I think that's the number that matters. Linear 10k+/core ACID TPS.
I'm not sure if I am supposed to be impressed or not. On my macbook pro with 1 redis master and 1 redis slave I can do about 300k updates / second to the master replicated to the slave. Comparable? Who knows?
These benchmarks don't mean much to me -- what matters is performance for your application.
I'll start by saying I dig Redis. I wish we had a storage engine that offered some of the kinds of flexibility that Redis does. If I were making Volt from scratch I would embed Redis or something with flexible schema and secondary indexes and spend less time working on SQL and more time working on distributed functionality.
>These benchmarks don't mean much to me -- what matters is performance for your application.
Very true. Think about how you would implement an application like Voter in Redis. Voter is an event counting app that applies business logic server side to do what is basically fraud detection.
Let's take a look at what each of these transactions is.
https://github.com/VoltDB/voltdb/blob/master/examples/voter/...
3 statements and an entire round trip to validate data is kosher.
1 insert/update statement that is actually updating two different summary views
You would need to implement your own locking to get the isolation necessary to implement the business logic and there would be at least two round trips for each vote. You would also have to write your own code to maintain the materialized view used to print the summary. Sure you could implement the validation logic client side, but then you lose the ability to transactionally update said logic and the state it depends on.
With the new scripting functionality in Redis you would be able to do it in one round trip but you would still end up writing your own code for the view and rollback. How you will attach a debugger to debug your server side logic is also worth considering. I mistakenly wrote that Redis doesn't cache Lua scripts, but it actually does.
Now that you have server side logic you will also probably want to start maintaining replicated state to make some of your transactions single shard transactions. With Redis you are on your own for updating replicated state in a way that stays consistent across the cluster even when restoring different shards.
Also, no matter how fast Redis is you may reach a point where you have run out of disk/network IO, or even CPU. Now you are looking at sharding Redis. You are probably going to want to maintain replicated state to make more of your transactions single shard transactions. With Redis you are on your own for updating replicated state in a way that stays consistent across the cluster even when restoring different shards.
Don't forget re-sharding and backups which are now your problem. You definitely want to be able to roll back to before that last application update that corrupted your data. Don't forget that you will need to bring the replicas up in the correct state as well. Preferably in parallel with bringing up the master since you are down right now, and it would be nice if it resharded at the same time since you took this opportunity to add capacity. Redis cluster is coming, but it isn't here today and depending on the application you may not want to risk it not being ready when the time comes.
You should at least double the number of ops (where an op is roundtrip to the DB) done by Volt if not more. That said, doing half as much work inside the procedure doesn't make it twice as fast and doing twice as much doesn't make it twice as slow. The dominant cost for such a trivial operation is not the operation itself. Redis is much better at this because it is single threaded and written in C. My bet is that the number of instructions and cache misses to accept and execute a command in Redis is much smaller because the execution engine isn't dealing with stored procedures, schema, undo logging, SQL, and code expecting to deal with distributed transactions even if none are in play.
>I'm not sure if I am supposed to be impressed or not.
So voltdb is an in memory SQL Db with some durability and replication functionality. Is there any good documents on how it compares with some of its competitors/ who are its competitors?
As an in-memory RDBMS, VoltDB competes directly against products like Oracle TimesTen and IBM's SolidDB. Comparatively, VoltDB's architecture is designed specifically to scale out on clusters of commodity servers (TimesTem and SolidDB are single-node datastores), giving Volt access to larger amounts of inexpensive main memory.
VoltDB also competes indirectly with NoSQL products like Cassandra, which achieve performance and scale primarily by sacrificing SQL and ACID transactions (Volt preserves both).
As an in-memory database, VoltDB offers durability through command logging and disaster recovery via a recently-introduced feature called database (WAN) replication.
I think durability should not be defined in terms of actual backing hardware. VoltDBs transactions commit and stay committed. If you restore from a backup, just like any other database, you'll have data loss until the backup point. What if my hard drive crashes when it loses power? At that point I'm in the same position as when I lose RAM.
VoltDB is often compared to other in-memory NoSQL systems, although I'm not aware of any that actually offer cross-server transactions. VoltDB is best when you have a high volume OLTP workload and need to really scale beyond what traditional RDBMS can do.
> What if my hard drive crashes when it loses power? At that point I'm in the same position as when I lose RAM.
Oh sure, except that the probability of losing your hard drive when you lose power is about 1 in a very big number. Whereas, the probability of losing the contents of your RAM when you lose power is 100%.
Now, MySQL (and in fact, most RDBMS outfits) provide tools for fixing corrupt data. But you actually need corrupt data to fix. That's the entire point of a hard drive.
I'm sure VoltDB would be perfect for a chat application or something of that nature. It would be terrible for anything that involved the transfer of $$$ (banking, ecommerce, etc).
In short, VoltDB offers tunable durability from single-node purely in-memory, all the way up to synchronous disk storage on multiple local nodes and multiple asynchronous copies in another data center. All options are fully transactional. All options are extremely performant.
We've worked very hard to make durability and performance not mutually exclusive. We have internally achieved six-figure TPS numbers using synchronous writes (requires a disk controller with a BBU, or a decent SSD).
I was making a more theoretical point that defining "durable" in terms of specific hardware isn't a very elegant definition.
In practice with VoltDB, you'll replicate to multiple machines synchronously (so you'd need to lose power to separate servers to lose data). VoltDB enterprise has command logging.
For a chat messages, I'm not sure VoltDB would be the _best_ choice, perf wise. Chat messages are usually more ephemeral - why go through the overhead of ACID for those?
For real $$$ transfers, you'll probably want synchronous WAN replication, to really ensure you never lose transactions.
Someone please correct me if I'm wrong, however I believe that VoltDB is highly tuned for a specific usage that benefits lots of small operations on large amounts of data. It is less beneficial, if not detrimental, for large operations on medium or large amounts of data.
You're entirely correct - VoltDB is very OLTP focused. Its predecessor, H-Store (decent paper here: http://www.vldb.org/pvldb/1/1454211.pdf ) ran on a single-thread-per-processor model that was very performant for short operations, but could obviously cause high latencies in the presence of long running operations. Presumably Volt runs on a similar model.
edit: ah, MichaelGG has explained it much better in another part of the comment thread!
Indeed, I'd never heard about it and found myself intrigued. After a bit of reading, it appears that VoltDB supports a very limited number of DDL and DML operations (essentially, you can create tables an views, and select, insert, update, and delete). There are a few basic aggregate functions like SUM and COUNT and no single-row functions. If you need to do anything at all complicated you're doing it in Java, or returning the results to the client program so it can do it.
VoltDB sound like a potentially great thing for an application that's really (and only) OLTP. But it's definitely not ready to replace Oracle, PostgreSQL, etc. for more general-purpose needs.
It's like node but isn't single threaded so scales over multiple cores without having to fork. Also it's polyglot so you can don't have to use JS if you don't want.
I'm hoping to publish some performance results vs node.js before our 1.0.final release in the next few weeks.
Is EC2 really the right way to do these tests? I would have thought it would introduce unwanted variables related to network congestion and server load caused by other EC2 users.
You might get a better number on bare metal, and you certainly won't get a worse one, so in a sense it's not ideal.
On the other hand, a big chunk of the web is run from AWS. Demonstrating performance there implies performance in many places. It's also an easily reproducible environment, which is helpful for benchmarking.
Fair question. We had some reservations about using EC2 for the reasons you mention. But we wanted to avoid any perceptions that we'd somehow cooked the books, so decided to run the tests on "neutral" gear.
If you look at the report details, you'll see some of the lumpiness your question suggests, particularly related to the network. The Node instances were beginning to starve when database server CPU utilization was still pretty low.
We would certainly have gotten better numbers if we ran these benchmarks on bare metal or a closely-tended cloud infrastructure. But we're a pretty low-BS group, so we'll stand behind what EC2 gave us.
- I assume you did the tests with direct SQL statements without any ORM? How would an ORM affect the test's performance (with an Node.JS-ORM of your choice, e.g. sequelize)?
- How does Node+VoltDB compare to i.e. Node+PostgreSQL or Node+MySQL
- What does in memory SQL exactly mean? How is data made persistent?
VoltDB doesn't really support ad-hoc SQL. It can execute ad-hoc by flipping a flag, but performance is achieved by using stored procedures. You write the sprocs in Java, and their compiler looks at the SQL statements and optimizes.
VoltDB smokes other RDBMS (Postgres, MySQL, MSSQL, etc.) for certain OLTP workloads because it stays purely in-memory, but removes all locking. Normal databases spend a huge amount of time dealing with locks, even if the memory is in RAM. VoltDB was based on the H-Store project; you can read more about the ideas at [1].
Making transactions execute serially[2] adds latency to an individual transaction, but the point with VoltDB is to send tons of transactions so that there's always work queued up. You should be able to get average latency of, say, 10ms for the type of app mentioned here. I'm surprised it's not mentioned in the blog post. (Maybe on EC2, the latency numbers were less-than-ideal, but throughput was fine.)
SQL and ACID have nothing to do with the backing store. Using RAM is just as valid as using disk. Practically, with VoltDB you're going to set the "k factor" - number of replicas you want. This allows you to lose k servers without suffering data loss (put batteries on the servers; that should be pretty good). There's a snapshot backup feature, and the enterprise edition has a disk logger to further minimize the loss window in case of catastrophic failure.
1: http://hstore.cs.brown.edu/publications/
2: More or less. Servers are sync'd via NTP, allowing one to know when it's safe to run a specific transaction. There's also some ways Volt might know when one tx is independent and safe to run before another.
To update what you've said with insider info, ad-hoc SQL performance is currently a big focus at VoltDB. We fully support ad-hoc access in auto-commit mode through our native clients or through JDBC. Currently, we can do hundreds or thousands of statements/sec, and we expect performance to increase dramatically in the next few months.
Typically, applications are not 100% ad-hoc on VoltDB, but we're starting to see more hybrid use cases. Stored procedures can be used for extreme throughput or for transactional requirements, but other parts of the app can be more flexible.
VoltDB does support cross-partition transactions. They're not as fast as single-partition transactions, of course, because you're doing them over a network. But distributed transactions have been a supported feature since 1.0.
Yes, the tests were run without an ORM. Although VoltDB is accessible via JDBC, best throughput is achieved with parameterized SQL embedded in Java stored procedures. That's how the Node.js benchmark app (Voter) was implemented.
Node+Volt will be much faster than Node+Postgres or Node+MySQL for many OLTP-style workloads. So it's a great fit for apps like financial trading, digital advertising, online gaming and network monitoring - each of which requires super-high throughput writes (at single ms latencies) and fast, simple analytics. Node+Postgres or Node+MySQL would be a better fit for general purpose database workloads (e.g., with large-grained reporting).
In-memory means that Volt stores all data in main memory. Durability is achieved through the use of a highly innovative command log, which logs all transaction invocations to persistent storage. You can tune this feature to log synchronously (100% guaranteed durability at reduced transaction latency) or asynchronously (slightly more "lossy", but less of a latency impact). VoltDB also recently released a full database replication feature that supports WAN replication for disaster recovery.
Just do it ten times in a row - I don't know why the normal operation of NTP doesn't achieve the same.
This tight sync seemed to increase throughput by 2%. Because that was not so significant I did not further investigate the consequences of good or bad time syncing.
The time sync that EC2 instances come with out of the box seemed to sometimes be good enough to run multiple VoltDB hosts without any additional syncing, sometimes not, and it seemed to vary quite a bit over time.
So really you must deal with NTP in EC2, but of course you need NOT get the sync to sub-millisecond values.
The intent of the test was to measure throughput. Since the API is asynchronous that means the client was submitting as much work as would fit in the queues at both client and server and any latency measurement would be a measurement of how long it takes to execute all queued requests in the pipeline and the pipeline is always kept full.
I am able to get sub-millisecond offsets (as reported by ntpq -p) with m1.large instances, but I have never left them running long. I am not sure what Henning got when he ran. I pointed him to my blog post on the topic http://www.afewmoreamps.com/2011/07/configuring-ntp-for-volt...
On bare metal ntpdate reports offsets that are sub microsecond.
I find that nodes sync up quickly if only one of them polls the public pool.
"Third is NewSQL, which obtains better performance and scale in the only way possible – with a totally new architecture. NewSQL delivers high performance and scalability while preserving the traditional ACID functions needed for OLTP. NewSQL systems like VoltDB deliver better throughput than NoSQL without requiring transactions to be supported in user code. NewSQL systems also preserve the high-level language capabilities of SQL."
Read the H-Store publications[1], which served as the starting point for Volt.
Basically, by staying in memory and eliminating locking, you get awesome performance. However, why should you have to give up SQL, tables, and transactions? Those things make developing apps easy. So why not get "NoSQL-like" performance, but keeping the good parts of "SQL"?
After looking around I'm still not sure what they mean by 'eliminating locking'. Locking isn't just overhead, it's needed for transactions- how are transactions handled if there is no locking? Does the speed come at the cost of dirty reads and writes?
VoltDB achieves concurrency via scheduling, rather than shared access to data structures.
There are concurrency primitives around networking/messaging and transaction scheduling, so there aren't truly ZERO locks in the system as a whole.
The big difference is that VoltDB has zero system-locks (mutexes,etc) around table data and indexes. It also has no logical locks/latches arounds indexes and rows like other systems. This makes executing SQL very fast. Because it doesn't wait on disk or on mutexes, SQL execution is typically bottlenecked on memory bandwidth.
I think they partition the data then serialize access in the partition. Makes things very fast assuming you only need to modify stuff atomically in one partition.
As far as open source: I was using the .NET client, and fixed a few issues with it. The folks at VoltDB were really responsive about working with me and accepting my patches. They also took time to explain the inner workings, sorta give me a launching point to investigating the core code. Pretty good experience for a commercial open source project.