This. Note that PHK's comments have proven entirely correct, and redis had to implement at least a limited form of multithreading.
I would be very careful about Antirez's older advice on this topic area. At one point he was designing his own "VM" algorithm that would work at 256 byte granularity, an idea that was never viable.
I believe that what happened in the latest 20 years actually shows that multiplexing, for very short-lived requests, won: multithreading is, very often, only used at this point only to scale the single multiplexed approach to multiple cores. Now on disk databases still use one thread per query models, but hardly anything that serves small objects uses the operating system paging as a cache, including on-disk stores that have their own paging mechanism and never trust the OS to do so.
The 256 byte thing was an old story, pre-memcached, that I recounted at some point, that has nothing to do with Redis. I implemented an on-disk cache for slow SQL queries: since the filesystem had issues with inodes, I created a 256 directories, each nested with 256 (so 256*256) in order to lower the number of files for every directory. This used to play a lot better with filesystems of 20 years ago.
Btw here the main point is: nobody is using, at this point, the OS paging as a caching layer in databases, and that was my point back then, and I don't see how it is not true. And, to serve small objects, multiplexing won and threading serves as a way to scale multiplexing itself (see memcached, for instance).
IMHO it is more interesting to provide arguments than to attack single persons you may not like :)
It's not only the "VM" experiments but generally the very basic approach to persistence that's kept Redis from being an actual database and has most users (hopefully) only using it as a cache.
Around the same period as this blog post antirez had also announced that he had written a B-Tree that could be potentially used to store Redis data. The code is still up: https://github.com/antirez/otree – needless to say serious storage engines like LevelDB or RocksDB are more than a single 1,000-line C file, while this looked more like a weekend project.
The persistence model that has stayed in Redis is also a pretty basic approach: fork + dump all memory to a single file. Write every command to a file and compact it by doing the same fork + scan/flush. None of this is efficient, and with a tens of GB of cache you get long pauses during fork() as the page table gets copied to the child process (yes just the table, not the memory pages themselves).
All this to say that his advice on disk I/O is to be taken in the context of his own experience in this domain.
Yeah, I completely agree. Redis makes a great cache with richer operations, but less efficiency than memcached. Using it as a system of record in any context will come back to haunt you in the worst way.
I first ran into the more minimal append only log persistence pattern with the jvm project Prevayler. It's not a totally unreasonable idea but has to be implemented carefully.
The fork as copy on write mechanism initially seems clever, but in practice is extremely brittle. There's a reason classic databases have a lot of sophisticated logic around checkpointing and truncating the write ahead log. It's not a problem that can be oversimplified.
I wouldn't be totally down on minimal btree implementations, but 1000 LOC is pushing it. Even LMDB as minimalist as it is, is around 10k as I recall.
I’ve noticed this sort of thing from Antirez before and it makes his comments seem dubious unless they come from his specific specialization.
He made some claims about Lua here some time ago, and if you read his code, it was clear he didn’t know how to use its C API and was criticizing it out of something he could have just read the manual over.
Software is hard, though, and there’s always someone who is more right, so, oh well.
Maybe a good strategy is to openly ask others if there are better solutions the reader knows of and to share them.
Yeah, I don't want to be overly negative, but Not Invented Here syndrome is a recurring theme with his work. Unfortunately that's quite prevalent among software developers as a whole. It's somewhat understandable: it's easier to just run with your imagination and start hacking something together than dig through research papers, textbooks, and blog posts.
My claim about the Lua C API was that, being a stack-based API, it is hard to use. It's a tradeoff between simplicity and usability from C. Please show me the code I wrote that shows how my ideas about lack of usability of the Lua C interface is due to me not reading the documentation.
Note: interpreters are not entirely out of my specialization, I wrote a Tcl interpreter that is still in use in embedded systems, participated in the Tcl community for a long time, reimplemented the Joy concatenative language extending it, and so forth.
