I spent today evaluating what kind of back-end data storage system to use for my little startup.
(Oh yeah, I am starting a company, did I forget to mention that? It's gonna be awesome. Details later.)
As many of you may know, I detest RDBMS databases, and have since University. Every time I have tried to use one, it has sucked, possibly with the exception of sqlite. Luckily the data I am working with in this project is a terrible fit for an RDBMS, so I was free to go exploring the open-source NoSQL universe with a clear conscience. Whee!
There are some pretty cool projects out there.
I found some nice Bigtable work-alikes (Cassandra, Hypertable), which tempted me a bit after my experience with Bigtable at Google. But I don't need all their features, the complexity and requirements would probably be more of a hindrance than a help to my project. I was tempted a bit more by Voldemort, which is a fair bit simpler, as it is a plain key/value store, not an ordered table. But still, it looked too complicated in some areas, and not mature enough in others.
Finally, I discovered redis: an in-memory, disk-backed hash table which natively supports the most important data types (ints, floats, lists, sets) and allows high-level atomic operations. It also knows how to do replication, a requirement for my project, and the implementation is simple enough that I should be able to promote a slave to master status in an emergency with little difficulty. The 2.0 release (already in beta) adds support for pub/sub and larger-than-RAM data-sets (swapping out unpopular entries).
In short, it looks totally awesome!
It was really weird to read about the features of redis, though - I had this ridiculous feeling of deja-vu.
It does the same things, in the same way: it is a single threaded, in-memory hash server. It has a human-readable client/server protocol, running over TCP. Every now and then it forks out a new process to write a checkpoint of the current DB state to disk. It can track incremental changes using an operation log. The up-coming 2.0 version even has pub/sub features (BottleNeck has "watches").
Freaky. But of course, redis is in almost every way superior to my naive Perl implementation. It is faster. It is properly tested. It has way more high-level operations, natively supports complex data-structures like lists and sets, and it supports replication. Version 2.0 even supports larger-than-RAM data-sets.
It's like I got to design the thing and set requirements, and then a much better programmer went and implemented it all (and more) while I wasn't looking.
I can't wait to start using it!
Comments are closed.