Dragonfly

A novel memory store that supports Redis and Memcached commands. For more detailed status of what's implemented - see below.

Features include:

1. High throughput reaching millions of QPS on a single node.
2. TLS support.
3. Pipelining mode.
4. A novel cache design, which does not require specifying eviction policies.
5. Memory efficiency that can save 20-40% for regular workloads and even more for cache like workloads

Building from source

I've tested the build on Ubuntu 21.04+. Requires: CMake, Ninja, boost, libunwind8-dev

sudo apt install ninja-build
sudo apt install libunwind-dev
sudo apt-get install libboost-all-dev

git clone --recursive https://github.com/romange/dragonfly
cd dragonfly && ./helio/blaze.sh -release
cd build-opt && ninja dragonfly

Running

./dragonfly --logtostderr

for more options, run ./dragonfly --help

Milestone - Source Available

API 1.0

• String family
  • SET
  • SETNX
  • GET
  • DECR
  • INCR
  • DECRBY
  • GETSET
  • INCRBY
  • MGET
  • MSET
  • MSETNX
  • SUBSTR
• Generic family
  • DEL
  • ECHO
  • EXISTS
  • EXPIRE
  • EXPIREAT
  • KEYS
  • PING
  • RENAME
  • RENAMENX
  • SELECT
  • TTL
  • TYPE
  • SORT
• Server Family
  • AUTH
  • QUIT
  • DBSIZE
  • BGSAVE
  • SAVE
  • DEBUG
  • EXEC
  • FLUSHALL
  • FLUSHDB
  • INFO
  • MULTI
  • SHUTDOWN
  • LASTSAVE
  • SLAVEOF/REPLICAOF
  • SYNC
• Set Family
  • SADD
  • SCARD
  • SDIFF
  • SDIFFSTORE
  • SINTER
  • SINTERSTORE
  • SISMEMBER
  • SMOVE
  • SPOP
  • SRANDMEMBER
  • SREM
  • SMEMBERS
  • SUNION
  • SUNIONSTORE
• List Family
  • LINDEX
  • LLEN
  • LPOP
  • LPUSH
  • LRANGE
  • LREM
  • LSET
  • LTRIM
  • RPOP
  • RPOPLPUSH
  • RPUSH
• SortedSet Family
  • ZADD
  • ZCARD
  • ZINCRBY
  • ZRANGE
  • ZRANGEBYSCORE
  • ZREM
  • ZREMRANGEBYSCORE
  • ZREVRANGE
  • ZSCORE
• Not sure whether these are required for the initial release.
  • BGREWRITEAOF
  • MONITOR
  • RANDOMKEY
  • MOVE

API 2.0

• List Family
  • BLPOP
  • BRPOP
  • BRPOPLPUSH
  • LINSERT
  • LPUSHX
  • RPUSHX
• String Family
  • SETEX
  • APPEND
  • PREPEND (dragonfly specific)
  • BITCOUNT
  • BITFIELD
  • BITOP
  • BITPOS
  • GETBIT
  • GETRANGE
  • INCRBYFLOAT
  • PSETEX
  • SETBIT
  • SETRANGE
  • STRLEN
• HashSet Family
  • HSET
  • HMSET
  • HDEL
  • HEXISTS
  • HGET
  • HMGET
  • HLEN
  • HINCRBY
  • HINCRBYFLOAT
  • HGETALL
  • HKEYS
  • HSETNX
  • HVALS
  • HSCAN
• PubSub family
  • PUBLISH
  • PUBSUB
  • PUBSUB CHANNELS
  • SUBSCRIBE
  • UNSUBSCRIBE
  • PSUBSCRIBE
  • PUNSUBSCRIBE
• Server Family
  • WATCH
  • UNWATCH
  • DISCARD
  • CLIENT LIST/SETNAME
  • CLIENT KILL/UNPAUSE/PAUSE/GETNAME/REPLY/TRACKINGINFO
  • COMMAND
  • COMMAND COUNT
  • COMMAND GETKEYS/INFO
  • CONFIG GET/REWRITE/SET/RESETSTAT
  • MIGRATE
  • ROLE
  • SLOWLOG
  • PSYNC
  • TIME
  • LATENCY...
• Generic Family
  • SCAN
  • PEXPIREAT
  • PEXPIRE
  • DUMP
  • EVAL
  • EVALSHA
  • OBJECT
  • PERSIST
  • PTTL
  • RESTORE
  • SCRIPT LOAD/EXISTS
  • SCRIPT DEBUG/KILL/FLUSH
• Set Family
  • SSCAN
• Sorted Set Family
  • ZCOUNT
  • ZINTERSTORE
  • ZLEXCOUNT
  • ZRANGEBYLEX
  • ZRANK
  • ZREMRANGEBYLEX
  • ZREMRANGEBYRANK
  • ZREVRANGEBYSCORE
  • ZREVRANK
  • ZUNIONSTORE
  • ZSCAN
• HYPERLOGLOG Family
  • PFADD
  • PFCOUNT
  • PFMERGE

In addition, we want to support efficient expiry (TTL) and cache eviction algorithms. We should implement basic memory management support. For Master/Slave replication we should design a distributed log format.

Memchache API

• set
• get
• replace
• add
• stats (partial)
• append
• prepend
• delete
• flush_all
• incr
• decr
• version
• quit

Commands that I prefer avoid implementing before launch:

• PUNSUBSCRIBE
• PSUBSCRIBE
• HYPERLOGLOG
• SCRIPT DEBUG
• OBJECT
• DUMP/RESTORE
• CLIENT

Also, I would omit keyspace notifications. For that I would like to deep dive and learn exact use-cases for this API.

Random commands we implemented as decorators along the way

• ROLE (2.8) decorator for for master withour replicas
• UNLINK (4.0) decorator for DEL command
• BGSAVE
• FUNCTION FLUSH

Milestone Stability

APIs 3,4,5 without cluster support, without modules, without memory introspection commands. Without geo commands and without support for keyspace notifications, without streams. Design config support. ~10-20 commands overall... Probably implement cluster-API decorators to allow cluster-configured clients to connect to a single instance.

• HSTRLEN

Design decisions along the way

Expiration deadlines with relative accuracy

Expiration ranges are limited to ~4 years. Moreover, expiration deadlines with millisecond precision (PEXPIRE/PSETEX etc) will be rounded to closest second for deadlines greater than 134217727ms (approximately 37 hours). Such rounding has less than 0.001% error which I hope is acceptable for large ranges. If it breaks your use-cases - talk to me or open an issue and explain your case.

For more detailed differences between this and Redis implementations see here.