innov
/
dragonfly
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
dragonfly/README.md

5.8 KiB
Raw Blame History

Dragonfly

ci-tests

A novel memory store that supports Redis and Memcached commands. For more detailed status of what's implemented - see below.

Features include:

  1. High throughput reaching millions of QPS on a single node.
  2. TLS support.
  3. Pipelining mode.
  4. A novel cache design, which does not require specifying eviction policies.
  5. Memory efficiency that can save 20-40% for regular workloads and even more for cache like workloads

Building from source

I've tested the build on Ubuntu 21.04+. Requires: CMake, Ninja, boost, libunwind8-dev

sudo apt install ninja-build
sudo apt install libunwind-dev
sudo apt-get install libboost-all-dev

git clone --recursive https://github.com/romange/dragonfly
cd dragonfly && ./helio/blaze.sh -release
cd build-opt && ninja dragonfly

Running

./dragonfly --logtostderr

for more options, run ./dragonfly --help

Milestone - Source Available

API 1.0

  • String family
    • SET
    • SETNX
    • GET
    • DECR
    • INCR
    • DECRBY
    • GETSET
    • INCRBY
    • MGET
    • MSET
    • MSETNX
    • SUBSTR
  • Generic family
    • DEL
    • ECHO
    • EXISTS
    • EXPIRE
    • EXPIREAT
    • KEYS
    • PING
    • RENAME
    • RENAMENX
    • SELECT
    • TTL
    • TYPE
    • SORT
  • Server Family
    • AUTH
    • QUIT
    • DBSIZE
    • BGSAVE
    • SAVE
    • DEBUG
    • EXEC
    • FLUSHALL
    • FLUSHDB
    • INFO
    • MULTI
    • SHUTDOWN
    • LASTSAVE
    • SLAVEOF/REPLICAOF
    • SYNC
  • Set Family
    • SADD
    • SCARD
    • SDIFF
    • SDIFFSTORE
    • SINTER
    • SINTERSTORE
    • SISMEMBER
    • SMOVE
    • SPOP
    • SRANDMEMBER
    • SREM
    • SMEMBERS
    • SUNION
    • SUNIONSTORE
  • List Family
    • LINDEX
    • LLEN
    • LPOP
    • LPUSH
    • LRANGE
    • LREM
    • LSET
    • LTRIM
    • RPOP
    • RPOPLPUSH
    • RPUSH
  • SortedSet Family
    • ZADD
    • ZCARD
    • ZINCRBY
    • ZRANGE
    • ZRANGEBYSCORE
    • ZREM
    • ZREMRANGEBYSCORE
    • ZREVRANGE
    • ZSCORE
  • Not sure whether these are required for the initial release.
    • BGREWRITEAOF
    • MONITOR
    • RANDOMKEY
    • MOVE

API 2.0

  • List Family
    • BLPOP
    • BRPOP
    • BRPOPLPUSH
    • LINSERT
    • LPUSHX
    • RPUSHX
  • String Family
    • SETEX
    • APPEND
    • PREPEND (dragonfly specific)
    • BITCOUNT
    • BITFIELD
    • BITOP
    • BITPOS
    • GETBIT
    • GETRANGE
    • INCRBYFLOAT
    • PSETEX
    • SETBIT
    • SETRANGE
    • STRLEN
  • HashSet Family
    • HSET
    • HMSET
    • HDEL
    • HEXISTS
    • HGET
    • HMGET
    • HLEN
    • HINCRBY
    • HINCRBYFLOAT
    • HGETALL
    • HKEYS
    • HSETNX
    • HVALS
    • HSCAN
  • PubSub family
    • PUBLISH
    • PUBSUB
    • PUBSUB CHANNELS
    • SUBSCRIBE
    • UNSUBSCRIBE
    • PSUBSCRIBE
    • PUNSUBSCRIBE
  • Server Family
    • WATCH
    • UNWATCH
    • DISCARD
    • CLIENT KILL/LIST/UNPAUSE/PAUSE/GETNAME/SETNAME/REPLY/TRACKINGINFO
    • COMMAND
    • COMMAND COUNT
    • COMMAND GETKEYS/INFO
    • CONFIG GET/REWRITE/SET/RESETSTAT
    • MIGRATE
    • ROLE
    • SLOWLOG
    • PSYNC
    • TIME
    • LATENCY...
  • Generic Family
    • SCAN
    • PEXPIREAT
    • PEXPIRE
    • DUMP
    • EVAL
    • EVALSHA
    • OBJECT
    • PERSIST
    • PTTL
    • RESTORE
    • SCRIPT LOAD
    • SCRIPT DEBUG/KILL/FLUSH/EXISTS
  • Set Family
    • SSCAN
  • Sorted Set Family
    • ZCOUNT
    • ZINTERSTORE
    • ZLEXCOUNT
    • ZRANGEBYLEX
    • ZRANK
    • ZREMRANGEBYLEX
    • ZREMRANGEBYRANK
    • ZREVRANGEBYSCORE
    • ZREVRANK
    • ZUNIONSTORE
    • ZSCAN
  • HYPERLOGLOG Family
    • PFADD
    • PFCOUNT
    • PFMERGE

In addition, we want to support efficient expiry (TTL) and cache eviction algorithms. We should implement basic memory management support. For Master/Slave replication we should design a distributed log format.

Memchache API

  • set
  • get
  • replace
  • add
  • stats (partial)
  • append
  • prepend
  • delete
  • flush_all
  • incr
  • decr
  • version
  • quit

Commands that I prefer avoid implementing before launch:

  • PUNSUBSCRIBE
  • PSUBSCRIBE
  • HYPERLOGLOG
  • SCRIPT DEBUG
  • OBJECT
  • DUMP/RESTORE
  • CLIENT

Also, I would omit keyspace notifications. For that I would like to deep dive and learn exact use-cases for this API.

Random commands we implemented as decorators along the way

  • ROLE (2.8) decorator for for master withour replicas
  • UNLINK (4.0) decorator for DEL command
  • BGSAVE
  • FUNCTION FLUSH

Milestone Stability

APIs 3,4,5 without cluster support, without modules, without memory introspection commands. Without geo commands and without support for keyspace notifications, without streams. Design config support. ~10-20 commands overall... Probably implement cluster-API decorators to allow cluster-configured clients to connect to a single instance.

  • HSTRLEN

Design decisions along the way

Expiration deadlines with relative accuracy

Expiration ranges are limited to ~4 years. Moreover, expiration deadlines with millisecond precision (PEXPIRE/PSETEX etc) will be rounded to closest second for deadlines greater than 134217727ms (approximately 37 hours). Such rounding has less than 0.001% error which I hope is acceptable for large ranges. If it breaks your use-cases - talk to me or open an issue and explain your case.

For more detailed differences between this and Redis implementations see here.