any benchmark against redis?

[hiqsociety]

any benchmark against redis?

[git-hulk]

It makes no sense to compare the Kvrocks with Redis directly since the storage and thread model was totally different. We have a simple performance benchmark on README: https://github.com/KvrocksLabs/kvrocks#performance, from the throughput side, the Kvrocks would be better than Redis in many commands.

[hiqsociety]

how much slower do u think kvrocks will be compared with redis?

[git-hulk]

It depends on the disk type, the latency of most commands on NVMe should be as good as Redis, SATA disks would much worse but we didn’t have any performance on it.

[hengsok]

@hiqsociety I ran a simple test using redis-benchmark tool, I can say that kvrocks perform very well as below using DigitalOcean. Eventhough it persists data to disk, it can be almost as fast as Redis/KeyDB. I reckon if we give it more cpu cores and ram, it would perform even better. Very good job to kvrocks team @git-hulk @ShooterIT. We are awaiting their cluster mode feature and hope they can make their cluster mode to work like Redis whereby it can automatically handle failover and high availability!

Single Node - General Purpose / 16 GB / 4 vCPUs/ NVMe instance
kvrocks

PING_INLINE: 88339.23 requests per second
PING_BULK: 92678.41 requests per second
SET: 58513.75 requests per second
GET: 96525.09 requests per second
INCR: 50968.40 requests per second
LPUSH: 46728.97 requests per second
RPUSH: 46970.41 requests per second
LPOP: 43308.79 requests per second
RPOP: 42973.79 requests per second
SADD: 53390.28 requests per second
HSET: 39261.88 requests per second
SPOP: 63211.12 requests per second
LPUSH (needed to benchmark LRANGE): 45187.53 requests per second
LRANGE_100 (first 100 elements): 44742.73 requests per second
LRANGE_300 (first 300 elements): 21542.44 requests per second
LRANGE_500 (first 450 elements): 15434.48 requests per second
LRANGE_600 (first 600 elements): 12137.39 requests per second
MSET (10 keys): 16835.02 requests per second

I didn't get to save the results for above machine spec for KeyDB which is a forked of Redis but run on multithreads.

4C (Shared), 8GB, NVMe, 10 worker threads
kvrocks

PING_INLINE: 59630.29 requests per second
PING_BULK: 63131.31 requests per second
SET: 34352.46 requests per second
GET: 56785.91 requests per second
INCR: 30156.82 requests per second
LPUSH: 26652.45 requests per second
RPUSH: 24527.84 requests per second
LPOP: 24166.26 requests per second
RPOP: 23424.69 requests per second
SADD: 22644.93 requests per second
HSET: 21720.24 requests per second
SPOP: 55187.64 requests per second
LPUSH (needed to benchmark LRANGE): 24764.73 requests per second
LRANGE_100 (first 100 elements): 24218.94 requests per second
LRANGE_300 (first 300 elements): 11956.00 requests per second
LRANGE_500 (first 450 elements): 8792.75 requests per second
LRANGE_600 (first 600 elements): 6853.07 requests per second
MSET (10 keys): 10051.26 requests per second

4C (Shared), 8GB, NVMe, 10 worker threads
KeyDB (purely memory based)

PING_INLINE: 57971.02 requests per second
PING_BULK: 61236.99 requests per second
SET: 59066.75 requests per second
GET: 56689.34 requests per second
INCR: 50301.81 requests per second
LPUSH: 52631.58 requests per second
RPUSH: 60679.61 requests per second
LPOP: 49578.58 requests per second
RPOP: 54525.63 requests per second
SADD: 58616.65 requests per second
HSET: 54495.91 requests per second
SPOP: 54555.38 requests per second
LPUSH (needed to benchmark LRANGE): 51975.05 requests per second
LRANGE_100 (first 100 elements): 35587.19 requests per second
LRANGE_300 (first 300 elements): 17590.15 requests per second
LRANGE_500 (first 450 elements): 12269.94 requests per second
LRANGE_600 (first 600 elements): 9795.28 requests per second
MSET (10 keys): 54614.96 requests per second

[git-hulk]

Many thanks for @hengsok benchmark data, those were benched without the -r option? which would use random keys instead of a single one. I think kvrocks would perform better with random keys since we have to lock the key when writing to prevent data race.

[ShooterIT]

And you should enable multi threads in redis-benchmark, such as --threads 8

[hengsok]

I have tried again following @git-hulk suggestions with the -r options using length of 8 and 16 for comparisons. I also experiment with changing the number of workers setting in the kvrocks.conf file.

I have noticed that during the SET test, below is the CPU and Memory stats from TOP. It seems kvrocks does not fully utilise the 32GB memory I provisioned. Strange. Is there any configuration to make sure kvrocks maximises memory usage?
CPU also only hover around 20-25% range.

I have tried your suggestions @ShooterIT but the current build of redis-benchmark no longer support --threads params.
We got the error
Invalid option "--threads" or option argument missing

I believe by specifying the -c which is the number of clients, redis-benchmark is already parallelising the requests up to 500 clients as we have specified in the test.

General Purpose / 32 GB / 8 vCPUs / NVMe
%Cpu(s): 20.2 us, 21.1 sy, 0.0 ni, 54.5 id, 0.0 wa, 0.0 hi, 4.1 si, 0.0 st
MiB Mem : 32116.9 total, 26104.4 free, 1001.6 used, 5010.9 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 30663.2 avail Mem

kvrocks (8 workers)
redis-benchmark -p 6666 -d 50 -c 500 -n 1000000 -r 16 -q
PING_INLINE: 75448.92 requests per second
PING_BULK: 78733.95 requests per second
SET: 62247.12 requests per second
GET: 81853.16 requests per second
INCR: 62664.49 requests per second
LPUSH: 48260.22 requests per second
RPUSH: 47114.25 requests per second
LPOP: 38842.49 requests per second
RPOP: 37030.18 requests per second
SADD: 49091.80 requests per second
HSET: 55285.27 requests per second
SPOP: 71525.64 requests per second
LPUSH (needed to benchmark LRANGE): 45203.87 requests per second
LRANGE_100 (first 100 elements): 41012.18 requests per second
LRANGE_300 (first 300 elements): 16218.23 requests per second
LRANGE_500 (first 450 elements): 12822.49 requests per second
LRANGE_600 (first 600 elements): 8941.90 requests per second
MSET (10 keys): 9546.72 requests per second

General Purpose / 32 GB / 8 vCPUs / NVMe
kvrocks (16 workers)
redis-benchmark -p 6666 -d 50 -c 500 -n 1000000 -r 16 -q
PING_INLINE: 77845.24 requests per second
PING_BULK: 80121.79 requests per second
SET: 59333.09 requests per second
GET: 74693.76 requests per second
INCR: 58661.34 requests per second
LPUSH: 45543.56 requests per second
RPUSH: 44507.74 requests per second
LPOP: 36072.43 requests per second
RPOP: 37378.98 requests per second
SADD: 48887.80 requests per second
HSET: 58654.47 requests per second
SPOP: 74521.20 requests per second
LPUSH (needed to benchmark LRANGE): 44004.40 requests per second
LRANGE_100 (first 100 elements): 39089.98 requests per second
LRANGE_300 (first 300 elements): 15401.44 requests per second
LRANGE_500 (first 450 elements): 12331.67 requests per second
LRANGE_600 (first 600 elements): 8924.75 requests per second
MSET (10 keys): 9741.18 requests per second

General Purpose / 32 GB / 8 vCPUs / NVMe
kvrocks (8 workers)
redis-benchmark -p 6666 -d 50 -c 500 -n 1000000 -r 8 -q
PING_INLINE: 79936.05 requests per second
PING_BULK: 79182.84 requests per second
SET: 62065.54 requests per second
GET: 73844.34 requests per second
INCR: 61087.35 requests per second
LPUSH: 43230.16 requests per second
RPUSH: 45491.77 requests per second
LPOP: 36296.32 requests per second
RPOP: 37767.20 requests per second
SADD: 48236.94 requests per second
HSET: 58775.12 requests per second
SPOP: 69189.79 requests per second
LPUSH (needed to benchmark LRANGE): 44255.62 requests per second
LRANGE_100 (first 100 elements): 38172.31 requests per second
LRANGE_300 (first 300 elements): 15107.57 requests per second
LRANGE_500 (first 450 elements): 12413.26 requests per second
LRANGE_600 (first 600 elements): 8837.20 requests per second
MSET (10 keys): 9850.18 requests per second

[ShooterIT]

hi @hengsok To use --threads in benchmark, you should use latest redis-benchmark, unstable branch of redis https://github.com/redis/redis. And -r is specified range, you should use a bigger number.
such as redis-benchmark -p 6666 -d 50 -c 500 -n 1000000 -r 10000000 -q --threads 8

[hengsok]

@ShooterIT Yes, I agree that the clients request did not max out. Just as a quick workaround, we tried to open 5 terminals and run redis-benchmark and we were getting more than 200K+ rps for GET and more than 101.5K+ rps for SET by adding the stats up. It's an impressive numbers for just 8 vCPUs.

Do you know why RAM utilisation is so low? We noticed during the running of the benchmark, RAM usage only reach as much as 4GB. Shouldn't kvrocks cache more in Memory or it mainly relies on reading and writing directly with Rocksdb without further memory caching optimization?

Finally, we really love this project a lot but there is a lack of documentations for how to setup kvrocks as Cluster.

I have read about PIKA + Codis, does kvrocks support Codis cluster? Thanks!

[ShooterIT]

For memory usage, please see https://github.com/KvrocksLabs/kvrocks/blob/unstable/kvrocks.conf#L324

Currently, cluster solution is experimental, we are developing a meta server to manage cluster, coming soon, if you want to have a try, please #302

Kvrocks support Codis cluster, but doesn't support to scale i.e. can't migrate slot date.

[hengsok]

@ShooterIT thanks for the tips.

Can't wait for your full cluster solution. I have tried the try_cluster. Does your implementation support dynamic addition of more nodes? If it supports, then how do we reassign slots? It seems your current implementation is still very manual based and I am not sure how the nodes are gonna handle moving of data from one node to the next to rebalance upon adding more nodes

[ShooterIT]

Coming soon!