How to limit RAM usage

[webfrank]

Hi, I've come to KVRocks from Redis as I need to limit RAM usage on a Job Queue process. The main process will create thousands of keys with some JSON inside, once the job is done it will update the key, not much job and no latency issue, only memory allocation. On redis I reach 16GB of RAM which cannot be handle on the current environment. I would like to lower this figure to 1/2GB mantaining the data and this is why I reached kvrocks.

I exchanged redis for kvrocks and it worked as expected (apart from a corruption error for which I've opened an issue), now I would like to limit ram usage, I set --rocksdb.block_cache_size 128 on command line but the ram is increasing. Which flags I need to set to limit it? (128 is just a test to limit with very low usage). Now I'm using 2.10.1 as per the corruption issue with 2.11.0

[PragmaTwice]

You can refer to here: #2726

[webfrank]

Hi, I've followed the post but it seems it is not working, I setup 128MB of cache and ram went up to 480MB. I made a second test limiting docker ram to 256MB and it stopped there so it seems it uses RAM up to the limit.

[PragmaTwice]

Could you explain a bit about "128MB cache" here? Is it a sum of block cache and write buffer?

[webfrank]

I tried to set --rocksdb.block_cache_size 128 but without limiting docker image doesn't not limi ram usage.

The only way I found to limit memory usage is to limit container memory and nothing else.

[git-hulk]

@webfrank You could reduce the following configurations as well:

• rocksdb.write_buffer_size default value is 64MiB.
• rocksdb.max_write_buffer_number default value is 4.

You could reduce the write_buffer_size to 16MiB and max_write_buffer_numbner to 1 if you write traffic is low.

[git-hulk]

@webfrank Does this work well for you after reducing those two parameters?

[webfrank]

Hi, the ram reduced but not so much as expected, it continues to increase up to container limits.

Now I'm trying with 256MB block cache and 4x16MB write cache and a container limit of 512MB and after several batches seems stable around 480MB

[rizkymma]

Hi, I have a similar question. How can I limit the RAM usage without impacting the write buffer? I have the following configurations:

kvrocks.conf.txt

TLDR:

root@rmas1:~# grep block_cache_size kvrocks.conf 
rocksdb.block_cache_size 256
rocksdb.metadata_block_cache_size 128
rocksdb.subkey_block_cache_size 128
root@rmas1:~# grep write_buffer kvrocks.conf 
rocksdb.write_buffer_size 64
rocksdb.max_write_buffer_number 4

I'm using kvrocks version 2.11.1

I have cascading replicas with a similar configuration. The topology is [master ---> cascading_replica ---> replica]

During a high volume of writes to the master, the memory usage increases in all three Kvrocks instances.

I loaded the master with a sizable database (~150MB) (~2mil keys) before doing the heavy writes.

Writes to master:

stuff=$(head -c 32000000 /dev/urandom | base64)
while true; do redis-cli -p 6666 -a $PASS set $(date --iso=ns) "${stuff:RANDOM:102400}"; sleep 0.001; done

Memory usage on cascading replica (not reported in rss_bytes metrics):

root@rmas2:~# systemctl status kvrocks --no-pager | grep Memo
     Memory: 1.9G (high: 2.0G max: 2.0G available: 508.0K)
root@rmas2:~# curl -Ss localhost:9121/metrics | grep rss
# HELP kvrocks_memory_used_rss_bytes memory_used_rss_bytes metric
# TYPE kvrocks_memory_used_rss_bytes gauge
kvrocks_memory_used_rss_bytes 5.03590912e+08

On master:

root@rmas1:~# systemctl status kvrocks | grep Memo
     Memory: 2.0G
root@rmas1:~# curl -Ss localhost:9121/metrics | grep rss
# HELP kvrocks_memory_used_rss_bytes memory_used_rss_bytes metric
# TYPE kvrocks_memory_used_rss_bytes gauge
kvrocks_memory_used_rss_bytes 5.19061504e+08

On replica:

root@rmas3:~# systemctl status kvrocks --no-pager | grep Memo
     Memory: 1.1G
root@rmas3:~# curl -Ss localhost:9121/metrics | grep rss
# HELP kvrocks_memory_used_rss_bytes memory_used_rss_bytes metric
# TYPE kvrocks_memory_used_rss_bytes gauge
kvrocks_memory_used_rss_bytes 1.55181056e+08

[hosting2000me]

This is a very big problem that prevents kvrocks from being used in production tasks. It consumes all available memory. If you limit it in a container, the container simply dies on resource-intensive tasks. This is really sad.

[git-hulk]

@rizkymma @hosting2000me We have multiple column families in Kvrocks, so you could reduce the write buffer number if you would like to control the memory usage:

• rocksdb.write_buffer_size 64
• rocksdb.max_write_buffer_number 4

You could reduce rocksdb.max_write_buffer_number to 2 and rocksdb.write_buffer_size to 32 is your value size is small.

[hosting2000me]

• rocksdb.write_buffer_size 64
• rocksdb.max_write_buffer_number 4

These are my current settings. This doesn't prevent the container from consuming 20GB(for my case) of RAM if it's not limited in the container. So, kvrocks caches all database accesses, if possible.

[git-hulk]

@hosting2000me @rizkymma @webfrank Could you please run with Kvrocks with the environment variable:

MALLOC_CONF=retain:false

Apache James also encounters similar issues when integrating with Kvrocks, and they report it might be related to Jemalloc's retain policy. For more information please refer to https://lists.apache.org/thread/xzn62z7xz3bvxwgsd9toh6wq2g2j8zng.