Config footgun: small LRU MaxInuse with default partitions can exhaust memory

[jrhy]

Summary

Hi, and first: thank you for Olric. I appreciate that it directly addresses problems where process-local caches and standalone cache services can both be awkward, especially around cold starts, per-process capacity, and not wanting to run another service. While experimenting with it, I accidentally found a configuration footgun that can make allocated memory grow far beyond DMaps.MaxInuse, even though inuse is correctly capped by LRU eviction.

The short version is: with the default PartitionCount=271 and default ramblock tableSize=1MiB, setting DMaps.MaxInuse below roughly 271MiB gives each partition less than one table's worth of in-use capacity:

DMaps.MaxInuse / PartitionCount < tableSize

A "just trying it out" value like MaxInuse=256MiB therefore lands in the bad zone.

I originally hit this while trying a relatively small MaxInuse value to get a feel for Olric's cache behavior under a limited sample workload. In hindsight, 256MiB is just below the default-config boundary: 271 partitions x 1MiB table size. So instead of only making the cache smaller, that value also put each partition's effective cap below one storage table.

Here are some steps to reproduce the problem, and a diagnosis, in case it's useful for steering future users away from the same problem. I'd be happy to contribute a small validation PR if you agree that's a reasonable improvement.

What I Observed

With:

DMaps.MaxInuse = 256MiB
PartitionCount = 271
EvictionPolicy = LRU
ramblock tableSize = 1MiB

the per-partition cap is:

256MiB / 271 ~= 967KiB

That is smaller than the default 1MiB ramblock table size.

In the environment where I first noticed this, the growth happened gradually over many hours under ordinary cache churn. The focused reproduction below drives writes much harder so the same behavior is visible in seconds.

In one 5-second local run, after about 2.4M Put operations, I saw:

Configuration
  MaxInuse:         256MiB
  PartitionCount:   271
  Per-partition cap: 967.3KiB
  Payload size:     512B
  Duration:         5s

Final
  Put operations:   2390446
  Entries retained: 484548
  Allocated:        1.1GiB
  Inuse:            256.0MiB
  Tables:           1084
  Allocated/Inuse:  4.23x

So inuse was capped at 256MiB, but allocated slab memory reached 1.1GiB after about 2.4M writes in this focused test.

Why I Think This Happens

From reading the code, MaxInuse is a per-node/member cap, then LRU enforcement divides it across the node's owned partitions.

Relevant code/comments:

• config/dmaps.go: MaxInuse is documented as maximum in-use memory "on a particular node"
• internal/dmap/put.go: LRU compares partition Inuse to maxInuse / ownedPartitionCount
• internal/ramblock/ramblock.go: default tableSize is 1 << 20, i.e. 1MiB

When the per-partition cap is smaller than tableSize, eviction can happen before a table fills. The storage engine can then accumulate many partially-used slabs: inuse looks healthy, but allocated grows much larger.

Proposed Fix: Reject This Configuration

The simplest starting point might be to reject configs where LRU MaxInuse is enabled and the effective per-partition cap is smaller than the storage engine table size:

DMaps.MaxInuse > 0
EvictionPolicy == LRU
DMaps.MaxInuse / PartitionCount < tableSize

For the default config, that would reject MaxInuse < 271MiB.

The error could say something like:

invalid DMaps.MaxInuse: per-partition MaxInuse is smaller than ramblock tableSize;
increase MaxInuse, reduce PartitionCount, or configure a smaller tableSize

This is admittedly a bit heavy-handed because it may reject configurations that currently start, but those configurations appear capable of producing surprising memory growth. I think failing fast is preferable to letting users discover the issue through OOM behavior.

Alternatives

If rejecting is too breaking, softer options could be:

• emit a warning during validation/startup
• auto-fit tableSize downward when the user did not explicitly configure it
• reject only when tableSize is still the default

But my instinct is that an explicit validation error is the clearest and safest first fix.

Reproduction

I put together a small standalone reproduction using public Olric APIs:

https://gist.github.com/jrhy/67761b8a442d876064790df2a6f18690

It starts a single local Olric member with the default PartitionCount=271, DMaps.MaxInuse=256MiB, and LRU enabled, then drives write churn and prints aggregate DMap stats from the public Stats API.