Bump default JVM heap and add OOM diagnostics#2044
Conversation
|
@cshannon I propose to update the default configuration like this. |
|
Hmm, I think this might need more discussion. I think the 2GB default is good but I don't know about the OOM heap dump as that will cause unexpected output to a file system for users. I think that needs to be configurable on set up maybe. We could bring it up for discussion on the dev list. |
|
@cshannon yes, I can add a optional flag to setup OOM. I'm just sharing my experience: I have to ask more of the ActiveMQ users to enable it it to investigate 😄 So having a way to enable it by default would be good (or at least documented). |
That sounds good, I personally set that flag myself so I am fine with it but I figure someone might not want the extra output. |
|
+1 to have a preconfigured but commented out set of options. Having it on by default can quickly lead to disk space usage issues -- there is also a 'where to write the files' problem |
|
@jbonofre - I think the memory updates make sense and I agree with those. For the OOM settings to dump to disk I think we talked about making that optional with a flag to turn on. I don't think it should be on by default because of the unexpected disk usage. Some people might configure large heap sizes. Also, I would recommend adding compression: |
|
@cshannon thanks! It makes sense. Let me update the PR accordingly. |
Raise the shipped default heap from -Xms64M/-Xmx1G to -Xms512M/-Xmx2G which is more representative of a production broker workload while still leaving headroom on small hosts (override via ACTIVEMQ_OPTS_MEMORY). Add ACTIVEMQ_OOM_OPTS with -XX:+ExitOnOutOfMemoryError so that an OutOfMemoryError terminates the JVM instead of leaving the broker running in a degraded state. Heap dump on OOM is disabled by default to avoid unexpected disk usage when large heap sizes are configured. Users can opt in by uncommenting the ACTIVEMQ_OOM_DUMP_OPTS lines in bin/setenv (or bin/activemq for the no-setenv fallback). The commented template includes -XX:HeapDumpGzipLevel=6 (JDK 17+) to compress the dump.
jbonofre
force-pushed
the
jvm-defaults-oom-diagnostics
branch
from
7155e2c to
9d18836
Compare
|
@cshannon Thanks for the feedback, I've updated the PR:
|
Per mattrpav's review feedback: setting -Xms512M by default would be misconstrued as a high baseline memory requirement. Keep the initial heap at the original 64M so the JVM starts small and grows on demand, while still raising the ceiling to 2G.
3 participants
-Xms64M -Xmx1Gto-Xms512M -Xmx2Ginassembly/src/release/bin/setenv. The previous default is undersized for a production broker; the new value is still modest and remains overridable viaACTIVEMQ_OPTS_MEMORY.ACTIVEMQ_OOM_OPTSvariable defaulting to-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$ACTIVEMQ_DATA -XX:+ExitOnOutOfMemoryErrorso that anOutOfMemoryErrorproduces a heap dump under the broker data directory and the JVM terminates rather than continuing in a degraded state.ACTIVEMQ_OOM_OPTSfallback insideassembly/src/release/bin/activemqso the diagnostics still apply ifsetenvis absent.Both new variables remain overridable from the environment or from any of the higher-priority configuration files (
/etc/default/activemq,~/.activemqrc,bin/setenv).