kvrocks always down,who can help

[jiujiu994]

Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: E20240828 05:40:40.021790 2384433 signal_util.h:34] Ooops! Apache Kvrocks version 2.9.0 got signal: Segmentation fault (11)
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: E20240828 05:40:40.026152 2384433 signal_util.h:36] Stack trace (most recent call first):
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #0  0x0000000000513989 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #1  0x00007f3081b87cef at /lib64/libpthread.so.0
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #2  0x0000000000e1f1b4 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #3  0x00000000006a4ce6 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #4  0x0000000000540e63 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #5  0x000000000065b767 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #6  0x000000000065dc96 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #7  0x00000000005377cb at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #8  0x0000000000c754ad at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #9  0x0000000000c7b77b at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #10 0x0000000000c7c12e at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #11 0x0000000000672509 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #12 0x0000000000e7fe33 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #13 0x00007f3081b7d1c9 at /lib64/libpthread.so.0
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #14 0x00007f3081263e72 at /lib64/libc.so.6
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: E20240828 05:40:40.026293 2384433 signal_util.h:37] It would be greatly appreciated if you could submit this crash to https://github.com/apache/kvrocks/issues along with the stacktrace above, logs and any relevant information.
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal systemd[1]: kvrocks.service: Main process exited, code=dumped, status=11/SEGV
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal systemd[1]: kvrocks.service: Failed with result 'core-dump'.
Aug 28 05:40:45 ip-10-151-101-141.ap-east-1.compute.internal systemd[1]: kvrocks.service: Service RestartSec=5s expired, scheduling restart.
Aug 28 05:40:45 ip-10-151-101-141.ap-east-1.compute.internal systemd[1]: kvrocks.service: Scheduled restart job, restart counter is at 62.
Aug 28 05:40:45 ip-10-151-101-141.ap-east-1.compute.internal systemd[1]: Stopped kvrocks.
---
# /usr/lib/systemd/system/kvrocks.service
[Unit]
Description=kvrocks

[Service]
User=kvrocks
Group=kvrocks
Type=simple
WorkingDirectory=/usr/local/kvrocks/
ExecStart=/usr/local/kvrocks/bin/kvrocks -c /usr/local/kvrocks/etc/kvrocks.conf
Restart=always
RestartSec=5
TimeoutSec=120
LimitNOFILE=65536
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
----
**OS: rocky 8.8
kvrocks version: 2.9.0**

[mapleFU]

Would you mind provide the gdb/lldb coredump backtrace?

[jiujiu994]

gdb /usr/local/kvrocks/bin/kvrocks /dev/core

warning: core file may not match specified executable file.
[New process 1]
Core was generated by `BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-477.10.1.el8_8.x86_64 root=UUID=36ec20a7-5'.
#0 0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
Backtrace stopped: Cannot access memory at address 0x0

[mapleFU]

This is not a coredump file
You can refer to https://wiki.archlinux.org/title/Core_dump

Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: E20240828 05:40:40.021790 2384433 signal_util.h:34] Ooops! Apache Kvrocks version 2.9.0 got signal: Segmentation fault (11)
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: E20240828 05:40:40.026152 2384433 signal_util.h:36] Stack trace (most recent call first):
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #0  0x0000000000513989 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #1  0x00007f3081b87cef at /lib64/libpthread.so.0
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #2  0x0000000000e1f1b4 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #3  0x00000000006a4ce6 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #4  0x0000000000540e63 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #5  0x000000000065b767 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #6  0x000000000065dc96 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #7  0x00000000005377cb at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #8  0x0000000000c754ad at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #9  0x0000000000c7b77b at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #10 0x0000000000c7c12e at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #11 0x0000000000672509 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #12 0x0000000000e7fe33 at /usr/local/kvrocks/bin/kvrocks
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #13 0x00007f3081b7d1c9 at /lib64/libpthread.so.0
Aug 28 05:40:40 ip-10-151-101-141.ap-east-1.compute.internal kvrocks[2384317]: #14 0x00007f3081263e72 at /lib64/libc.so.6

I think the full stack trace is neccessary to debug here.

[jiujiu994]

ERROR Logs

[root@ixxxx]# /data/kvrocks/logs/kvrocks.ERROR
Log file created at: 2024/08/28 07:12:32
Running on machine: ip-10-10-11-11
Running duration (h:mm:ss): 0:00:00
Log line format: [IWEF]yyyymmdd hh:mm:ss.uuuuuu threadid file:line] msg
E20240828 07:12:32.872967 2034969 signal_util.h:34] Ooops! Apache Kvrocks version 2.9.0 got signal: Segmentation fault (11)
E20240828 07:12:32.880759 2034969 signal_util.h:36] Stack trace (most recent call first):
#0 0x0000000000513989 at /usr/local/kvrocks/bin/kvrocks
#1 0x00007fd468f1ecef at /lib64/libpthread.so.0
#2 0x0000000000e1f1b4 at /usr/local/kvrocks/bin/kvrocks
#3 0x00000000006a4ce6 at /usr/local/kvrocks/bin/kvrocks
#4 0x0000000000540e63 at /usr/local/kvrocks/bin/kvrocks
#5 0x000000000065b767 at /usr/local/kvrocks/bin/kvrocks
#6 0x000000000065dc96 at /usr/local/kvrocks/bin/kvrocks
#7 0x00000000005377cb at /usr/local/kvrocks/bin/kvrocks
#8 0x0000000000c754ad at /usr/local/kvrocks/bin/kvrocks
#9 0x0000000000c7b77b at /usr/local/kvrocks/bin/kvrocks
#10 0x0000000000c7c12e at /usr/local/kvrocks/bin/kvrocks
#11 0x0000000000672509 at /usr/local/kvrocks/bin/kvrocks
#12 0x0000000000e7fe33 at /usr/local/kvrocks/bin/kvrocks
#13 0x00007fd468f141c9 at /lib64/libpthread.so.0
#14 0x00007fd4685fae72 at /lib64/libc.so.6
E20240828 07:12:32.880887 2034969 signal_util.h:37] It would be greatly appreciated if you could submit this crash to https://github.com/apache/kvrocks/issues along with the stacktrace above, logs and any relevant information.