Resolved callstacks in crash reporter dialog on Linux for local builds (Native backend)

[ryan-roberts]

Questions for maintainers

1. Is this the right approach for getting resolved callstacks in the crash dialog on Linux with the Native backend? Or is there a better hook point we're missing?
2. Would you consider taking this upstream? The OnCrash override + sidecar pattern seems generally useful for any UE project that doesn't upload debug symbols to sentry.io.
3. Is there appetite for handling this entirely in C++ (avoiding the bash/python wrapper) — e.g., by patching the sentry_value_t event directly in OnCrash before returning it? We tried this but the event's thread stacktraces didn't seem to propagate to the envelope that the crash reporter reads.

Problem

When using the Native backend (UseNativeBackend=True) on Linux, the Sentry crash reporter dialog shows only raw hex addresses in the stacktrace tab of the Sentry crash dialog. Resolved callstacks are helpful for local developer workflows. We use Sentry with our cross-platform UE5.6 title with self-hosted Sentry back-end data.

Before switching to the Sentry crash dialog we were using UE's crash dialog which did show resolved callstacks. UE's Linux toolchain obtains this functionality by running objcopy --strip-all post-link, moving symbols into a .debug sidecar and UE also generates .sym files for runtime symbol resolution via FPlatformStackWalk. By comparison Sentry-native's out-of-process crash handler (sentry-crash) only captures a minidump and has no knowledge of UE's proprietary .sym format.

Current Solution in sentry-unreal

Two-part approach that resolves symbols in-process at crash time and injects them into the envelope before the crash reporter displays it:

Part A: In-process symbol resolution (C++)

Override OnCrash in FLinuxSentrySubsystem to:

1. Unwind the crash stack via sentry_unwind_stack_from_ucontext()
2. Resolve each frame using FPlatformStackWalk::ProgramCounterToHumanReadableString() (reads UE's .sym files)
3. Write resolved frames to a JSON sidecar (resolved_frames.json) using async-signal-safe POSIX I/O
4. Print the resolved callstack to stderr for log capture

Part B: Envelope patching (wrapper script)

Replace Sentry.CrashReporter with a bash wrapper that:

1. Reads the resolved frames sidecar
2. Patches the envelope's crashed thread stacktrace.frames with the resolved data
3. Execs the real crash reporter binary (renamed to Sentry.CrashReporter.real)

The crash dialog now displays function names + source locations instead of hex addresses.

Project configuration (DefaultEngine.ini)

The relevant [/Script/Sentry.SentrySettings] configuration for this feature:

[/Script/Sentry.SentrySettings]
InitAutomatically=True
UseNativeBackend=True
EnableExternalCrashReporter=True
EnableCrashReporterContextPropagation=True
AttachStacktrace=True

Key settings:

• UseNativeBackend=True — Required. Switches from Crashpad to the Native backend (sentry-crash), which is what provides the OnCrash callback and sentry_unwind_stack_from_ucontext(). Without this, the Crashpad backend is used and the OnCrash override never fires.
• EnableExternalCrashReporter=True — Required. This causes sentry-crash to invoke Sentry.CrashReporter (our wrapper) after writing the envelope. Without this, no crash dialog appears and the wrapper is never called.
• EnableCrashReporterContextPropagation=True — Passes release/environment info to the crash reporter process.

Diff

The code changes are small (~110 lines of C++, ~140 lines of bash/python wrapper):

diff --git a/Source/Sentry/Private/Linux/LinuxSentrySubsystem.h b/Source/Sentry/Private/Linux/LinuxSentrySubsystem.h
index 1043e6db..420940cb 100644
--- a/Source/Sentry/Private/Linux/LinuxSentrySubsystem.h
+++ b/Source/Sentry/Private/Linux/LinuxSentrySubsystem.h
@@ -12,6 +12,8 @@ public:
 	virtual void InitWithSettings(const USentrySettings* settings, const FSentryCallbackHandlers& callbackHandlers) override;
 
 protected:
+	virtual sentry_value_t OnCrash(const sentry_ucontext_t* uctx, sentry_value_t event, void* closure) override;
+
 	virtual void ConfigureHandlerPath(sentry_options_t* Options) override;
 	virtual void ConfigureDatabasePath(sentry_options_t* Options) override;
 	virtual void ConfigureCertsPath(sentry_options_t* Options) override;

diff --git a/Source/Sentry/Private/Linux/LinuxSentrySubsystem.cpp b/Source/Sentry/Private/Linux/LinuxSentrySubsystem.cpp
index f3e0ac26..9f816c48 100644
--- a/Source/Sentry/Private/Linux/LinuxSentrySubsystem.cpp
+++ b/Source/Sentry/Private/Linux/LinuxSentrySubsystem.cpp
@@ -9,14 +9,119 @@
 #include "Utils/SentryPlatformDetectionUtils.h"
 
 #include "GenericPlatform/GenericPlatformOutputDevices.h"
+#include "HAL/PlatformStackWalk.h"
 #include "Misc/Paths.h"
 
+#include <fcntl.h>
+#include <unistd.h>
+
 #if USE_SENTRY_NATIVE
 
+static FString ResolvedFramesSidecarPath;
+
+sentry_value_t FLinuxSentrySubsystem::OnCrash(const sentry_ucontext_t* uctx, sentry_value_t event, void* closure)
+{
+	static constexpr size_t MaxFrames = 64;
+	void* Addresses[MaxFrames];
+
+	size_t FrameCount = sentry_unwind_stack_from_ucontext(uctx, Addresses, MaxFrames);
+
+	if (FrameCount > 0 && !ResolvedFramesSidecarPath.IsEmpty())
+	{
+		// Write resolved frames to sidecar (for crash reporter wrapper) and stderr (for log).
+		// Uses POSIX I/O only (async-signal-safe).
+		int fd = open(TCHAR_TO_UTF8(*ResolvedFramesSidecarPath), O_WRONLY | O_CREAT | O_TRUNC, 0644);
+		if (fd >= 0)
+		{
+			write(fd, "[\n", 2);
+		}
+
+		write(STDERR_FILENO, "\n[sentry] Crash callstack:\n", 27);
+
+		bool first = true;
+		for (size_t i = FrameCount; i > 0; --i)
+		{
+			uint64 PC = reinterpret_cast<uint64>(Addresses[i - 1]);
+
+			ANSICHAR HumanReadable[1024];
+			HumanReadable[0] = '\0';
+			FPlatformStackWalk::ProgramCounterToHumanReadableString(
+				static_cast<int32>(FrameCount - i), PC, HumanReadable, sizeof(HumanReadable));
+
+			// Log frame to stderr
+			if (HumanReadable[0] != '\0')
+			{
+				write(STDERR_FILENO, "  ", 2);
+				size_t hrLen = 0;
+				while (HumanReadable[hrLen] != '\0') ++hrLen;
+				write(STDERR_FILENO, HumanReadable, hrLen);
+				write(STDERR_FILENO, "\n", 1);
+			}
+
+			// Escape/sanitize for valid JSON (ASCII printable only)
+			ANSICHAR Escaped[2048];
+			int EscIdx = 0;
+			for (int j = 0; HumanReadable[j] != '\0' && EscIdx < (int)sizeof(Escaped) - 2; ++j)
+			{
+				char c = HumanReadable[j];
+				if (c == '"' || c == '\\')
+				{
+					Escaped[EscIdx++] = '\\';
+					Escaped[EscIdx++] = c;
+				}
+				else if (c >= 0x20 && c < 0x7F)
+				{
+					Escaped[EscIdx++] = c;
+				}
+				else
+				{
+					Escaped[EscIdx++] = '?';
+				}
+			}
+			Escaped[EscIdx] = '\0';
+
+			if (fd >= 0)
+			{
+				char LineBuf[2400];
+				int LineLen;
+				if (HumanReadable[0] != '\0')
+				{
+					LineLen = snprintf(LineBuf, sizeof(LineBuf),
+						"%s{\"instruction_addr\":\"0x%llx\",\"function\":\"%s\"}",
+						first ? "" : ",\n",
+						(unsigned long long)PC, Escaped);
+				}
+				else
+				{
+					LineLen = snprintf(LineBuf, sizeof(LineBuf),
+						"%s{\"instruction_addr\":\"0x%llx\"}",
+						first ? "" : ",\n",
+						(unsigned long long)PC);
+				}
+				if (LineLen > 0)
+				{
+					write(fd, LineBuf, LineLen);
+				}
+			}
+			first = false;
+		}
+
+		if (fd >= 0)
+		{
+			write(fd, "\n]\n", 3);
+			close(fd);
+		}
+	}
+
+	return FGenericPlatformSentrySubsystem::OnCrash(uctx, event, closure);
+}
+
 void FLinuxSentrySubsystem::InitWithSettings(const USentrySettings* Settings, const FSentryCallbackHandlers& CallbackHandlers)
 {
 	FGenericPlatformSentrySubsystem::InitWithSettings(Settings, CallbackHandlers);
 
+	ResolvedFramesSidecarPath = FPaths::Combine(GetDatabasePath(), TEXT("resolved_frames.json"));
+
 	if (Settings->EnableExternalCrashReporter)
 	{
 		ConfigureCrashReporterAppearance(Settings);

diff --git a/Source/Sentry/Sentry.Build.cs b/Source/Sentry/Sentry.Build.cs
index de828a00..5d29b664 100644
--- a/Source/Sentry/Sentry.Build.cs
+++ b/Source/Sentry/Sentry.Build.cs
@@ -220,6 +220,7 @@ public class Sentry : ModuleRules
 			if (bEnableExternalCrashReporter)
 			{
 				RuntimeDependencies.Add(Path.Combine(PlatformBinariesPath, "Sentry.CrashReporter"), Path.Combine(PlatformThirdPartyPath, "Sentry.CrashReporter"));
+				RuntimeDependencies.Add(Path.Combine(PlatformBinariesPath, "Sentry.CrashReporter.real"), Path.Combine(PlatformThirdPartyPath, "Sentry.CrashReporter.real"));
 				StageCrashReporterResources(Target);
 			}

Wrapper script (Sentry.CrashReporter replaces the binary, real binary renamed to Sentry.CrashReporter.real):

#!/usr/bin/env bash
# Wrapper around Sentry.CrashReporter that patches the envelope with resolved symbols
# before the crash reporter UI displays the stacktrace.

SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
REAL_REPORTER="$SCRIPT_DIR/Sentry.CrashReporter.real"
ENVELOPE_PATH="$1"

if [ -z "$ENVELOPE_PATH" ] || [ ! -f "$ENVELOPE_PATH" ]; then
    exec "$REAL_REPORTER" "$@"
fi

# The sidecar is written by OnCrash into the .sentry-native database directory.
ENVELOPE_DIR="$(cd "$(dirname "$ENVELOPE_PATH")" && pwd)"
SIDECAR=""
for CANDIDATE in "$ENVELOPE_DIR/resolved_frames.json" "$ENVELOPE_DIR/../resolved_frames.json"; do
    if [ -f "$CANDIDATE" ]; then
        SIDECAR="$CANDIDATE"
        break
    fi
done

if [ -n "$SIDECAR" ]; then
    python3 - "$ENVELOPE_PATH" "$SIDECAR" << 'PYTHON_EOF'
import json, sys, os

envelope_path = sys.argv[1]
sidecar_path = sys.argv[2]

try:
    with open(sidecar_path, 'rb') as f:
        resolved_frames = json.loads(f.read().decode('utf-8', errors='replace'))
except Exception:
    sys.exit(0)

if not resolved_frames:
    sys.exit(0)

with open(envelope_path, 'rb') as f:
    envelope_data = f.read()

lines = envelope_data.split(b'\n', 2)
if len(lines) < 3:
    sys.exit(0)

envelope_header = lines[0]
try:
    item_header = json.loads(lines[1])
except Exception:
    sys.exit(0)

event_length = item_header.get('length', 0)
rest = lines[2]
event_json_raw = rest[:event_length]
remainder = rest[event_length:]

try:
    event = json.loads(event_json_raw)
except Exception:
    sys.exit(0)

# Inject resolved frames into crashed thread
threads = event.get('threads', {})
values = threads.get('values', []) if isinstance(threads, dict) else []

patched = False
for thread in values:
    if thread.get('crashed'):
        thread['stacktrace'] = {'frames': resolved_frames}
        patched = True
        break

if not patched:
    thread_entry = {'id': 0, 'crashed': True, 'stacktrace': {'frames': resolved_frames}}
    if isinstance(threads, dict) and 'values' in threads:
        threads['values'].insert(0, thread_entry)
    else:
        event['threads'] = {'values': [thread_entry]}
    patched = True

if patched:
    new_event_json = json.dumps(event, separators=(',', ':')).encode('utf-8')
    item_header['length'] = len(new_event_json)
    new_item_header = json.dumps(item_header, separators=(',', ':')).encode('utf-8')
    with open(envelope_path, 'wb') as f:
        f.write(envelope_header + b'\n')
        f.write(new_item_header + b'\n')
        f.write(new_event_json)
        f.write(remainder)

try:
    os.unlink(sidecar_path)
except Exception:
    pass
PYTHON_EOF
fi

exec "$REAL_REPORTER" "$@"

Notes

• Async-signal safety: The C++ OnCrash runs in signal context. We use only write(), open(), close(), snprintf() (all async-signal-safe). FPlatformStackWalk::ProgramCounterToHumanReadableString() reads .sym files that are already mmap'd — this is what UE's own crash handler does.
• Sentry.io: The patched envelope is uploaded, so sentry.io will show resolved frames. However, per sentry-native#1147, server-side minidump processing may overwrite event frames during ingestion.
• Platform scope: This is Linux-specific. Windows uses PDB + Crashpad (different architecture). Mac Native backend could use the same pattern.
• Dependencies: python3 is required at runtime for envelope patching (present on all supported Linux targets).

[linear-code]

UNREAL-148

[jpnurmi]

Hey, it might soon work out of the box. There are some related unreleased changes in the pipeline:

• feat: symbol demangling sentry-desktop-crash-reporter#212
• fix(native): resolve symbol names for crashed thread on Linux sentry-native#1764
• feat(native): symbolicated stacktraces for non-crashed threads on Linux sentry-native#1747