Skip to content

Commit 63e28ea

Browse files
Copilotagockeeduardo-vpEduardo Velardejkotas
authored
[NativeAOT] Print OOM message before Abort() on Linux (#125311)
On Linux, NativeAOT processes terminating due to `OutOfMemoryException` (e.g. with `DOTNET_GCHeapHardLimit` set) printed only `Aborted` with no diagnostic context. ## Root cause `RuntimeExceptionHelpers.FailFast` detects the preallocated OOM exception via `minimalFailFast = (exception == PreallocatedOutOfMemoryException.Instance)` and skips **all** stderr output to avoid heap allocations — leaving the user with no indication of why the process died. ## Fix In the `minimalFailFast` path, print a hardcoded OOM message to stderr before calling `Abort()`. The write is wrapped in `try/catch {}` so a secondary allocation failure silently falls through to the existing abort path. **Before:** `Aborted` **After:** `Out of memory.` ## Test Added a new `OomHandling` smoke test in `src/tests/baseservices/exceptions/OutOfMemoryException/`. The test spawns itself as a subprocess with `DOTNET_GCHeapHardLimit=20000000` (32 MB) set, waits for the subprocess to run out of memory, and verifies that some OOM message appears in stderr. This covers both the preallocated OOM path (the fix) and the existing unhandled-exception path. The test is skipped on mobile and browser platforms that do not support process spawning. <!-- START COPILOT ORIGINAL PROMPT --> <details> <summary>Original prompt</summary> > > ---- > > *This section details on the original issue you should resolve* > > <issue_title>[NativeAOT] Out of memory reporting on Linux</issue_title> > <issue_description>### Repro > ```csharp > var l = new List<object>(); > for (; ; ) l.Add(new object()); > ``` > > Run the native aot compiled binary with with `export DOTNET_GCHeapHardLimit=2000000` set > > ### Actual result > > `Aborted` > > ### Expected result > > `Out of memory.` > > (Reported by partner team.)</issue_description> > > ## Comments on the Issue (you are @copilot in this section) > > <comments> > <comment_new><author>@</author><body> > Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas > See info in [area-owners.md](https://github.com/dotnet/runtime/blob/main/docs/area-owners.md) if you want to be subscribed. > <details> > <summary>Issue Details</summary> > <hr /> > > ### Repro > ```csharp > var l = new List<object>(); > for (; ; ) l.Add(new object()); > ``` > > Run the native aot compiled binary with with `export DOTNET_GCHeapHardLimit=2000000` set > > ### Actual result > > `Aborted` > > ### Expected result > > `Process is terminating due to OutOfMemoryException` > > (Reported by partner team.) > > <table> > <tr> > <th align="left">Author:</th> > <td>jkotas</td> > </tr> > <tr> > <th align="left">Assignees:</th> > <td>-</td> > </tr> > <tr> > <th align="left">Labels:</th> > <td> > > `area-NativeAOT-coreclr` > > </td> > </tr> > <tr> > <th align="left">Milestone:</th> > <td>8.0.0</td> > </tr> > </table> > </details></body></comment_new> > </comments> > </details> <!-- START COPILOT CODING AGENT SUFFIX --> - Fixes #82337 <!-- START COPILOT CODING AGENT TIPS --> --- 🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. [Learn more about Advanced Security.](https://gh.io/cca-advanced-security) --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: agocke <515774+agocke@users.noreply.github.com> Co-authored-by: Eduardo Velarde <32459232+eduardo-vp@users.noreply.github.com> Co-authored-by: Eduardo Velarde <evelardepola@microsoft.com> Co-authored-by: Jan Kotas <jkotas@microsoft.com> Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
1 parent 1b366f4 commit 63e28ea

4 files changed

Lines changed: 195 additions & 23 deletions

File tree

‎src/coreclr/nativeaot/System.Private.CoreLib/src/System/RuntimeExceptionHelpers.cs‎

Lines changed: 43 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -201,16 +201,23 @@ internal static void SerializeCrashInfo(RhFailFastReason reason, string? message
201201
int previousState = Interlocked.CompareExchange(ref s_crashInfoPresent, -1, 0);
202202
if (previousState == 0)
203203
{
204-
CrashInfo crashInfo = new();
204+
try
205+
{
206+
CrashInfo crashInfo = new();
205207

206-
crashInfo.Open(reason, Thread.CurrentOSThreadId, message ?? GetStringForFailFastReason(reason));
207-
if (exception != null)
208+
crashInfo.Open(reason, Thread.CurrentOSThreadId, message ?? GetStringForFailFastReason(reason));
209+
if (exception != null)
210+
{
211+
crashInfo.WriteException(exception);
212+
}
213+
crashInfo.Close();
214+
s_triageBufferAddress = crashInfo.TriageBufferAddress;
215+
s_triageBufferSize = crashInfo.TriageBufferSize;
216+
}
217+
catch
208218
{
209-
crashInfo.WriteException(exception);
219+
// If crash info serialization fails (for example, due to OOM), proceed without it.
210220
}
211-
crashInfo.Close();
212-
s_triageBufferAddress = crashInfo.TriageBufferAddress;
213-
s_triageBufferSize = crashInfo.TriageBufferSize;
214221

215222
s_crashInfoPresent = 1;
216223
}
@@ -235,8 +242,20 @@ internal static unsafe void FailFast(string? message = null, Exception? exceptio
235242
ulong previousThreadId = Interlocked.CompareExchange(ref s_crashingThreadId, currentThreadId, 0);
236243
if (previousThreadId == 0)
237244
{
238-
bool minimalFailFast = (exception == PreallocatedOutOfMemoryException.Instance);
239-
if (!minimalFailFast)
245+
bool minimalFailFast = exception == PreallocatedOutOfMemoryException.Instance;
246+
if (minimalFailFast)
247+
{
248+
// Minimal OOM fail-fast path: avoid heap allocations as much as possible, but still
249+
// report that OOM is the reason for the crash.
250+
try
251+
{
252+
// Try to print the same short message CoreCLR prints.
253+
Internal.Console.Error.Write("Out of memory.");
254+
Internal.Console.Error.WriteLine();
255+
}
256+
catch { }
257+
}
258+
else
240259
{
241260
Internal.Console.Error.Write(((exception == null) || (reason is RhFailFastReason.EnvironmentFailFast or RhFailFastReason.AssertionFailure)) ?
242261
"Process terminated. " : "Unhandled exception. ");
@@ -266,8 +285,21 @@ internal static unsafe void FailFast(string? message = null, Exception? exceptio
266285

267286
if ((exception != null) && (reason is not RhFailFastReason.AssertionFailure))
268287
{
269-
Internal.Console.Error.Write(exception.ToString());
270-
Internal.Console.Error.WriteLine();
288+
try
289+
{
290+
Internal.Console.Error.Write(exception.ToString());
291+
Internal.Console.Error.WriteLine();
292+
}
293+
catch
294+
{
295+
// If ToString() fails (for example, due to OOM), fall back to printing just the type name.
296+
try
297+
{
298+
Internal.Console.Error.Write(exception.GetType().FullName);
299+
Internal.Console.Error.WriteLine();
300+
}
301+
catch { }
302+
}
271303
}
272304

273305
#if TARGET_WINDOWS

‎src/libraries/System.Private.CoreLib/src/Internal/Console.Unix.cs‎

Lines changed: 19 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -10,25 +10,32 @@ namespace Internal
1010
public static partial class Console
1111
{
1212
[MethodImplAttribute(MethodImplOptions.NoInlining)]
13-
public static unsafe void Write(string s)
13+
public static void Write(string s)
1414
{
15-
byte[] bytes = Encoding.UTF8.GetBytes(s);
16-
fixed (byte* pBytes = bytes)
17-
{
18-
Interop.Sys.Log(pBytes, bytes.Length);
19-
}
15+
WriteCore(s, error: false);
2016
}
2117

2218
public static partial class Error
2319
{
2420
[MethodImplAttribute(MethodImplOptions.NoInlining)]
25-
public static unsafe void Write(string s)
21+
public static void Write(string s)
22+
{
23+
WriteCore(s, error: true);
24+
}
25+
}
26+
27+
private static unsafe void WriteCore(string s, bool error)
28+
{
29+
int byteCount = Encoding.UTF8.GetByteCount(s);
30+
Span<byte> bytes = (uint)byteCount < 1024 ? stackalloc byte[byteCount] : new byte[byteCount];
31+
int cbytes = Encoding.UTF8.GetBytes(s, bytes);
32+
33+
fixed (byte* pBytes = bytes)
2634
{
27-
byte[] bytes = Encoding.UTF8.GetBytes(s);
28-
fixed (byte* pBytes = bytes)
29-
{
30-
Interop.Sys.LogError(pBytes, bytes.Length);
31-
}
35+
if (error)
36+
Interop.Sys.LogError(pBytes, cbytes);
37+
else
38+
Interop.Sys.Log(pBytes, cbytes);
3239
}
3340
}
3441
}

‎src/tests/baseservices/exceptions/OutOfMemoryException/OutOfMemoryException.cs‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
// Licensed to the .NET Foundation under one or more agreements.
2+
// The .NET Foundation licenses this file to you under the MIT license.
3+
4+
// This test verifies that an out-of-memory condition produces a diagnostic
5+
// message on stderr before the process terminates.
6+
//
7+
// The test spawns itself as a subprocess with a small GC heap limit set via
8+
// DOTNET_GCHeapHardLimit so that the subprocess reliably runs out of memory.
9+
// The outer process then validates that the subprocess wrote the expected
10+
// OOM message to its standard error stream.
11+
12+
using System;
13+
using System.Collections.Generic;
14+
using System.Diagnostics;
15+
16+
class OutOfMemoryExceptionTest
17+
{
18+
const int Pass = 100;
19+
const int Fail = -1;
20+
const int TimeoutMilliseconds = 60 * 1000;
21+
22+
const string AllocateSmallArg = "--allocate-small";
23+
const string AllocateLargeArg = "--allocate-large";
24+
// The standard unhandled-exception path ("Unhandled exception. System.OutOfMemoryException...")
25+
// contains this token. The minimal OOM fail-fast path may only print a short "Out of memory." message.
26+
// The test validates that some OOM diagnostic is printed rather than just "Aborted" with no context.
27+
const string ExpectedOomToken = "OutOfMemoryException";
28+
const string ExpectedMinimalOomToken = "Out of memory.";
29+
30+
static int Main(string[] args)
31+
{
32+
if (args.Length > 0 && args[0] == AllocateSmallArg)
33+
{
34+
// Pre-allocate a flat array for storage.
35+
object[] storage = new object[8192];
36+
int idx = 0;
37+
// We expect ~2048 iterations in the first loop and ~64 iterations in the second.
38+
try { while (idx < storage.Length) storage[idx++] = GC.AllocateArray<byte>(16 * 1024, pinned: true); } catch (OutOfMemoryException) { }
39+
try { while (idx < storage.Length) storage[idx++] = GC.AllocateArray<byte>(256, pinned: true); } catch (OutOfMemoryException) { }
40+
// < 280 bytes free.
41+
// Use the smallest possible allocation to exhaust the last scraps.
42+
while (idx < storage.Length) storage[idx++] = GC.AllocateArray<byte>(1, pinned: true);
43+
return Fail;
44+
}
45+
46+
if (args.Length > 0 && args[0] == AllocateLargeArg)
47+
{
48+
// Subprocess mode: allocate 128 KB chunks until OOM is triggered.
49+
// This leaves some free memory when OOM fires, exercising the code
50+
// path where GetRuntimeException may allocate a new OutOfMemoryException.
51+
var list = new List<byte[]>();
52+
while (true) list.Add(new byte[128 * 1024]);
53+
}
54+
55+
// Controller mode: launch subprocesses with a GC heap limit and verify their output.
56+
int result = RunSubprocess(AllocateSmallArg, "small allocations");
57+
if (result != Pass)
58+
return result;
59+
60+
return RunSubprocess(AllocateLargeArg, "large allocations");
61+
}
62+
63+
static int RunSubprocess(string allocateArg, string description)
64+
{
65+
Console.WriteLine($"Testing OOM with {description}...");
66+
67+
string fileName = Environment.ProcessPath;
68+
string[] arguments = TestLibrary.Utilities.IsNativeAot
69+
? [allocateArg]
70+
: [typeof(OutOfMemoryExceptionTest).Assembly.Location, allocateArg];
71+
72+
var psi = new ProcessStartInfo(fileName, arguments)
73+
{
74+
RedirectStandardOutput = true,
75+
RedirectStandardError = true,
76+
};
77+
// 32 MB GC heap limit (0x2000000): small enough to exhaust quickly but large enough for startup.
78+
psi.Environment["DOTNET_GCHeapHardLimit"] = "0x2000000";
79+
psi.Environment["DOTNET_DbgEnableMiniDump"] = "0";
80+
81+
ProcessTextOutput output;
82+
try
83+
{
84+
output = Process.RunAndCaptureText(psi, TimeSpan.FromMilliseconds(TimeoutMilliseconds));
85+
}
86+
catch (TimeoutException)
87+
{
88+
Console.WriteLine($"Subprocess timed out after {TimeoutMilliseconds / 1000} seconds.");
89+
return Fail;
90+
}
91+
92+
if (output.ExitStatus.ExitCode == 0 || output.ExitStatus.ExitCode == Pass)
93+
{
94+
Console.WriteLine($"Subprocess exit code: {output.ExitStatus.ExitCode}");
95+
Console.WriteLine($"Subprocess stderr: {output.StandardError}");
96+
Console.WriteLine("Expected a non-success exit code from the OOM subprocess.");
97+
return Fail;
98+
}
99+
100+
string stderr = output.StandardError;
101+
102+
// Even in the small allocations case, the runtime might still have enough memory to construct
103+
// an OutOfMemoryException and print the full diagnostic.
104+
// Either token is acceptable, but at least one should be present to confirm that OOM was the reason for termination.
105+
if (!(stderr.Contains(ExpectedOomToken) || stderr.Contains(ExpectedMinimalOomToken)))
106+
{
107+
Console.WriteLine($"Subprocess exit code: {output.ExitStatus.ExitCode}");
108+
Console.WriteLine($"Subprocess stderr: {stderr}");
109+
Console.WriteLine("Expected OOM diagnostic token not found in subprocess stderr.");
110+
return Fail;
111+
}
112+
113+
return Pass;
114+
}
115+
}

‎src/tests/baseservices/exceptions/OutOfMemoryException/OutOfMemoryException.csproj‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
<Project Sdk="Microsoft.NET.Sdk">
2+
<PropertyGroup>
3+
<OutputType>Exe</OutputType>
4+
<CLRTestPriority>0</CLRTestPriority>
5+
<!-- This test spawns a subprocess; not supported on mobile, browser, or WASI platforms -->
6+
<CLRTestTargetUnsupported Condition="'$(TargetsAppleMobile)' == 'true' or '$(TargetsAndroid)' == 'true' or '$(TargetsBrowser)' == 'true' or '$(TargetsWasi)' == 'true'">true</CLRTestTargetUnsupported>
7+
<!-- https://github.com/dotnet/runtime/issues/129135 -->
8+
<CLRTestTargetUnsupported Condition="'$(TargetsOSX)' == 'true' and '$(TargetArchitecture)' == 'x64'">true</CLRTestTargetUnsupported>
9+
<RequiresProcessIsolation>true</RequiresProcessIsolation>
10+
<ReferenceXUnitWrapperGenerator>false</ReferenceXUnitWrapperGenerator>
11+
<!-- Mono doesn't enforce DOTNET_GCHeapHardLimit as a GC heap limit -->
12+
<DisableProjectBuild Condition="'$(RuntimeFlavor)' == 'mono'">true</DisableProjectBuild>
13+
</PropertyGroup>
14+
<ItemGroup>
15+
<Compile Include="OutOfMemoryException.cs" />
16+
<ProjectReference Include="$(TestLibraryProjectPath)" />
17+
</ItemGroup>
18+
</Project>

0 commit comments

Comments
 (0)