Re-enable Microsoft.Extensions.Hosting shutdown functional coverage in CI and harden shutdown startup/Helix/NET481 compatibility by Copilot · Pull Request #128252 · dotnet/runtime

Copilot · 2026-05-15T12:32:50Z

main PR

This PR now includes the full follow-up hardening required by CI feedback after re-enabling Microsoft.Extensions.Hosting functional shutdown coverage.

Final scope of changes

Re-enabled Hosting FunctionalTests in CI (removed CI gating/active issue suppression).
Updated shutdown test deployment to run from deployed output (AppContext.BaseDirectory) instead of source-tree project discovery.
Removed the load-bearing PreservePublishedApplicationForDebugging dependency by using a borrowed published app path that does not delete test output on cleanup.
Replaced external kill process usage in shutdown signaling with libc signal calls (SIGINT/SIGTERM) for Helix/minimal-image compatibility.
Kept Unix cleanup best-effort and resilient:
- graceful termination first (SIGTERM)
- hard-kill fallback (Process.Kill()) when needed
- tolerance for missing process tools / constrained environments
Increased startup/shutdown wait windows to 180 seconds for stress/JIT-stress stability.
Fixed the shutdown test startup race by wiring output observation before DeployAsync() starts process output capture, so the initial Started line cannot be missed.
Fixed net481 compatibility by using Marshal.GetLastWin32Error().

Validation

./build.sh clr+libs -rc release succeeds.
./build.sh clr+libs+host -rc release -lc release succeeds.
Microsoft.Extensions.Hosting.Functional.Tests builds successfully for net481.
dotnet build /t:test Microsoft.Extensions.Hosting.Functional.Tests.csproj -c Release -f net11.0 /p:TestFilter=Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests passes (2/2).
CI-feedback-driven shutdown/process-cleanup paths were updated to match Helix runtime constraints and avoid prior failure modes.

Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/f2762648-8aaf-44cd-a0bb-269b2a6b371d Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

rosebyte · 2026-05-18T11:31:53Z

@copilot , the shutdown tests are failing with this error on helix machines:

Comments
{ "HelixJobId": "bf5b8a6f-0bdd-40b3-9fbc-f0eec7148f40", "HelixWorkItemName": "Microsoft.Extensions.Hosting.Functional.Tests" }

Error message
System.IO.DirectoryNotFoundException : Could not find Microsoft.Extensions.Hosting.TestApp.csproj from base directory '/root/helix/work/workitem/e/'.

Stack trace
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.GetApplicationPath() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 173
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 53
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 35
--- End of stack trace from previous location ---

Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/e8bb9c1b-1794-4ebc-a563-bec6823fec1c Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

Copilot · 2026-05-18T11:55:24Z

@copilot , the shutdown tests are failing with this error on helix machines:

...

Fixed in b9cd702. I removed the source-tree lookup and switched the shutdown tests to run TestApp from the deployed test output (AppContext.BaseDirectory) using a custom ApplicationPublisher, so Helix no longer requires Microsoft.Extensions.Hosting.TestApp.csproj to exist in the workitem.

rosebyte · 2026-05-19T07:33:11Z

@copilot , this time it failed with this error:

Test Failures (700 tests failed)
🔹 [All failing tests from runtime]

⚠️ 698 failing tests not displayed for this pipeline due to space limitations

❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun [Console] [Details] [Artifacts] [0.70% failure rate]
[ 🚧 Create issue against infrastructure team] [ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestRun() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 34
--- End of stack trace from previous location ---
❌ Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown [Console] [Details] [Artifacts] [0.69% failure rate]
[ 🚧 Create issue against infrastructure team] [ 📄 Create issue in this repo]
Failing Configuration
AzureLinux.3.0.ArmArch.Open
Exception Message
System.ComponentModel.Win32Exception : An error occurred trying to start process 'pgrep' with working directory '/root/helix/work/workitem/e'. No such file or directory
CallStack
at Microsoft.Win32.SafeHandles.SafeProcessHandle.ForkAndExecProcess(ProcessStartInfo startInfo, String resolvedFilename, String[] argv, IDictionary2 env, String cwd, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, Boolean usesTerminal, SafeHandle[] inheritedHandles, Holder& waitStateHolder, Boolean throwOnNoExec) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 369 at Microsoft.Win32.SafeHandles.SafeProcessHandle.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles, Holder& waitStateHolder) in /_/src/libraries/System.Diagnostics.Process/src/Microsoft/Win32/SafeHandles/SafeProcessHandle.Unix.cs:line 213 at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo, SafeFileHandle stdinHandle, SafeFileHandle stdoutHandle, SafeFileHandle stderrHandle, SafeHandle[] inheritedHandles) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs:line 363 at System.Diagnostics.Process.Start() in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1272 at System.Diagnostics.Process.Start(ProcessStartInfo startInfo) in /_/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.cs:line 1383 at Microsoft.Extensions.Internal.ProcessExtensions.RunProcessAndWaitForExit(String fileName, String arguments, TimeSpan timeout, String& stdout) in /_/src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 93 at Microsoft.Extensions.Internal.ProcessExtensions.GetAllChildIdsUnix(Int32 parentId, ISet1 children, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 44
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process, TimeSpan timeout) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 33
at Microsoft.Extensions.Internal.ProcessExtensions.KillTree(Process process) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/ProcessExtensions.cs:line 17
at Microsoft.Extensions.Hosting.IntegrationTesting.ApplicationDeployer.ShutDownIfAnyHostProcess(Process hostProcess) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/ApplicationDeployer.cs:line 126
at Microsoft.Extensions.Hosting.IntegrationTesting.SelfHostDeployer.Dispose() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/IntegrationTesting/src/Deployers/SelfHostDeployer.cs:line 179
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ExecuteShutdownTest(String testName, String shutdownMechanic) in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 107
at Microsoft.AspNetCore.Hosting.FunctionalTests.ShutdownTests.ShutdownTestWaitForShutdown() in //src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs:line 41
--- End of stack trace from previous location ---

…fter Kill() Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

rosebyte · 2026-06-05T08:09:38Z

@copilot , you only addressed one of my comments, resolve all of them.

… 180s for stress environments Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

Copilot · 2026-06-05T08:15:46Z

@copilot , you only addressed one of my comments, resolve all of them.

Fixed in f9172b4. I've addressed both remaining issues:

KillProcessUnix now uses kill -TERM for graceful termination (SIGTERM) instead of Process.Kill() (SIGKILL)
Test timeouts increased from 60 seconds to 180 seconds (3 minutes) for startup and shutdown waits to handle stress/JIT-stress Helix environments

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

            if (!process.HasExited)
            {
                process.Kill();
+                // Wait for the process to actually exit after Kill() before accessing ExitCode
+                if (!process.WaitForExit(5000))


rosebyte · 2026-06-10T14:05:26Z

+                    if (!process.WaitForExit((int)timeout.TotalMilliseconds))
+                    {
+                        throw new TimeoutException($"Process {processId} did not exit within the allotted timeout of {timeout}.");
+                    }


@copilot, OK, do it that way.

Addressed in 16bc0339: KillProcessUnix no longer throws on SIGTERM timeout; it now escalates to Process.Kill() (hard kill) as best-effort cleanup instead of failing disposal.

rosebyte · 2026-06-10T14:06:16Z

+            catch (InvalidOperationException)
+            {
+                // Ignore if process has already exited.
+            }
+            catch (Win32Exception)
+            {
+                // Ignore permission or process-not-found errors (e.g., kill not available).
+            }


@copilot, implement the fix.

Implemented in 16bc0339: Unix termination now uses libc kill(pid, SIGTERM) and falls back to Process.Kill() when graceful termination fails (including Win32Exception/error paths), so orphaned processes are still terminated.

svick · 2026-06-05T14:33:44Z

+                var psi = new ProcessStartInfo
+                {
+                    FileName = "kill",
+                    Arguments = $"-TERM {processId}",
+                    RedirectStandardOutput = true,
+                    RedirectStandardError = true,
+                    UseShellExecute = false
+                };


On .Net 11, you could use Process.GetProcessById(processId).SafeHandle.Signal(PosixSignal.SIGTERM). I think this only runs on Unix on .Net 11, so it could make sense to use that instead.

Or you could use the PInvoke approach from ShutdownTests. Both calls to kill should probably use the same approach.

@copilot , this sounds a good idea, implement it.

Implemented in 16bc033. ProcessExtensions.KillProcessUnix now uses libc kill(pid, SIGTERM) (matching the ShutdownTests signal approach), waits for exit, and falls back to Process.Kill() if graceful termination fails.

Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

rosebyte

@copilot, I left two comments to implement.

rosebyte · 2026-06-10T14:05:26Z

+                    if (!process.WaitForExit((int)timeout.TotalMilliseconds))
+                    {
+                        throw new TimeoutException($"Process {processId} did not exit within the allotted timeout of {timeout}.");
+                    }


@copilot, OK, do it that way.

rosebyte · 2026-06-10T14:06:16Z

+            catch (InvalidOperationException)
+            {
+                // Ignore if process has already exited.
+            }
+            catch (Win32Exception)
+            {
+                // Ignore permission or process-not-found errors (e.g., kill not available).
+            }


@copilot, implement the fix.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

+                await started.Task.WaitAsync(TimeSpan.FromSeconds(180));

                SendShutdownSignal(deployer.HostProcess);



rosebyte · 2026-06-10T16:40:50Z

@copilot , I left comments for you, please implement them.

Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

Copilot · 2026-06-10T17:20:42Z

@copilot , I left comments for you, please implement them.

Fixed in 14a6f4c97. I addressed the remaining shutdown-test startup race by subscribing before DeployAsync() starts output capture and forwarding early process output through SelfHostDeployer, so the test can no longer miss the initial Started line.

Initial plan

a12d2b9

Copilot AI assigned Copilot and rosebyte May 15, 2026

Copilot AI requested review from Copilot and removed request for Copilot May 15, 2026 12:32

Copilot started work on behalf of rosebyte May 15, 2026 12:32 View session

Copilot AI linked an issue May 15, 2026 that may be closed by this pull request

Enable FunctionalTests/TestApp on Extensions Hosting #34090

Open

github-actions Bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label May 15, 2026

Re-enable Hosting functional TestApp shutdown tests

6bc1499

Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/f2762648-8aaf-44cd-a0bb-269b2a6b371d Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

Copilot AI requested review from Copilot and removed request for Copilot May 15, 2026 13:03

Copilot AI changed the title ~~[WIP] Fix FunctionalTests/TestApp on Extensions Hosting~~ Re-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverage May 15, 2026

Copilot AI requested a review from rosebyte May 15, 2026 13:07

Copilot finished work on behalf of rosebyte May 15, 2026 13:07

github-actions Bot added area-Extensions-Hosting and removed needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners labels May 15, 2026

This was referenced May 16, 2026

The Operation will be canceled. The next steps may not contain expected logs. dotnet/dnceng#3008

Open

browser-wasm linux Release LibraryTests queues timing out #117974

Open

Copilot started work on behalf of rosebyte May 18, 2026 11:32 View session

Fix Hosting shutdown tests on Helix without source tree

b9cd702

Agent-Logs-Url: https://github.com/dotnet/runtime/sessions/e8bb9c1b-1794-4ebc-a563-bec6823fec1c Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

Copilot AI requested review from Copilot and removed request for Copilot May 18, 2026 11:55

Copilot AI changed the title ~~Re-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverage~~ Re-enable Microsoft.Extensions.Hosting FunctionalTests/TestApp shutdown coverage across local and Helix environments May 18, 2026

Copilot finished work on behalf of rosebyte May 18, 2026 11:56

Copilot finished work on behalf of rosebyte June 5, 2026 07:54

Copilot started work on behalf of rosebyte June 5, 2026 07:54 View session

Fix race condition in WaitForExitOrKill by waiting for process exit a…

01891b0

…fter Kill() Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

Copilot AI requested review from Copilot and removed request for Copilot June 5, 2026 08:01

Copilot finished work on behalf of rosebyte June 5, 2026 08:02

Copilot started work on behalf of rosebyte June 5, 2026 08:09 View session

Use kill -TERM for graceful termination and increase test timeouts to…

f9172b4

… 180s for stress environments Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

Copilot AI requested review from Copilot and removed request for Copilot June 5, 2026 08:15

Copilot finished work on behalf of rosebyte June 5, 2026 08:16

Merge branch 'main' into copilot/enable-functional-tests-on-hosting

5986a65

Copilot AI review requested due to automatic review settings June 5, 2026 12:12

rosebyte marked this pull request as ready for review June 5, 2026 12:12

Copilot started reviewing on behalf of rosebyte June 5, 2026 12:12 View session

rosebyte requested review from cincuranet, mrek-msft and svick June 5, 2026 12:13

Copilot AI reviewed Jun 5, 2026

View reviewed changes

svick reviewed Jun 5, 2026

View reviewed changes

Use libc SIGTERM in Hosting test process cleanup

16bc033

Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

build-analysis Bot mentioned this pull request Jun 6, 2026

DnsGetHostAddresses_LocalhostWithTrailingDot_ReturnsLoopback - SocketException : Name or service not known #128759

Open

rosebyte requested changes Jun 10, 2026

View reviewed changes

Merge branch 'main' into copilot/enable-functional-tests-on-hosting

bd3dc0c

Copilot AI reviewed Jun 10, 2026

View reviewed changes

Comment thread src/libraries/Microsoft.Extensions.Hosting/tests/FunctionalTests/ShutdownTests.cs

Comment on lines +97 to 100

await started.Task.WaitAsync(TimeSpan.FromSeconds(180));

SendShutdownSignal(deployer.HostProcess);

Fix Hosting shutdown output subscription race

14a6f4c

Co-authored-by: rosebyte <14963300+rosebyte@users.noreply.github.com>

		await started.Task.WaitAsync(TimeSpan.FromSeconds(180));

		SendShutdownSignal(deployer.HostProcess);

Conversation

Copilot AI commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Final scope of changes

Validation

Uh oh!

rosebyte commented May 18, 2026

Uh oh!

Copilot AI commented May 18, 2026

Uh oh!

rosebyte commented May 19, 2026

Uh oh!

rosebyte commented Jun 5, 2026

Uh oh!

Copilot AI commented Jun 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

rosebyte Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

rosebyte Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

svick Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

rosebyte Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

rosebyte left a comment

Choose a reason for hiding this comment

Uh oh!

rosebyte Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

rosebyte Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

rosebyte commented Jun 10, 2026

Uh oh!

Copilot AI commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented May 15, 2026 •

edited

Loading