fix shutdown tests

Project: dotnet/runtime | Language: C# | Concepts: error-handling, testing-strategy, configuration-management | Difficulty: mid
Link: https://github.com/dotnet/runtime/pull/129853

The problem

Shutdown-related functional tests were failing in CI environments (notably Helix) because the test infrastructure assumed the presence of a global dotnet executable on the system PATH. In some test layouts (e.g., NativeAOT, ReadyToRun, or single-file publish), this assumption does not hold, causing process launches to fail. Additionally, the shutdown logic could throw exceptions when attempting to clean up Process objects that were never started, masking the original failure and making diagnostics difficult.

Why it's non-trivial

A naive fix might simply "add dotnet to PATH" or catch more exceptions, but this would not generalize across all test environments or publishing modes. The test runner must robustly locate the correct dotnet host for the current test layout (which may not be globally installed), handle architecture-specific launches, and ensure that cleanup code does not introduce new errors or mask the real cause of test failures.

The approach

1. Export the runtime path for test processes

The shell script that launches tests now exports the RUNTIME_PATH environment variable, making it available to subprocesses:

-      RUNTIME_PATH=$2
+      # Exported so that tests which need to launch a portable application of their own can find the
+      # same host the test run itself was handed. RunnerTemplate.cmd's "set" already does this.
+      export RUNTIME_PATH=$2

This ensures that any process launched by the tests can reliably find the correct dotnet host, regardless of the system PATH.

2. Robustly resolve the dotnet muxer path

The logic for finding the dotnet executable was rewritten to first check the RUNTIME_PATH variable, then fall back to the current process's executable if necessary:

+        public static string DotNetMuxerPath { get; } = FindDotNetMuxer();
...
+        private static string FindDotNetMuxer()
+        {
+            var runtimePath = Environment.GetEnvironmentVariable(RuntimePathVariableName);
+            if (!string.IsNullOrEmpty(runtimePath))
+            {
+                var fromRunScript = Path.Combine(runtimePath, DotNetExecutableName);
+                if (File.Exists(fromRunScript))
+                {
+                    return fromRunScript;
+                }
+            }
+#if NETFRAMEWORK
+            return null;
+#else
+            // Outside the run script the only host we can vouch for is the one running the tests, which is
+            // the muxer itself on every leg that does not publish the tests as a self-contained application.
+            var processPath = Environment.ProcessPath;
+            return string.Equals(Path.GetFileName(processPath), DotNetExecutableName, StringComparison.OrdinalIgnoreCase)
+                ? processPath
+                : null;
+#endif
+        }

This approach avoids reliance on global PATH and ensures the correct host is used, even in Helix or self-contained publish scenarios.

3. Harden shutdown/cleanup logic

The shutdown logic was updated to avoid throwing exceptions if the process was never started or already disposed:

-        protected void ShutDownIfAnyHostProcess(Process hostProcess)
-        {
-            if (hostProcess != null && !hostProcess.HasExited)
+        protected void ShutDownIfAnyHostProcess(Process hostProcess)
+        {
+            if (hostProcess is not null && IsRunning(hostProcess))
             {
                 ...
-                if (!hostProcess.HasExited)
+                if (IsRunning(hostProcess))
                 {
-                    Logger.LogWarning("Unable to terminate the host process with process Id '{processId}", hostProcess.Id);
+                    Logger.LogWarning("Unable to terminate the host process with process Id '{processId}'", hostProcess.Id);
                 }
                 else
                 {
                     Logger.LogInformation("Successfully terminated the host process with process Id '{processId}'", hostProcess.Id);
                 }
             }
         }
 
+        // Process.HasExited throws InvalidOperationException ("No process is associated with this object")
+        // when the process was never started (and also after the Process has been disposed, which disassociates
+        // it). Treat that as "not running" so shutdown cleanup stays non-throwing rather than masking the
+        // original start failure with a misleading exception.
+        private static bool IsRunning(Process hostProcess)
+        {
+            try
+            {
+                return !hostProcess.HasExited;
+            }
+            catch (InvalidOperationException)
+            {
+                return false;
+            }
+        }

This prevents misleading exceptions during cleanup and ensures the original error is surfaced.

4. Only run shutdown tests when the environment supports it

The shutdown tests are now conditionally enabled only when the test environment can actually deploy and launch the test application:

-        [Fact]
+        [ConditionalFact(typeof(ShutdownTests), nameof(CanDeployTestApplication))]
         [PlatformSpecific(TestPlatforms.Linux)]
         public async Task ShutdownTestRun()
         {
             await ExecuteShutdownTest(nameof(ShutdownTestRun), "Run");
         }
...
+        public static bool CanDeployTestApplication
+        {
+            get
+            {
+                if (DotNetCommands.DotNetMuxerPath is null)
+                {
+                    return false;
+                }
+
+                var applicationPath = Path.Combine(AppContext.BaseDirectory, TestApplicationName);
+                return File.Exists(applicationPath + ".dll") && File.Exists(applicationPath + ".deps.json");
+            }
+        }

This avoids false negatives in CI runs where the correct host or dependencies are not present.

5. Remove previous test exclusions

The PR removes explicit project exclusions for these tests in certain publish modes, as the new logic makes them robust:

-    <!-- https://github.com/dotnet/runtime/issues/129832 -->
-    <ProjectExclusions Include="$(MSBuildThisFileDirectory)Microsoft.Extensions.Hosting/tests/FunctionalTests/Microsoft.Extensions.Hosting.Functional.Tests.csproj" />

From the review thread

jkotas:
Is this really the case?
RemoteExecutor is launching a ton of process in CI/Helix machines. It does not have a retry loop like this and we do not see a problems with that.

rosebyte:
You were right, and it turned out not to be transient at all. TestReadyToRun/TestNativeAot
imply TestSingleFile=true, so the test assembly is published self-contained and launched via
its own apphost. On Unix a bare "dotnet" is resolved against the running executable's directory
before PATH, so on the normal legs it picks up the testhost muxer, but on these legs there is no
muxer next to the apphost and Process.Start fails outright. The retry loop is gone.

jkotas:
Where is the RunScriptHost going to be set when the tests run in Helix?
There is existing RUNTIME_PATH environment variable that is used by the test infra today. Can we use that instead of inventing out own?

rosebyte:
Good idea, changed to dotnet lookup instead.

Takeaways

Do not assume global executables are available in CI/test environments: Always resolve tool paths from the test's actual deployment context or explicit environment variables.
Guard cleanup logic against partially-initialized resources: Cleanup code should not throw if a process was never started or already disposed; this prevents masking the real failure cause.
Conditionally enable tests based on environment capabilities: Use runtime checks to determine if required dependencies are present, and skip tests gracefully when not.
Prefer existing environment variables and conventions over inventing new ones: This improves compatibility and reduces surprises in complex build/test infrastructure.