From 8a85683a9c2aa2522ad8dd6a3ed41c146a9c4d18 Mon Sep 17 00:00:00 2001 From: = Date: Mon, 14 Sep 2026 21:13:00 +0200 Subject: [PATCH 01/10] docs: add the Native AOT PR 2 implementation plan Co-Authored-By: Claude Opus 5 (1M context) --- ...09-14-native-aot-pr2-native-build-smoke.md | 1340 +++++++++++++++++ 1 file changed, 1340 insertions(+) create mode 100644 docs/superpowers/plans/2026-09-14-native-aot-pr2-native-build-smoke.md diff --git a/docs/superpowers/plans/2026-09-14-native-aot-pr2-native-build-smoke.md b/docs/superpowers/plans/2026-09-14-native-aot-pr2-native-build-smoke.md new file mode 100644 index 0000000..7cd1348 --- /dev/null +++ b/docs/superpowers/plans/2026-09-14-native-aot-pr2-native-build-smoke.md @@ -0,0 +1,1340 @@ +# Native AOT PR 2: Native Build and PR Smoke Test Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make `ProjGraph.Cli` and `ProjGraph.Mcp` publish as working Native AOT executables on demand, and add a CI job that checks the native binaries against the JIT build of the same commit on every PR. + +**Architecture:** AOT stays opt-in: no csproj sets `PublishAot`, so `dotnet pack` and `publish.yml` don't change. The CLI gets trimmer root assemblies for Spectre.Console.Cli's reflection, two justified suppressions, and build-time AOT/trim analyzers. The MCP server gets the analyzers and stops treating third-party ILC warnings as errors. A new test project, `ProjGraph.Tests.Smoke.Aot`, launches a native and a JIT build side by side and requires identical output. It skips unless four environment variables point at the builds, so the existing CI matrix and local `dotnet test` are unaffected. A new `aot-smoke` job on `ubuntu-latest` publishes both executables and runs the suite. + +**Tech Stack:** .NET 10 (SDK per `global.json`; verified on 10.0.401), C#, xUnit 2.9 + FluentAssertions 8, ModelContextProtocol 2.2.0 (stdio client), Spectre.Console.Cli 0.55.0, GitHub Actions. + +**Spec:** `docs/superpowers/specs/2026-09-14-native-aot-design.md` (section "PR 2: native build and PR smoke test"). Read it before starting. PR 1 (`docs/superpowers/plans/2026-09-14-native-aot-pr1-library-preparation.md`) is merged as `86082ea`. + +## Global Constraints + +- `TreatWarningsAsErrors=true`, `AnalysisMode=All`, and `EnforceCodeStyleInBuild=true` apply to every project, tests included. Sonar, Roslynator, VS Threading, and CA analyzers all run during build. Any warning is a build failure. Private members that have a `` also need their ``/``/`` elements (RCS1140/RCS1141). +- XML documentation is required on all public APIs (`GenerateDocumentationFile=true`). Test projects have CS1591 disabled, but the Roslynator rules above still apply to any doc comment you write. +- CI runs `dotnet format ProjGraph.slnx --no-restore --verify-no-changes`, which must pass. +- Package versions are central (`Directory.Packages.props`). This PR adds no package. +- **No csproj sets `PublishAot`.** AOT is enabled only with `-p:PublishAot=true` on the command line (spec PR 2 intro, exit criterion). +- No change to any command, option, MCP tool, prompt, resource, or output format (spec Non-goals). +- Smoke-test oracle: native output is compared against the JIT build of the **same commit**, never against committed samples (spec decision 7). Don't regenerate samples. +- Don't edit `ProjGraph.slnx` with `dotnet sln add`: it rewrites every `` as ``. Insert the one line by hand. +- Always pass `--artifacts-path` to a native `dotnet publish` (see deviation 2). Run native publishes one at a time: each ILC run uses several GB of RAM, and back-to-back publishes on a 16 GB machine have been OOM-killed. + +## Deviations from the spec (intentional; mention them in the PR description) + +1. **No `PublishSingleFile` condition on the MCP project.** Spec §2.1 makes `PublishSingleFile` conditional on `PublishAot != true`. Verified on SDK 10.0.401: with it left unconditional, the AOT publish succeeds and produces the same 42,946,344-byte binary, so the condition is a no-op. PR 3 removes `PublishSingleFile` entirely. +2. **`--artifacts-path` on the native publish.** `ProjGraph.Mcp` is self-contained, so its `dotnet build` output already lives in `bin/Release/net10.0/linux-x64/`. An in-place `dotnet publish -r linux-x64 -p:PublishAot=true` reuses that directory: it deletes the JIT apphost and rewrites `ProjGraph.Mcp.runtimeconfig.json` with AOT feature switches (`IsDynamicCodeSupported=false`, …), which corrupts the JIT reference. With `--artifacts-path`, all 301 `ProjGraph.*` files under `src/*/bin` and `tests/*/bin` were byte-identical before and after both publishes. +3. **Reference variables accept a `.dll` or an executable.** A path ending in `.dll` is started through the `dotnet` host (`DOTNET_HOST_PATH` when `dotnet test` provides it); anything else is started directly. The MCP reference is `src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp.dll`: running a self-contained build's `.dll` through `dotnet` works (verified). +4. **A fifth variable, `PROJGRAPH_SMOKE_REQUIRED`.** When it is `true`, `[AotSmokeFact]` never skips, so a missing or misspelled variable in CI fails the required job instead of passing it with 26 skipped tests. +5. **Standard error is compared too, and `--output` runs share one path.** The CLI writes status, warnings, and `Saved to ` to stderr. Spectre wraps that line at 80 columns when output is redirected (verified), so replacing two different temp paths can't make stderr comparable. Both builds write to the same path one after the other, and the file is read and deleted in between. +6. **Reference outcomes are pinned.** Every CLI case asserts that the reference exits 0 (or non-zero for the two failure cases), and every MCP tool case asserts that the reference returns `isError != true`. Otherwise a wrong path that makes both builds fail identically would pass as parity. +7. **Extra cases.** PR 1's final review showed that only exe-launching tests catch reflection-off JSON bugs, and that parity needs edge inputs: + - CLI: a solution with a malformed project (`visualize` and `stats`), a `.sln` that lists the same project twice (fails on both builds), and a class diagram using `ConcurrentDictionary`/`Barrier`/`[Required]` (bound only through the embedded reference assemblies). + - MCP: `get_project_stats` on a solution with a malformed project (the `warnings` array), `prompts/get` (serializes `IEnumerable` through `McpJsonContext`), and server capabilities in the handshake. +8. **Parameter name.** The spec's MCP case says `options.inheritance`; the real parameter is `options.includeInheritance`. +9. **Shared MCP servers.** Disposing a stdio `McpClient` takes about 5 s in SDK 2.2.0 (the existing `McpTransportTests` pay this too; the servers themselves exit at once on stdin EOF). One native and one reference server are shared across the MCP test class. That brings the whole suite from 1 min 51 s to about 20 s, and it mirrors how clients hold one long session. +10. **Legacy `.sln` fixture.** Besides the five `modular-architecture` projects, it contains a solution folder, a `NestedProjects` section, and one classic C# project-type GUID, so the SolutionPersistence filtering path runs natively. + +## Verified facts this plan relies on + +All of these were checked on 2026-09-14 in a throwaway worktree at `86082ea` (linux-x64, SDK 10.0.401, no clang installed; ILC fell back to gcc): + +- Unmodified CLI, `dotnet publish -r linux-x64 -p:PublishAot=true`: fails with `Program.cs(26,19): error IL3050` (`CommandApp(ITypeRegistrar)` has `RequiresDynamicCodeAttribute`) and `DependencyInjection.cs(29,9): error IL2067` (`TypeRegistrar.Register` → `AddSingleton(IServiceCollection, Type, Type)`). `-p:IlcTreatWarningsAsErrors=false` doesn't help, because these come from the compile-time analyzers. +- CLI with both suppressions but **no** `TrimmerRootAssembly`: publishes, then every command crashes with `Unhandled exception. Spectre.Console.Cli.CommandRuntimeException: Could not get settings type for command of type 'ProjGraph.Cli.Commands.VisualizeCommand'.` and exit code 134. +- Unmodified MCP, AOT publish: fails with ILC errors `IL2104` (Microsoft.CodeAnalysis produced trim warnings) and `IL3000` (`CommonCompiler.GetAssemblyLocation`). With `IlcTreatWarningsAsErrors=false` it publishes and works (PR 1 already fixed the runtime blockers). +- The MCP project builds clean with `EnableTrimAnalyzer`/`EnableAotAnalyzer` on. The CLI reports only the two diagnostics above. +- With the configuration in Tasks 2–3: the CLI binary is ~36.6 MB and the MCP binary ~42.9 MB. All 26 smoke tests pass native vs JIT. The suite also passes JIT apphost vs JIT `.dll`, skips all 26 tests with no variables set, and fails when the "native" CLI is `/bin/true`. +- `stats` output differs between runs only in the `Analysis time` row. `stats ProjGraph.slnx --tpo 3` exits 255 on both builds and prints `Error: Unknown option 'tpo'.` to stdout. +- `artifacts/` is already in `.gitignore`; `out/` is not. + +## File Structure + +| File | Change | Responsibility | +| --- | --- | --- | +| `tests/ProjGraph.Tests.Smoke.Aot/ProjGraph.Tests.Smoke.Aot.csproj` | Create | Test project; references only `Tests.Shared` (no product projects) | +| `tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeCommand.cs` | Create | How to start a build: executable directly, `.dll` via `dotnet` | +| `tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeEnvironment.cs` | Create | Reads the `PROJGRAPH_SMOKE_*` variables; finds the repository root | +| `tests/ProjGraph.Tests.Smoke.Aot/Helpers/AotSmokeFactAttribute.cs` | Create | `[AotSmokeFact]`: skips unless configured (or required) | +| `tests/ProjGraph.Tests.Smoke.Aot/Helpers/ProcessRunner.cs` | Create | Runs a CLI build from the repo root, captures exit code/stdout/stderr | +| `tests/ProjGraph.Tests.Smoke.Aot/Helpers/McpServerPair.cs` | Create | Lazily started native + reference MCP clients shared by a test class | +| `tests/ProjGraph.Tests.Smoke.Aot/Fixtures/legacy.sln` | Create | Legacy `.sln` referencing the `modular-architecture` sample projects | +| `tests/ProjGraph.Tests.Smoke.Aot/CliParityTests.cs` | Create | 16 CLI parity cases | +| `tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs` | Create | 10 MCP parity cases | +| `ProjGraph.slnx` | Modify | Adds the smoke project | +| `src/ProjGraph.Cli/ProjGraph.Cli.csproj` | Modify | Analyzers, `IlcTreatWarningsAsErrors=false`, trimmer root assemblies | +| `src/ProjGraph.Cli/Program.cs` | Modify | IL3050 suppression on `Main` | +| `src/ProjGraph.Cli/Infrastructure/DependencyInjection.cs` | Modify | IL2067 suppression on `TypeRegistrar.Register` | +| `src/ProjGraph.Mcp/ProjGraph.Mcp.csproj` | Modify | Analyzers, `IlcTreatWarningsAsErrors=false` | +| `.github/workflows/ci.yml` | Modify | New `aot-smoke` job | +| `CLAUDE.md`, `CONTRIBUTING.md`, `ARCHITECTURE.md` | Modify | List the smoke project; explain how to run it locally | + +--- + +### Task 1: Smoke test harness and CLI parity cases + +The harness and the CLI cases, verified without any native build: the JIT apphost is compared with the JIT `.dll`. Those are two launch paths for the same code, so they must agree, and a deliberately different "native" binary must fail. + +**Files:** +- Create: `tests/ProjGraph.Tests.Smoke.Aot/ProjGraph.Tests.Smoke.Aot.csproj` +- Create: `tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeCommand.cs` +- Create: `tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeEnvironment.cs` +- Create: `tests/ProjGraph.Tests.Smoke.Aot/Helpers/AotSmokeFactAttribute.cs` +- Create: `tests/ProjGraph.Tests.Smoke.Aot/Helpers/ProcessRunner.cs` +- Create: `tests/ProjGraph.Tests.Smoke.Aot/Fixtures/legacy.sln` +- Create: `tests/ProjGraph.Tests.Smoke.Aot/CliParityTests.cs` +- Modify: `ProjGraph.slnx`, `CLAUDE.md`, `CONTRIBUTING.md`, `ARCHITECTURE.md` + +**Interfaces:** +- Consumes: `ProjGraph.Tests.Shared.Helpers.TestDirectory` (`DirectoryPath`, `CreateFile(string relativePath, string content)` which creates subdirectories and returns the full path, `IDisposable`). +- Produces (used by Task 3): + - `internal sealed record SmokeCommand(string FileName, IReadOnlyList LeadingArguments)` with `static SmokeCommand For(string path)`. + - `internal static class SmokeEnvironment` with `string RepositoryRoot`, `bool IsRequired`, `SmokeCommand CliNative`, `CliReference`, `McpNative`, `McpReference`, `IReadOnlyList GetMissingVariables()`, and `string GetRootPath(string relativePath)`. + - `public sealed class AotSmokeFactAttribute : FactAttribute`. + - `internal sealed record ProcessResult(int ExitCode, string StandardOutput, string StandardError)` and `internal static class ProcessRunner` with `Task RunAsync(SmokeCommand command, IReadOnlyList arguments)`. + +- [ ] **Step 1: Create the feature branch and commit this plan** + +```bash +git switch develop +git pull --ff-only +git switch -c feat/aot-native-smoke +git add docs/superpowers/plans/2026-09-14-native-aot-pr2-native-build-smoke.md +git commit -m "docs: add the Native AOT PR 2 implementation plan" +``` + +(If the plan file isn't in the working tree, it was already committed. Skip the `git add`/`commit`.) + +- [ ] **Step 2: Create the project file** + +Create `tests/ProjGraph.Tests.Smoke.Aot/ProjGraph.Tests.Smoke.Aot.csproj`: + +```xml + + + + true + + + + + + + + + + + + + + + + + + + + + + +``` + +Then add one line to `ProjGraph.slnx` by hand, between the `Tests.Shared` and `Tests.Unit.ClassDiagram` entries (keep the file's `"/>` style): + +```xml + + + +``` + +- [ ] **Step 3: Create the helpers** + +Create `tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeCommand.cs`: + +```csharp +namespace ProjGraph.Tests.Smoke.Aot.Helpers; + +/// +/// How to start one build of a tool: an executable is started directly, and a framework-dependent +/// .dll is started through the dotnet host. +/// +/// The program to start. +/// Arguments that precede the tool's own arguments. +internal sealed record SmokeCommand(string FileName, IReadOnlyList LeadingArguments) +{ + /// + /// Creates the command that starts the build at . + /// + /// The absolute path to an executable or a .dll. + /// The command. + public static SmokeCommand For(string path) + { + if (!path.EndsWith(".dll", StringComparison.OrdinalIgnoreCase)) + { + return new SmokeCommand(path, []); + } + + // dotnet test exports the host that runs the tests, so the reference build runs on the same SDK. + var host = Environment.GetEnvironmentVariable("DOTNET_HOST_PATH"); + return new SmokeCommand(string.IsNullOrEmpty(host) ? "dotnet" : host, [path]); + } +} +``` + +Create `tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeEnvironment.cs`: + +```csharp +namespace ProjGraph.Tests.Smoke.Aot.Helpers; + +/// +/// The smoke suite's configuration: the four builds under comparison, read from environment +/// variables, and the repository root that every command runs from. +/// +internal static class SmokeEnvironment +{ + private const string CliNativeVariable = "PROJGRAPH_SMOKE_CLI_NATIVE"; + private const string CliReferenceVariable = "PROJGRAPH_SMOKE_CLI_REFERENCE"; + private const string McpNativeVariable = "PROJGRAPH_SMOKE_MCP_NATIVE"; + private const string McpReferenceVariable = "PROJGRAPH_SMOKE_MCP_REFERENCE"; + private const string RequiredVariable = "PROJGRAPH_SMOKE_REQUIRED"; + + private static readonly string[] RequiredVariables = + [CliNativeVariable, CliReferenceVariable, McpNativeVariable, McpReferenceVariable]; + + private static readonly Lazy LazyRepositoryRoot = new(FindRepositoryRoot); + + /// + /// Gets the repository root: the nearest ancestor of the test output directory that contains + /// ProjGraph.slnx. + /// + public static string RepositoryRoot => LazyRepositoryRoot.Value; + + /// + /// Gets a value indicating whether the suite must run. CI sets PROJGRAPH_SMOKE_REQUIRED=true, + /// so a misspelled or missing variable fails the job instead of skipping every test. + /// + public static bool IsRequired => + string.Equals(Environment.GetEnvironmentVariable(RequiredVariable), "true", StringComparison.OrdinalIgnoreCase); + + /// Gets the native CLI executable. + public static SmokeCommand CliNative => Resolve(CliNativeVariable); + + /// Gets the JIT CLI build of the same commit. + public static SmokeCommand CliReference => Resolve(CliReferenceVariable); + + /// Gets the native MCP server executable. + public static SmokeCommand McpNative => Resolve(McpNativeVariable); + + /// Gets the JIT MCP server build of the same commit. + public static SmokeCommand McpReference => Resolve(McpReferenceVariable); + + /// + /// Lists the required variables that are unset or empty. + /// + /// The missing variable names, empty when the suite is fully configured. + public static IReadOnlyList GetMissingVariables() + { + return + [ + .. RequiredVariables.Where(name => string.IsNullOrWhiteSpace(Environment.GetEnvironmentVariable(name))) + ]; + } + + /// + /// Resolves an absolute path under the repository root. + /// + /// A path relative to the repository root, using / separators. + /// The absolute path. + public static string GetRootPath(string relativePath) + { + return Path.GetFullPath(Path.Combine(RepositoryRoot, relativePath)); + } + + private static SmokeCommand Resolve(string variable) + { + var value = Environment.GetEnvironmentVariable(variable); + if (string.IsNullOrWhiteSpace(value)) + { + throw new InvalidOperationException($"{variable} is not set."); + } + + // A configured path that does not exist is a broken CI setup, never a reason to skip. + var path = Path.GetFullPath(value, RepositoryRoot); + return File.Exists(path) + ? SmokeCommand.For(path) + : throw new FileNotFoundException($"{variable} points to a file that does not exist.", path); + } + + private static string FindRepositoryRoot() + { + for (var directory = new DirectoryInfo(AppContext.BaseDirectory); + directory is not null; + directory = directory.Parent) + { + if (File.Exists(Path.Combine(directory.FullName, "ProjGraph.slnx"))) + { + return directory.FullName; + } + } + + throw new InvalidOperationException($"No ProjGraph.slnx found above {AppContext.BaseDirectory}."); + } +} +``` + +Create `tests/ProjGraph.Tests.Smoke.Aot/Helpers/AotSmokeFactAttribute.cs` (the explicit `AttributeUsage` is required by Sonar S3993): + +```csharp +namespace ProjGraph.Tests.Smoke.Aot.Helpers; + +/// +/// A fact that runs only when all four PROJGRAPH_SMOKE_* variables are set, so a plain +/// dotnet test ProjGraph.slnx reports the smoke suite as skipped instead of failing it. When +/// PROJGRAPH_SMOKE_REQUIRED=true, nothing is skipped and a missing variable fails the test. +/// +[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)] +public sealed class AotSmokeFactAttribute : FactAttribute +{ + /// + /// Initializes a new instance of the class. + /// + public AotSmokeFactAttribute() + { + var missing = SmokeEnvironment.GetMissingVariables(); + if (missing.Count > 0 && !SmokeEnvironment.IsRequired) + { + Skip = $"Native AOT smoke suite not configured; set {string.Join(", ", missing)}."; + } + } +} +``` + +Create `tests/ProjGraph.Tests.Smoke.Aot/Helpers/ProcessRunner.cs`: + +```csharp +using System.Diagnostics; +using System.Text; + +namespace ProjGraph.Tests.Smoke.Aot.Helpers; + +/// +/// The observable result of one CLI invocation. +/// +/// The process exit code. +/// Everything written to standard output. +/// Everything written to standard error. +internal sealed record ProcessResult(int ExitCode, string StandardOutput, string StandardError); + +/// +/// Runs a build of the CLI from the repository root and captures its output. +/// +internal static class ProcessRunner +{ + private static readonly TimeSpan Timeout = TimeSpan.FromMinutes(3); + + /// + /// Runs with and waits for it to exit. + /// + /// The build to run. + /// The CLI arguments. + /// The exit code and the captured output streams. + /// Thrown when the process cannot be started. + /// Thrown when the process does not exit in time; it is killed. + public static async Task RunAsync(SmokeCommand command, IReadOnlyList arguments) + { + var startInfo = new ProcessStartInfo(command.FileName) + { + WorkingDirectory = SmokeEnvironment.RepositoryRoot, + RedirectStandardOutput = true, + RedirectStandardError = true, + StandardOutputEncoding = Encoding.UTF8, + StandardErrorEncoding = Encoding.UTF8, + UseShellExecute = false + }; + foreach (var argument in command.LeadingArguments.Concat(arguments)) + { + startInfo.ArgumentList.Add(argument); + } + + using var process = Process.Start(startInfo) + ?? throw new InvalidOperationException($"Could not start {command.FileName}."); + using var timeout = new CancellationTokenSource(Timeout); + + // Both streams are drained concurrently so a full pipe buffer can never block the child. + var standardOutput = process.StandardOutput.ReadToEndAsync(timeout.Token); + var standardError = process.StandardError.ReadToEndAsync(timeout.Token); + try + { + await process.WaitForExitAsync(timeout.Token); + } + catch (OperationCanceledException) + { + process.Kill(entireProcessTree: true); + throw new TimeoutException( + $"'{command.FileName} {string.Join(' ', arguments)}' did not exit within {Timeout}."); + } + + return new ProcessResult(process.ExitCode, await standardOutput, await standardError); + } +} +``` + +- [ ] **Step 4: Create the legacy `.sln` fixture** + +Create `tests/ProjGraph.Tests.Smoke.Aot/Fixtures/legacy.sln` with exactly this content (tabs inside `Global`; `.gitattributes` normalizes `*.sln` line endings). The project paths are relative to the fixture directory: + +```text +Microsoft Visual Studio Solution File, Format Version 12.00 +# Visual Studio Version 17 +VisualStudioVersion = 17.0.31903.59 +MinimumVisualStudioVersion = 10.0.40219.1 +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "modular-architecture", "modular-architecture", "{0A5E0000-0000-4000-8000-000000000001}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "App", "..\..\..\samples\visualize\modular-architecture\App\App.csproj", "{0A5E0000-0000-4000-8000-000000000002}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Core", "..\..\..\samples\visualize\modular-architecture\Core\Core.csproj", "{0A5E0000-0000-4000-8000-000000000003}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Infrastructure", "..\..\..\samples\visualize\modular-architecture\Infrastructure\Infrastructure.csproj", "{0A5E0000-0000-4000-8000-000000000004}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Services", "..\..\..\samples\visualize\modular-architecture\Services\Services.csproj", "{0A5E0000-0000-4000-8000-000000000005}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Shared", "..\..\..\samples\visualize\modular-architecture\Shared\Shared.csproj", "{0A5E0000-0000-4000-8000-000000000006}" +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|Any CPU = Debug|Any CPU + Release|Any CPU = Release|Any CPU + EndGlobalSection + GlobalSection(NestedProjects) = preSolution + {0A5E0000-0000-4000-8000-000000000002} = {0A5E0000-0000-4000-8000-000000000001} + {0A5E0000-0000-4000-8000-000000000003} = {0A5E0000-0000-4000-8000-000000000001} + EndGlobalSection +EndGlobal +``` + +Sanity check after the build in Step 6: `dotnet src/ProjGraph.Cli/bin/Release/net10.0/ProjGraph.Cli.dll visualize tests/ProjGraph.Tests.Smoke.Aot/Fixtures/legacy.sln --format mermaid` exits 0 and prints a graph titled `legacy.sln` with the five nodes `App`, `Core`, `Infrastructure`, `Services`, `Shared`. + +- [ ] **Step 5: Write the CLI parity cases** + +Create `tests/ProjGraph.Tests.Smoke.Aot/CliParityTests.cs`: + +```csharp +using ProjGraph.Tests.Shared.Helpers; +using ProjGraph.Tests.Smoke.Aot.Helpers; +using System.Text.RegularExpressions; + +namespace ProjGraph.Tests.Smoke.Aot; + +/// +/// Runs CLI commands through the native executable and through the JIT build of the same commit, +/// and requires the same exit code, standard output, standard error, and written diagram file. +/// +public sealed partial class CliParityTests : IDisposable +{ + private const string ModularSolution = "samples/visualize/modular-architecture/ModularArchitecture.slnx"; + + private readonly TestDirectory _temp = new(); + + public void Dispose() + { + _temp.Dispose(); + } + + [AotSmokeFact] + public async Task Visualize_Mermaid_ShouldMatchReference() + { + await AssertDiagramParityAsync("visualize", ModularSolution, "--format", "mermaid"); + } + + [AotSmokeFact] + public async Task Visualize_Tree_ShouldMatchReference() + { + await AssertDiagramParityAsync("visualize", ModularSolution, "--format", "tree"); + } + + [AotSmokeFact] + public async Task Visualize_Flat_ShouldMatchReference() + { + await AssertDiagramParityAsync("visualize", ModularSolution, "--format", "flat"); + } + + [AotSmokeFact] + public async Task Visualize_LegacySln_ShouldMatchReference() + { + await AssertDiagramParityAsync( + "visualize", "tests/ProjGraph.Tests.Smoke.Aot/Fixtures/legacy.sln", "--format", "mermaid"); + } + + [AotSmokeFact] + public async Task Erd_ComplexEcommerce_ShouldMatchReference() + { + await AssertDiagramParityAsync("erd", "samples/erd/complex-ecommerce/Data/MyDbContext.cs"); + } + + [AotSmokeFact] + public async Task Erd_SimpleContext_ShouldMatchReference() + { + await AssertDiagramParityAsync("erd", "samples/erd/simple-context/EntityFramework/MyDbContext.cs"); + } + + [AotSmokeFact] + public async Task ClassDiagram_DesignPatterns_ShouldMatchReference() + { + await AssertDiagramParityAsync( + "classdiagram", "samples/classdiagram/design-patterns/Domain/Order.cs", + "--inheritance", "--dependencies", "--depth", "2", "--properties", "true", "--functions", "true"); + } + + [AotSmokeFact] + public async Task ClassDiagram_ComplexHierarchy_ShouldMatchReference() + { + await AssertDiagramParityAsync( + "classdiagram", "samples/classdiagram/complex-hierarchy/Domain/Models/CEO.cs", + "--inheritance", "--dependencies", "--depth", "5", "--properties", "true", "--functions", "true"); + } + + [AotSmokeFact] + public async Task ClassDiagram_SimpleHierarchy_ShouldMatchReference() + { + await AssertDiagramParityAsync( + "classdiagram", "samples/classdiagram/simple-hierarchy/Models/Admin.cs", + "--inheritance", "--dependencies", "--depth", "2", "--properties", "true", "--functions", "true"); + } + + [AotSmokeFact] + public async Task ClassDiagram_SourceDirectory_ShouldMatchReference() + { + await AssertDiagramParityAsync("classdiagram", "src", "--inheritance", "--dependencies"); + } + + [AotSmokeFact] + public async Task ClassDiagram_ConcurrencyAndAnnotationTypes_ShouldMatchReference() + { + // BCL types that bind only through the embedded reference assemblies (spec §1.5). + _temp.CreateFile("Types/Types.csproj", + "net10.0"); + var file = _temp.CreateFile("Types/Inventory.cs", + """ + using System.Collections.Concurrent; + using System.ComponentModel.DataAnnotations; + + namespace Types; + + public class Inventory : Dictionary, IDisposable + { + [Required] + public string Name { get; set; } = ""; + + public ConcurrentDictionary Items { get; } = new(); + + public Barrier? Gate { get; set; } + + public IEnumerable Available() => Items.Values; + + public void Dispose() => Gate?.Dispose(); + } + + public record StockItem(string Sku, decimal Price); + """); + + await AssertDiagramParityAsync("classdiagram", file, "--inheritance", "--dependencies"); + } + + [AotSmokeFact] + public async Task Visualize_SolutionWithMalformedProject_ShouldMatchReference() + { + var solution = CreateSolutionWithMalformedProject(); + + await AssertDiagramParityAsync("visualize", solution, "--format", "mermaid"); + } + + [AotSmokeFact] + public async Task Visualize_SlnListingAProjectTwice_ShouldFailLikeReference() + { + _temp.CreateFile("Good/Good.csproj", + "net10.0"); + var solution = _temp.CreateFile("Duplicate.sln", + """ + Microsoft Visual Studio Solution File, Format Version 12.00 + Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Good", "Good\Good.csproj", "{0A5E0000-0000-4000-8000-000000000001}" + EndProject + Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "GoodAgain", "Good\Good.csproj", "{0A5E0000-0000-4000-8000-000000000002}" + EndProject + """); + + await AssertParityAsync(expectSuccess: false, "visualize", solution, "--format", "mermaid"); + } + + [AotSmokeFact] + public async Task Stats_Repository_ShouldMatchReferenceApartFromTiming() + { + await AssertStatsParityAsync("stats", "ProjGraph.slnx"); + } + + [AotSmokeFact] + public async Task Stats_SolutionWithMalformedProject_ShouldMatchReferenceApartFromTiming() + { + var solution = CreateSolutionWithMalformedProject(); + + await AssertStatsParityAsync("stats", solution); + } + + [AotSmokeFact] + public async Task Stats_UnknownOption_ShouldFailLikeReference() + { + // Strict parsing rejects the typo before the command runs (Spectre's error rendering path). + await AssertParityAsync(expectSuccess: false, "stats", "ProjGraph.slnx", "--tpo", "3"); + } + + private string CreateSolutionWithMalformedProject() + { + _temp.CreateFile("Good/Good.csproj", + "net10.0"); + _temp.CreateFile("Bad/Bad.csproj", ""); + return _temp.CreateFile("Malformed.slnx", + ""); + } + + private static async Task AssertParityAsync(bool expectSuccess, params string[] arguments) + { + var (native, reference) = await RunBothAsync(arguments); + + AssertReferenceOutcome(reference, expectSuccess); + AssertSameResult(native, reference); + } + + /// + /// Compares a diagram command twice: printed to the console, then written with --output. + /// Both builds write to the same path one after the other, because the "Saved to" line on + /// standard error names the path and Spectre wraps it at the console width, so two different + /// paths could not be compared exactly. + /// + /// The CLI arguments, without --output. + private async Task AssertDiagramParityAsync(params string[] arguments) + { + await AssertParityAsync(expectSuccess: true, arguments); + + var outputPath = Path.Combine(_temp.DirectoryPath, "output", "diagram.mmd"); + string[] withOutput = [.. arguments, "--output", outputPath]; + + var native = await ProcessRunner.RunAsync(SmokeEnvironment.CliNative, withOutput); + File.Exists(outputPath).Should().BeTrue($"the native build must write the diagram; stderr:\n{native.StandardError}"); + var nativeFile = await File.ReadAllBytesAsync(outputPath); + File.Delete(outputPath); + + var reference = await ProcessRunner.RunAsync(SmokeEnvironment.CliReference, withOutput); + AssertReferenceOutcome(reference, expectSuccess: true); + var referenceFile = await File.ReadAllBytesAsync(outputPath); + + AssertSameResult(native, reference); + nativeFile.Should().Equal(referenceFile, "the native build must write the same diagram bytes"); + } + + private static async Task AssertStatsParityAsync(params string[] arguments) + { + var (native, reference) = await RunBothAsync(arguments); + + AssertReferenceOutcome(reference, expectSuccess: true); + native.ExitCode.Should().Be(reference.ExitCode); + NormalizeStats(native.StandardOutput).Should().Be(NormalizeStats(reference.StandardOutput)); + native.StandardError.Should().Be(reference.StandardError); + } + + private static async Task<(ProcessResult Native, ProcessResult Reference)> RunBothAsync(string[] arguments) + { + var native = await ProcessRunner.RunAsync(SmokeEnvironment.CliNative, arguments); + var reference = await ProcessRunner.RunAsync(SmokeEnvironment.CliReference, arguments); + return (native, reference); + } + + /// + /// Pins the reference outcome, so a case where both builds fail the same way (a wrong path, a + /// missing sample) cannot pass as parity. + /// + /// The reference build's result. + /// Whether the reference build must exit with 0. + private static void AssertReferenceOutcome(ProcessResult reference, bool expectSuccess) + { + if (expectSuccess) + { + reference.ExitCode.Should().Be(0, $"the reference build must succeed; stderr:\n{reference.StandardError}"); + } + else + { + reference.ExitCode.Should().NotBe(0, "the reference build must reject this input"); + } + } + + private static void AssertSameResult(ProcessResult native, ProcessResult reference) + { + native.ExitCode.Should().Be(reference.ExitCode, $"native stderr:\n{native.StandardError}"); + native.StandardOutput.Should().Be(reference.StandardOutput); + native.StandardError.Should().Be(reference.StandardError); + } + + /// + /// Drops the wall-clock "Analysis time" row and collapses runs of spaces, because the timing + /// value's width can shift the table's padding. + /// + /// The captured standard output of stats. + /// The output without timing, for comparison. + private static string NormalizeStats(string output) + { + return string.Join('\n', output.ReplaceLineEndings("\n") + .Split('\n') + .Where(line => !line.TrimStart().StartsWith("Analysis time", StringComparison.Ordinal)) + .Select(line => RunOfSpaces().Replace(line, " "))); + } + + [GeneratedRegex(" {2,}")] + private static partial Regex RunOfSpaces(); +} +``` + +- [ ] **Step 6: Build and confirm the suite skips by default** + +```bash +dotnet build ProjGraph.slnx -c Release +dotnet test tests/ProjGraph.Tests.Smoke.Aot -c Release --no-build +``` + +Expected: build succeeds with 0 warnings. Tests report `Skipped: 16, Total: 16` (no failures). + +- [ ] **Step 7: Verify the harness JIT apphost vs JIT `.dll` (must pass)** + +The MCP variables must be set because `[AotSmokeFact]` needs all four. No MCP tests exist yet. + +```bash +export PROJGRAPH_SMOKE_CLI_NATIVE=src/ProjGraph.Cli/bin/Release/net10.0/ProjGraph.Cli +export PROJGRAPH_SMOKE_CLI_REFERENCE=src/ProjGraph.Cli/bin/Release/net10.0/ProjGraph.Cli.dll +export PROJGRAPH_SMOKE_MCP_NATIVE=src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp +export PROJGRAPH_SMOKE_MCP_REFERENCE=src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp.dll +dotnet test tests/ProjGraph.Tests.Smoke.Aot -c Release --no-build +``` + +Expected: `Passed: 16`. (On Windows the apphost is `ProjGraph.Cli.exe` and the MCP RID directory is `win-x64`.) + +- [ ] **Step 8: Verify the harness detects divergence and a missing variable (must fail)** + +```bash +PROJGRAPH_SMOKE_CLI_NATIVE=/bin/true dotnet test tests/ProjGraph.Tests.Smoke.Aot -c Release --no-build --filter "FullyQualifiedName~Visualize_Mermaid" +``` + +Expected: `Failed: 1`, with `Expected native.StandardOutput to be "```mermaid` in the message. + +```bash +env -u PROJGRAPH_SMOKE_CLI_NATIVE PROJGRAPH_SMOKE_REQUIRED=true dotnet test tests/ProjGraph.Tests.Smoke.Aot -c Release --no-build --filter "FullyQualifiedName~Visualize_Mermaid" +``` + +Expected: `Failed: 1` with `System.InvalidOperationException : PROJGRAPH_SMOKE_CLI_NATIVE is not set.` Then `unset` the four variables. + +- [ ] **Step 9: Document the project** + +In `CLAUDE.md`, add a row to the "Test projects" table after `Tests.Contract`: + +```markdown +| `Tests.Smoke.Aot` | Native AOT vs JIT parity for the CLI and MCP executables (skipped unless `PROJGRAPH_SMOKE_*` is set; runs in the `aot-smoke` CI job) | +``` + +In `CONTRIBUTING.md`, add the same row to the "Test Organisation" table after `Tests.Contract`. Keep the table's padded column alignment: widen every row of the table so the columns line up, because markdownlint checks it. + +In `ARCHITECTURE.md`, add a line to the `tests/` tree after `ProjGraph.Tests.Contract`: + +```text +│ ├── ProjGraph.Tests.Smoke.Aot # Native AOT vs JIT parity (CI aot-smoke job) +``` + +- [ ] **Step 10: Format check and commit** + +```bash +dotnet format ProjGraph.slnx --verify-no-changes +git add ProjGraph.slnx tests/ProjGraph.Tests.Smoke.Aot CLAUDE.md CONTRIBUTING.md ARCHITECTURE.md +git commit -m "test: add the Native AOT smoke suite with CLI parity cases" +``` + +--- + +### Task 2: CLI native publish configuration + +**Files:** +- Modify: `src/ProjGraph.Cli/ProjGraph.Cli.csproj` +- Modify: `src/ProjGraph.Cli/Program.cs` +- Modify: `src/ProjGraph.Cli/Infrastructure/DependencyInjection.cs` + +**Interfaces:** +- Consumes: the Task 1 suite and its four variables. +- Produces: `dotnet publish src/ProjGraph.Cli -c Release -r linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build -o artifacts/native/cli` yields a working `artifacts/native/cli/ProjGraph.Cli`. Task 4's CI job runs this exact command. + +- [ ] **Step 1: Confirm the native publish fails today** + +```bash +dotnet publish src/ProjGraph.Cli -c Release -r linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build -o artifacts/native/cli +``` + +Expected: FAIL with `Program.cs(26,19): error IL3050: Using member 'Spectre.Console.Cli.CommandApp.CommandApp(ITypeRegistrar)' which has 'RequiresDynamicCodeAttribute'…` and `DependencyInjection.cs(29,9): error IL2067: 'implementationType' argument does not satisfy 'DynamicallyAccessedMemberTypes.PublicConstructors'…`. + +- [ ] **Step 2: Turn on the analyzers at build time** + +In `src/ProjGraph.Cli/ProjGraph.Cli.csproj`, after the `` line in the first `PropertyGroup`, add: + +```xml + + true + true + false +``` + +Run: `dotnet build src/ProjGraph.Cli -c Release` +Expected: FAIL with the same IL3050 and IL2067 errors. A plain build now catches them too. + +- [ ] **Step 3: Add the two suppressions** + +In `src/ProjGraph.Cli/Program.cs`, add `using System.Diagnostics.CodeAnalysis;` after `using Spectre.Console.Cli;`, and put the attribute on `Main`: + +```csharp + [UnconditionalSuppressMessage("AOT", "IL3050:RequiresDynamicCode", + Justification = "Spectre.Console.Cli reflects over the command and settings types. ProjGraph.Cli and " + + "Spectre.Console.Cli are trimmer root assemblies, so that metadata is kept, and " + + "tests/ProjGraph.Tests.Smoke.Aot runs every command natively.")] + public static int Main(string[] args) +``` + +In `src/ProjGraph.Cli/Infrastructure/DependencyInjection.cs`, add `using System.Diagnostics.CodeAnalysis;` after `using Spectre.Console.Cli;`, and put the attribute on `TypeRegistrar.Register(Type, Type)` (after its XML doc comment): + +```csharp + [UnconditionalSuppressMessage("Trimming", "IL2067:DynamicallyAccessedMembers", + Justification = "Spectre.Console.Cli registers the command and settings types it discovers. They live in " + + "ProjGraph.Cli and Spectre.Console.Cli, which are trimmer root assemblies, so their " + + "public constructors are kept; tests/ProjGraph.Tests.Smoke.Aot covers every command.")] + public void Register(Type service, Type implementation) +``` + +Run: `dotnet build src/ProjGraph.Cli -c Release` +Expected: build succeeds, 0 warnings. + +- [ ] **Step 4: Publish and confirm the smoke suite catches the Spectre crash** + +```bash +dotnet publish src/ProjGraph.Cli -c Release -r linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build -o artifacts/native/cli +artifacts/native/cli/ProjGraph.Cli stats ProjGraph.slnx; echo "exit=$?" +``` + +Expected: publish succeeds, with third-party warnings only (`IL2104` for Spectre.Console.Cli and Microsoft.CodeAnalysis, `IL3053`, `IL3000`). The run crashes with `Unhandled exception. Spectre.Console.Cli.CommandRuntimeException: Could not get settings type for command of type 'ProjGraph.Cli.Commands.VisualizeCommand'.` and `exit=134`. + +```bash +dotnet build ProjGraph.slnx -c Release +export PROJGRAPH_SMOKE_CLI_NATIVE=artifacts/native/cli/ProjGraph.Cli +export PROJGRAPH_SMOKE_CLI_REFERENCE=src/ProjGraph.Cli/bin/Release/net10.0/ProjGraph.Cli.dll +export PROJGRAPH_SMOKE_MCP_NATIVE=src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp +export PROJGRAPH_SMOKE_MCP_REFERENCE=src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp.dll +dotnet test tests/ProjGraph.Tests.Smoke.Aot -c Release --no-build +``` + +Expected: FAIL, all 16 CLI cases (exit code 134 vs the reference's). + +- [ ] **Step 5: Root the reflected assemblies** + +In `src/ProjGraph.Cli/ProjGraph.Cli.csproj`, after the `Spectre.Console.Cli` `PackageReference` `ItemGroup`, add (items, not `-p:` properties; the spike showed properties don't work): + +```xml + + + + + +``` + +- [ ] **Step 6: Republish and run the suite** + +```bash +dotnet publish src/ProjGraph.Cli -c Release -r linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build -o artifacts/native/cli +dotnet test tests/ProjGraph.Tests.Smoke.Aot -c Release --no-build +``` + +Expected: `Passed: 16`. The binary is about 36.6 MB. + +- [ ] **Step 7: Confirm the JIT CLI is unchanged** + +```bash +unset PROJGRAPH_SMOKE_CLI_NATIVE PROJGRAPH_SMOKE_CLI_REFERENCE PROJGRAPH_SMOKE_MCP_NATIVE PROJGRAPH_SMOKE_MCP_REFERENCE +dotnet build ProjGraph.slnx -c Release +dotnet test tests/ProjGraph.Tests.Integration.Cli -c Release --no-build +git grep -n "PublishAot" -- '*.csproj' '*.props' +``` + +Expected: build 0 warnings, `Tests.Integration.Cli` all pass, and `git grep` prints nothing. + +- [ ] **Step 8: Commit** + +```bash +dotnet format ProjGraph.slnx --verify-no-changes +git add src/ProjGraph.Cli +git commit -m "feat(cli): make the CLI publishable as Native AOT" +``` + +--- + +### Task 3: MCP parity cases and MCP native publish configuration + +**Files:** +- Create: `tests/ProjGraph.Tests.Smoke.Aot/Helpers/McpServerPair.cs` +- Create: `tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs` +- Modify: `src/ProjGraph.Mcp/ProjGraph.Mcp.csproj` + +**Interfaces:** +- Consumes: `SmokeEnvironment` (`McpNative`, `McpReference`, `RepositoryRoot`, `GetRootPath`), `SmokeCommand`, `AotSmokeFactAttribute`, `TestDirectory` (Task 1). From ModelContextProtocol 2.2.0: `StdioClientTransport`, `StdioClientTransportOptions` (`Name`, `Command`, `Arguments`, `WorkingDirectory`), `McpClient.CreateAsync`, `McpClient.ServerInfo`, `ServerCapabilities`, `ListToolsAsync(ListToolsRequestParams, CancellationToken)`, `ListPromptsAsync(ListPromptsRequestParams, CancellationToken)`, `GetPromptAsync(string, IReadOnlyDictionary, …)`, `ReadResourceAsync(Uri, …)`, `CallToolAsync(string, IReadOnlyDictionary, …)`, and `McpJsonUtilities.DefaultOptions`. +- Produces: `public sealed class McpServerPair : IAsyncLifetime` with `Task<(McpClient Native, McpClient Reference)> GetClientsAsync()`. `dotnet publish src/ProjGraph.Mcp -c Release -r linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build -o artifacts/native/mcp` yields a working `artifacts/native/mcp/ProjGraph.Mcp`. + +- [ ] **Step 1: Create the shared server pair** + +Create `tests/ProjGraph.Tests.Smoke.Aot/Helpers/McpServerPair.cs`: + +```csharp +using ModelContextProtocol.Client; + +namespace ProjGraph.Tests.Smoke.Aot.Helpers; + +/// +/// One native and one reference MCP server, shared by every test in a class. Stopping a stdio +/// client takes seconds, and one long-lived session per build is also how clients use the server. +/// The servers start on first use, so a skipped suite never launches them. +/// +public sealed class McpServerPair : IAsyncLifetime +{ + private McpClient? _native; + private McpClient? _reference; + + /// + /// Connects to both servers on the first call and returns the same clients afterwards. + /// + /// The native and the reference client. + public async Task<(McpClient Native, McpClient Reference)> GetClientsAsync() + { + _native ??= await ConnectAsync(SmokeEnvironment.McpNative); + _reference ??= await ConnectAsync(SmokeEnvironment.McpReference); + return (_native, _reference); + } + + /// + public Task InitializeAsync() + { + return Task.CompletedTask; + } + + /// + public async Task DisposeAsync() + { + if (_native is not null) + { + await _native.DisposeAsync(); + } + + if (_reference is not null) + { + await _reference.DisposeAsync(); + } + } + + private static async Task ConnectAsync(SmokeCommand command) + { + var transport = new StdioClientTransport(new StdioClientTransportOptions + { + Name = "ProjGraph AOT smoke", + Command = command.FileName, + Arguments = [.. command.LeadingArguments], + WorkingDirectory = SmokeEnvironment.RepositoryRoot + }); + + return await McpClient.CreateAsync(transport); + } +} +``` + +- [ ] **Step 2: Write the MCP parity cases** + +Create `tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs` (`ReadResourceAsync` takes a `Uri` because CA2234 rejects the string overload): + +```csharp +using ModelContextProtocol; +using ModelContextProtocol.Client; +using ModelContextProtocol.Protocol; +using ProjGraph.Tests.Shared.Helpers; +using ProjGraph.Tests.Smoke.Aot.Helpers; +using System.Text.Json; +using System.Text.Json.Nodes; + +namespace ProjGraph.Tests.Smoke.Aot; + +/// +/// Starts the native MCP server and the JIT build of the same commit over stdio, sends both the +/// same requests, and requires identical protocol results. +/// +/// The native and reference servers shared by this class. +public sealed class McpParityTests(McpServerPair servers) : IClassFixture +{ + [AotSmokeFact] + public async Task Initialize_ShouldAdvertiseTheSameServer() + { + var (native, reference) = await servers.GetClientsAsync(); + + reference.ServerInfo.Name.Should().Be("ProjGraph"); + AssertSameJson(native.ServerInfo, reference.ServerInfo); + AssertSameJson(native.ServerCapabilities, reference.ServerCapabilities); + } + + [AotSmokeFact] + public async Task ListTools_ShouldMatchReference() + { + await AssertSameResultAsync(client => client.ListToolsAsync(new ListToolsRequestParams())); + } + + [AotSmokeFact] + public async Task ListPrompts_ShouldMatchReference() + { + await AssertSameResultAsync(client => client.ListPromptsAsync(new ListPromptsRequestParams())); + } + + [AotSmokeFact] + public async Task GetPrompt_ShouldMatchReference() + { + // Prompt results serialize IEnumerable through McpJsonContext. + var path = SmokeEnvironment.GetRootPath("samples/classdiagram/complex-hierarchy/Domain/Models/CEO.cs"); + + await AssertSameResultAsync(client => client.GetPromptAsync( + "class_structure_review", new Dictionary { ["path"] = path })); + } + + [AotSmokeFact] + public async Task ReadWelcomeResource_ShouldMatchReference() + { + await AssertSameResultAsync(client => client.ReadResourceAsync(new Uri("projgraph://welcome"))); + } + + [AotSmokeFact] + public async Task GetProjectGraph_ShouldMatchReference() + { + await AssertSameToolResultAsync("get_project_graph", new Dictionary + { + ["path"] = SmokeEnvironment.GetRootPath("samples/visualize/modular-architecture/ModularArchitecture.slnx") + }); + } + + [AotSmokeFact] + public async Task GetProjectStats_ShouldMatchReference() + { + await AssertSameToolResultAsync("get_project_stats", new Dictionary + { + ["path"] = SmokeEnvironment.GetRootPath("samples/visualize/modular-architecture/ModularArchitecture.slnx") + }); + } + + [AotSmokeFact] + public async Task GetProjectStats_SolutionWithMalformedProject_ShouldMatchReference() + { + // The warnings array is built with JsonNode APIs that fail without reflection-based JSON. + using var temp = new TestDirectory(); + temp.CreateFile("Good/Good.csproj", + "net10.0"); + temp.CreateFile("Bad/Bad.csproj", ""); + var solution = temp.CreateFile("Malformed.slnx", + ""); + + var reference = await AssertSameToolResultAsync( + "get_project_stats", new Dictionary { ["path"] = solution }); + + var warnings = JsonNode.Parse(JoinText(reference))?["warnings"]?.AsArray(); + warnings.Should().NotBeNullOrEmpty("the malformed project must surface as a warning"); + } + + [AotSmokeFact] + public async Task GetClassDiagram_ShouldMatchReference() + { + await AssertSameToolResultAsync("get_class_diagram", new Dictionary + { + ["path"] = SmokeEnvironment.GetRootPath("samples/classdiagram/complex-hierarchy/Domain/Models/CEO.cs"), + ["options"] = new Dictionary { ["includeInheritance"] = true } + }); + } + + [AotSmokeFact] + public async Task GetErd_ShouldMatchReference() + { + await AssertSameToolResultAsync("get_erd", new Dictionary + { + ["path"] = SmokeEnvironment.GetRootPath("samples/erd/complex-ecommerce/Data/MyDbContext.cs") + }); + } + + private async Task AssertSameResultAsync(Func> request) + { + var (native, reference) = await servers.GetClientsAsync(); + + var referenceResult = await request(reference); + AssertSameJson(await request(native), referenceResult); + return referenceResult; + } + + private async Task AssertSameToolResultAsync( + string toolName, Dictionary arguments) + { + var reference = await AssertSameResultAsync(client => client.CallToolAsync(toolName, arguments)); + + // Pins the reference outcome, so an error both builds return identically cannot pass as parity. + reference.IsError.Should().NotBe(true, JoinText(reference)); + return reference; + } + + private static void AssertSameJson(T native, T reference) + { + var nativeJson = JsonSerializer.SerializeToNode(native, McpJsonUtilities.DefaultOptions); + var referenceJson = JsonSerializer.SerializeToNode(reference, McpJsonUtilities.DefaultOptions); + + JsonNode.DeepEquals(nativeJson, referenceJson).Should().BeTrue( + $"the native server must return the reference result.\nNative:\n{nativeJson}\nReference:\n{referenceJson}"); + } + + private static string JoinText(CallToolResult result) + { + return string.Join("\n", result.Content.OfType().Select(block => block.Text)); + } +} +``` + +- [ ] **Step 3: Verify the MCP cases JIT apphost vs JIT `.dll` (must pass)** + +```bash +dotnet build ProjGraph.slnx -c Release +export PROJGRAPH_SMOKE_CLI_NATIVE=src/ProjGraph.Cli/bin/Release/net10.0/ProjGraph.Cli +export PROJGRAPH_SMOKE_CLI_REFERENCE=src/ProjGraph.Cli/bin/Release/net10.0/ProjGraph.Cli.dll +export PROJGRAPH_SMOKE_MCP_NATIVE=src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp +export PROJGRAPH_SMOKE_MCP_REFERENCE=src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp.dll +dotnet test tests/ProjGraph.Tests.Smoke.Aot -c Release --no-build --filter "FullyQualifiedName~McpParityTests" +``` + +Expected: `Passed: 10`, in seconds rather than ~10 s per test (the shared pair costs ~10 s once, at class teardown). + +- [ ] **Step 4: Confirm the MCP native publish fails today** + +```bash +dotnet publish src/ProjGraph.Mcp -c Release -r linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build -o artifacts/native/mcp +``` + +Expected: FAIL with ILC errors including `Microsoft.CodeAnalysis.dll : error IL2104: Assembly 'Microsoft.CodeAnalysis' produced trim warnings` and `CommonCompiler.cs(175): error IL3000: …'System.Reflection.Assembly.Location.get' always returns an empty string…`. + +- [ ] **Step 5: Configure the MCP project** + +In `src/ProjGraph.Mcp/ProjGraph.Mcp.csproj`, after `true` (leave that line unchanged; see deviation 1), add: + +```xml + + + true + true + false +``` + +Run: `dotnet build src/ProjGraph.Mcp -c Release` +Expected: build succeeds, 0 warnings. + +- [ ] **Step 6: Publish both executables and run the whole suite natively** + +Publish one at a time: + +```bash +dotnet publish src/ProjGraph.Cli -c Release -r linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build -o artifacts/native/cli +dotnet publish src/ProjGraph.Mcp -c Release -r linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build -o artifacts/native/mcp +export PROJGRAPH_SMOKE_CLI_NATIVE=artifacts/native/cli/ProjGraph.Cli +export PROJGRAPH_SMOKE_MCP_NATIVE=artifacts/native/mcp/ProjGraph.Mcp +dotnet test tests/ProjGraph.Tests.Smoke.Aot -c Release --no-build +``` + +Expected: `Passed: 26`, about 20 s. The MCP binary is about 42.9 MB. + +- [ ] **Step 7: Confirm the publish didn't touch the JIT reference, and that the MCP suites are unchanged** + +```bash +ls src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp +grep -c IsDynamicCodeSupported src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp.runtimeconfig.json +unset PROJGRAPH_SMOKE_CLI_NATIVE PROJGRAPH_SMOKE_CLI_REFERENCE PROJGRAPH_SMOKE_MCP_NATIVE PROJGRAPH_SMOKE_MCP_REFERENCE +dotnet test tests/ProjGraph.Tests.Integration.Mcp -c Release --no-build +dotnet test tests/ProjGraph.Tests.Contract -c Release --no-build +``` + +Expected: the apphost exists; `grep -c` prints `0` (no AOT feature switches in the JIT runtimeconfig); both suites pass, including `McpSurfaceSnapshotTests` and `McpTransportTests`. + +- [ ] **Step 8: Commit** + +```bash +dotnet format ProjGraph.slnx --verify-no-changes +git add src/ProjGraph.Mcp/ProjGraph.Mcp.csproj tests/ProjGraph.Tests.Smoke.Aot +git commit -m "feat(mcp): make the MCP server publishable as Native AOT and add MCP parity cases" +``` + +--- + +### Task 4: `aot-smoke` CI job + +**Files:** +- Modify: `.github/workflows/ci.yml` +- Modify: `CONTRIBUTING.md` + +**Interfaces:** +- Consumes: the publish commands from Tasks 2–3, and the suite with `PROJGRAPH_SMOKE_REQUIRED` from Task 1. +- Produces: a GitHub Actions check named `aot-smoke` (the job id, since the job sets no `name`). + +- [ ] **Step 1: Add the job** + +Append to `.github/workflows/ci.yml`, under `jobs:` after the `build` job (same indentation as `build:`): + +```yaml + + aot-smoke: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v7 + + - name: Setup .NET + uses: actions/setup-dotnet@v6 + with: + global-json-file: global.json + cache: true + cache-dependency-path: Directory.Packages.props + + - name: Install Native AOT prerequisites + run: | + sudo apt-get update + sudo apt-get install -y --no-install-recommends clang zlib1g-dev + + - name: Restore dependencies + run: dotnet restore ProjGraph.slnx + + - name: Build + run: dotnet build ProjGraph.slnx --no-restore --configuration Release + + # --artifacts-path keeps the native publish out of src/*/bin and obj. ProjGraph.Mcp is + # self-contained, so an in-place publish would overwrite the JIT reference build's apphost + # and runtimeconfig.json in bin/Release/net10.0/linux-x64. + - name: Publish native executables + run: | + dotnet publish src/ProjGraph.Cli --configuration Release --runtime linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build --output artifacts/native/cli + dotnet publish src/ProjGraph.Mcp --configuration Release --runtime linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build --output artifacts/native/mcp + + - name: Smoke test native executables + env: + PROJGRAPH_SMOKE_REQUIRED: 'true' + PROJGRAPH_SMOKE_CLI_NATIVE: artifacts/native/cli/ProjGraph.Cli + PROJGRAPH_SMOKE_CLI_REFERENCE: src/ProjGraph.Cli/bin/Release/net10.0/ProjGraph.Cli.dll + PROJGRAPH_SMOKE_MCP_NATIVE: artifacts/native/mcp/ProjGraph.Mcp + PROJGRAPH_SMOKE_MCP_REFERENCE: src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp.dll + run: dotnet test tests/ProjGraph.Tests.Smoke.Aot --no-build --configuration Release --logger "console;verbosity=normal" +``` + +- [ ] **Step 2: Run the job's commands locally from a clean clone** + +This proves the YAML's commands work from scratch, with no leftovers from earlier tasks. A clone only sees committed state, so commit the workflow first: + +```bash +git add .github/workflows/ci.yml && git commit -m "ci: add the aot-smoke job" +rm -rf /tmp/pg-aot-ci && git clone --quiet --branch feat/aot-native-smoke . /tmp/pg-aot-ci +cd /tmp/pg-aot-ci +dotnet restore ProjGraph.slnx +dotnet build ProjGraph.slnx --no-restore --configuration Release +dotnet publish src/ProjGraph.Cli --configuration Release --runtime linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build --output artifacts/native/cli +dotnet publish src/ProjGraph.Mcp --configuration Release --runtime linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build --output artifacts/native/mcp +PROJGRAPH_SMOKE_REQUIRED=true \ +PROJGRAPH_SMOKE_CLI_NATIVE=artifacts/native/cli/ProjGraph.Cli \ +PROJGRAPH_SMOKE_CLI_REFERENCE=src/ProjGraph.Cli/bin/Release/net10.0/ProjGraph.Cli.dll \ +PROJGRAPH_SMOKE_MCP_NATIVE=artifacts/native/mcp/ProjGraph.Mcp \ +PROJGRAPH_SMOKE_MCP_REFERENCE=src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp.dll \ +dotnet test tests/ProjGraph.Tests.Smoke.Aot --no-build --configuration Release +cd - && rm -rf /tmp/pg-aot-ci +``` + +Expected: `Passed: 26, Skipped: 0`. The job's step names and commands must match these exactly. If a `git status` in the clone shows anything untracked outside `artifacts/`, `bin/`, and `obj/`, fix that before continuing. + +- [ ] **Step 3: Confirm the existing matrix is unaffected** + +From the repository (not the clone), run the matrix's commands: + +```bash +dotnet restore ProjGraph.slnx +dotnet build ProjGraph.slnx --no-restore --configuration Release +dotnet format ProjGraph.slnx --no-restore --verify-no-changes +dotnet test ProjGraph.slnx --no-build --configuration Release +``` + +Expected: build 0 warnings, format clean, every suite green, and `ProjGraph.Tests.Smoke.Aot` reports `Skipped: 26`. + +- [ ] **Step 4: Document running the suite locally** + +In `CONTRIBUTING.md`, after the "Test Organisation" table, add: + +````markdown +### Native AOT smoke tests + +`Tests.Smoke.Aot` compares Native AOT builds of the CLI and MCP server with the JIT build of the +same commit. It is skipped unless the `PROJGRAPH_SMOKE_*` variables are set, and the `aot-smoke` +CI job runs it on every PR. To run it locally on Linux (requires `clang` or `gcc`; the `--artifacts-path` +keeps the self-contained MCP publish from overwriting the JIT build): + +```bash +dotnet build ProjGraph.slnx -c Release +dotnet publish src/ProjGraph.Cli -c Release -r linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build -o artifacts/native/cli +dotnet publish src/ProjGraph.Mcp -c Release -r linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build -o artifacts/native/mcp +PROJGRAPH_SMOKE_CLI_NATIVE=artifacts/native/cli/ProjGraph.Cli \ +PROJGRAPH_SMOKE_CLI_REFERENCE=src/ProjGraph.Cli/bin/Release/net10.0/ProjGraph.Cli.dll \ +PROJGRAPH_SMOKE_MCP_NATIVE=artifacts/native/mcp/ProjGraph.Mcp \ +PROJGRAPH_SMOKE_MCP_REFERENCE=src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp.dll \ +dotnet test tests/ProjGraph.Tests.Smoke.Aot -c Release --no-build +``` +```` + +- [ ] **Step 5: Commit** + +```bash +git add CONTRIBUTING.md +git commit -m "docs: explain how to run the Native AOT smoke suite locally" +``` + +--- + +### Task 5: Pull request and required check + +Pushing, opening the PR, and changing branch protection are outward-facing. Confirm with the user before each. + +- [ ] **Step 1: Push and open the PR (after user confirmation)** + +```bash +git push -u origin feat/aot-native-smoke +gh pr create --base develop --title "feat: Native AOT build and PR smoke test (PR 2 of 3)" --body-file +``` + +The description must include: +- What changed (CLI/MCP configuration, the smoke suite, the `aot-smoke` job) and that no csproj sets `PublishAot`. +- The ten "Deviations from the spec" above. +- Binary sizes and the local suite timing (~20 s). +- A request for the maintainer to add `aot-smoke` as a required status check on `develop`. +- It ends with the attribution line from the session's system reminder. + +- [ ] **Step 2: Wait for CI** + +```bash +gh pr checks --watch +``` + +Expected: the three `build` matrix jobs and `aot-smoke` all pass. In the `aot-smoke` log, the test summary shows `Passed: 26` and `Skipped: 0`. If `aot-smoke` fails on the runner but passed locally, read the log before changing anything (use superpowers:systematic-debugging). The likeliest runner-only differences are toolchain packages and console-width or encoding detection under GitHub Actions. Both builds see the same environment, so any output difference is a real finding. + +- [ ] **Step 3: Make `aot-smoke` required** + +This is a repository-settings change, so it belongs to the user (or run it only with their explicit go-ahead). After the check has reported at least once: GitHub → Settings → Branches (or Rules) → `develop` → required status checks → add `aot-smoke`. This fulfils the spec's PR 2 exit criterion "the `aot-smoke` job is green and required". + +## Exit criteria (spec, PR 2) + +- `aot-smoke` is green on the PR and marked required on `develop`. +- No csproj or props file sets `PublishAot` (`git grep -n PublishAot -- '*.csproj' '*.props'` is empty). +- The existing CI matrix is green, with `Tests.Smoke.Aot` skipped there. From 6cc74ce1223a07f55e7eea87990c6a18ab37e1e3 Mon Sep 17 00:00:00 2001 From: = Date: Mon, 14 Sep 2026 21:19:39 +0200 Subject: [PATCH 02/10] test: add the Native AOT smoke suite with CLI parity cases Co-Authored-By: Claude Sonnet 5 --- ARCHITECTURE.md | 1 + CLAUDE.md | 1 + CONTRIBUTING.md | 15 +- ProjGraph.slnx | 1 + .../CliParityTests.cs | 270 ++++++++++++++++++ .../Fixtures/legacy.sln | 26 ++ .../Helpers/AotSmokeFactAttribute.cs | 22 ++ .../Helpers/ProcessRunner.cs | 65 +++++ .../Helpers/SmokeCommand.cs | 27 ++ .../Helpers/SmokeEnvironment.cs | 96 +++++++ .../ProjGraph.Tests.Smoke.Aot.csproj | 27 ++ 11 files changed, 544 insertions(+), 7 deletions(-) create mode 100644 tests/ProjGraph.Tests.Smoke.Aot/CliParityTests.cs create mode 100644 tests/ProjGraph.Tests.Smoke.Aot/Fixtures/legacy.sln create mode 100644 tests/ProjGraph.Tests.Smoke.Aot/Helpers/AotSmokeFactAttribute.cs create mode 100644 tests/ProjGraph.Tests.Smoke.Aot/Helpers/ProcessRunner.cs create mode 100644 tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeCommand.cs create mode 100644 tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeEnvironment.cs create mode 100644 tests/ProjGraph.Tests.Smoke.Aot/ProjGraph.Tests.Smoke.Aot.csproj diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index a4c21b5..667bd3b 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -22,6 +22,7 @@ ProjGraph.slnx │ ├── ProjGraph.Tests.Unit.* # Unit tests per library │ ├── ProjGraph.Tests.Integration.* # Integration tests for CLI and MCP │ ├── ProjGraph.Tests.Contract # MCP contract & DI wiring tests +│ ├── ProjGraph.Tests.Smoke.Aot # Native AOT vs JIT parity (CI aot-smoke job) │ └── ProjGraph.Tests.Shared # Shared test helpers └── samples/ # Sample projects used by integration tests ``` diff --git a/CLAUDE.md b/CLAUDE.md index b96d350..3e74f73 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -83,6 +83,7 @@ Application/ | `Tests.Integration.Cli` | CLI end-to-end tests | | `Tests.Integration.Mcp` | MCP tool integration tests | | `Tests.Contract` | MCP contract validation & DI wiring | +| `Tests.Smoke.Aot` | Native AOT vs JIT parity for the CLI and MCP executables (skipped unless `PROJGRAPH_SMOKE_*` is set; runs in the `aot-smoke` CI job) | | `Tests.Shared` | Shared helpers (`TestDirectory`, `TestPathHelper`) | ### Release diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index ca617f8..7a6f8b5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -58,13 +58,14 @@ dotnet test tests/ProjGraph.Tests.Unit.ClassDiagram --filter "ClassAnalysisDepth ### Test Organisation -| Project | Purpose | -|-------------------------|----------------------------------------------------| -| `Tests.Unit.*` | Unit tests per library | -| `Tests.Integration.Cli` | CLI end-to-end tests | -| `Tests.Integration.Mcp` | MCP tool integration tests | -| `Tests.Contract` | MCP contract validation & DI wiring | -| `Tests.Shared` | Shared helpers (`TestDirectory`, `TestPathHelper`) | +| Project | Purpose | +|-------------------------|--------------------------------------------------------------------------------------------------------------------------------------| +| `Tests.Unit.*` | Unit tests per library | +| `Tests.Integration.Cli` | CLI end-to-end tests | +| `Tests.Integration.Mcp` | MCP tool integration tests | +| `Tests.Contract` | MCP contract validation & DI wiring | +| `Tests.Smoke.Aot` | Native AOT vs JIT parity for the CLI and MCP executables (skipped unless `PROJGRAPH_SMOKE_*` is set; runs in the `aot-smoke` CI job) | +| `Tests.Shared` | Shared helpers (`TestDirectory`, `TestPathHelper`) | ## Adding a New Feature diff --git a/ProjGraph.slnx b/ProjGraph.slnx index 4afec82..0e38175 100644 --- a/ProjGraph.slnx +++ b/ProjGraph.slnx @@ -44,6 +44,7 @@ + diff --git a/tests/ProjGraph.Tests.Smoke.Aot/CliParityTests.cs b/tests/ProjGraph.Tests.Smoke.Aot/CliParityTests.cs new file mode 100644 index 0000000..769cfd2 --- /dev/null +++ b/tests/ProjGraph.Tests.Smoke.Aot/CliParityTests.cs @@ -0,0 +1,270 @@ +using ProjGraph.Tests.Shared.Helpers; +using ProjGraph.Tests.Smoke.Aot.Helpers; +using System.Text.RegularExpressions; + +namespace ProjGraph.Tests.Smoke.Aot; + +/// +/// Runs CLI commands through the native executable and through the JIT build of the same commit, +/// and requires the same exit code, standard output, standard error, and written diagram file. +/// +public sealed partial class CliParityTests : IDisposable +{ + private const string ModularSolution = "samples/visualize/modular-architecture/ModularArchitecture.slnx"; + + private readonly TestDirectory _temp = new(); + + public void Dispose() + { + _temp.Dispose(); + } + + [AotSmokeFact] + public async Task Visualize_Mermaid_ShouldMatchReference() + { + await AssertDiagramParityAsync("visualize", ModularSolution, "--format", "mermaid"); + } + + [AotSmokeFact] + public async Task Visualize_Tree_ShouldMatchReference() + { + await AssertDiagramParityAsync("visualize", ModularSolution, "--format", "tree"); + } + + [AotSmokeFact] + public async Task Visualize_Flat_ShouldMatchReference() + { + await AssertDiagramParityAsync("visualize", ModularSolution, "--format", "flat"); + } + + [AotSmokeFact] + public async Task Visualize_LegacySln_ShouldMatchReference() + { + await AssertDiagramParityAsync( + "visualize", "tests/ProjGraph.Tests.Smoke.Aot/Fixtures/legacy.sln", "--format", "mermaid"); + } + + [AotSmokeFact] + public async Task Erd_ComplexEcommerce_ShouldMatchReference() + { + await AssertDiagramParityAsync("erd", "samples/erd/complex-ecommerce/Data/MyDbContext.cs"); + } + + [AotSmokeFact] + public async Task Erd_SimpleContext_ShouldMatchReference() + { + await AssertDiagramParityAsync("erd", "samples/erd/simple-context/EntityFramework/MyDbContext.cs"); + } + + [AotSmokeFact] + public async Task ClassDiagram_DesignPatterns_ShouldMatchReference() + { + await AssertDiagramParityAsync( + "classdiagram", "samples/classdiagram/design-patterns/Domain/Order.cs", + "--inheritance", "--dependencies", "--depth", "2", "--properties", "true", "--functions", "true"); + } + + [AotSmokeFact] + public async Task ClassDiagram_ComplexHierarchy_ShouldMatchReference() + { + await AssertDiagramParityAsync( + "classdiagram", "samples/classdiagram/complex-hierarchy/Domain/Models/CEO.cs", + "--inheritance", "--dependencies", "--depth", "5", "--properties", "true", "--functions", "true"); + } + + [AotSmokeFact] + public async Task ClassDiagram_SimpleHierarchy_ShouldMatchReference() + { + await AssertDiagramParityAsync( + "classdiagram", "samples/classdiagram/simple-hierarchy/Models/Admin.cs", + "--inheritance", "--dependencies", "--depth", "2", "--properties", "true", "--functions", "true"); + } + + [AotSmokeFact] + public async Task ClassDiagram_SourceDirectory_ShouldMatchReference() + { + await AssertDiagramParityAsync("classdiagram", "src", "--inheritance", "--dependencies"); + } + + [AotSmokeFact] + public async Task ClassDiagram_ConcurrencyAndAnnotationTypes_ShouldMatchReference() + { + // BCL types that bind only through the embedded reference assemblies (spec §1.5). + _temp.CreateFile("Types/Types.csproj", + "net10.0"); + var file = _temp.CreateFile("Types/Inventory.cs", + """ + using System.Collections.Concurrent; + using System.ComponentModel.DataAnnotations; + + namespace Types; + + public class Inventory : Dictionary, IDisposable + { + [Required] + public string Name { get; set; } = ""; + + public ConcurrentDictionary Items { get; } = new(); + + public Barrier? Gate { get; set; } + + public IEnumerable Available() => Items.Values; + + public void Dispose() => Gate?.Dispose(); + } + + public record StockItem(string Sku, decimal Price); + """); + + await AssertDiagramParityAsync("classdiagram", file, "--inheritance", "--dependencies"); + } + + [AotSmokeFact] + public async Task Visualize_SolutionWithMalformedProject_ShouldMatchReference() + { + var solution = CreateSolutionWithMalformedProject(); + + await AssertDiagramParityAsync("visualize", solution, "--format", "mermaid"); + } + + [AotSmokeFact] + public async Task Visualize_SlnListingAProjectTwice_ShouldFailLikeReference() + { + _temp.CreateFile("Good/Good.csproj", + "net10.0"); + var solution = _temp.CreateFile("Duplicate.sln", + """ + Microsoft Visual Studio Solution File, Format Version 12.00 + Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Good", "Good\Good.csproj", "{0A5E0000-0000-4000-8000-000000000001}" + EndProject + Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "GoodAgain", "Good\Good.csproj", "{0A5E0000-0000-4000-8000-000000000002}" + EndProject + """); + + await AssertParityAsync(expectSuccess: false, "visualize", solution, "--format", "mermaid"); + } + + [AotSmokeFact] + public async Task Stats_Repository_ShouldMatchReferenceApartFromTiming() + { + await AssertStatsParityAsync("stats", "ProjGraph.slnx"); + } + + [AotSmokeFact] + public async Task Stats_SolutionWithMalformedProject_ShouldMatchReferenceApartFromTiming() + { + var solution = CreateSolutionWithMalformedProject(); + + await AssertStatsParityAsync("stats", solution); + } + + [AotSmokeFact] + public async Task Stats_UnknownOption_ShouldFailLikeReference() + { + // Strict parsing rejects the typo before the command runs (Spectre's error rendering path). + await AssertParityAsync(expectSuccess: false, "stats", "ProjGraph.slnx", "--tpo", "3"); + } + + private string CreateSolutionWithMalformedProject() + { + _temp.CreateFile("Good/Good.csproj", + "net10.0"); + _temp.CreateFile("Bad/Bad.csproj", ""); + return _temp.CreateFile("Malformed.slnx", + ""); + } + + private static async Task AssertParityAsync(bool expectSuccess, params string[] arguments) + { + var (native, reference) = await RunBothAsync(arguments); + + AssertReferenceOutcome(reference, expectSuccess); + AssertSameResult(native, reference); + } + + /// + /// Compares a diagram command twice: printed to the console, then written with --output. + /// Both builds write to the same path one after the other, because the "Saved to" line on + /// standard error names the path and Spectre wraps it at the console width, so two different + /// paths could not be compared exactly. + /// + /// The CLI arguments, without --output. + private async Task AssertDiagramParityAsync(params string[] arguments) + { + await AssertParityAsync(expectSuccess: true, arguments); + + var outputPath = Path.Combine(_temp.DirectoryPath, "output", "diagram.mmd"); + string[] withOutput = [.. arguments, "--output", outputPath]; + + var native = await ProcessRunner.RunAsync(SmokeEnvironment.CliNative, withOutput); + File.Exists(outputPath).Should().BeTrue($"the native build must write the diagram; stderr:\n{native.StandardError}"); + var nativeFile = await File.ReadAllBytesAsync(outputPath); + File.Delete(outputPath); + + var reference = await ProcessRunner.RunAsync(SmokeEnvironment.CliReference, withOutput); + AssertReferenceOutcome(reference, expectSuccess: true); + var referenceFile = await File.ReadAllBytesAsync(outputPath); + + AssertSameResult(native, reference); + nativeFile.Should().Equal(referenceFile, "the native build must write the same diagram bytes"); + } + + private static async Task AssertStatsParityAsync(params string[] arguments) + { + var (native, reference) = await RunBothAsync(arguments); + + AssertReferenceOutcome(reference, expectSuccess: true); + native.ExitCode.Should().Be(reference.ExitCode); + NormalizeStats(native.StandardOutput).Should().Be(NormalizeStats(reference.StandardOutput)); + native.StandardError.Should().Be(reference.StandardError); + } + + private static async Task<(ProcessResult Native, ProcessResult Reference)> RunBothAsync(string[] arguments) + { + var native = await ProcessRunner.RunAsync(SmokeEnvironment.CliNative, arguments); + var reference = await ProcessRunner.RunAsync(SmokeEnvironment.CliReference, arguments); + return (native, reference); + } + + /// + /// Pins the reference outcome, so a case where both builds fail the same way (a wrong path, a + /// missing sample) cannot pass as parity. + /// + /// The reference build's result. + /// Whether the reference build must exit with 0. + private static void AssertReferenceOutcome(ProcessResult reference, bool expectSuccess) + { + if (expectSuccess) + { + reference.ExitCode.Should().Be(0, $"the reference build must succeed; stderr:\n{reference.StandardError}"); + } + else + { + reference.ExitCode.Should().NotBe(0, "the reference build must reject this input"); + } + } + + private static void AssertSameResult(ProcessResult native, ProcessResult reference) + { + native.ExitCode.Should().Be(reference.ExitCode, $"native stderr:\n{native.StandardError}"); + native.StandardOutput.Should().Be(reference.StandardOutput); + native.StandardError.Should().Be(reference.StandardError); + } + + /// + /// Drops the wall-clock "Analysis time" row and collapses runs of spaces, because the timing + /// value's width can shift the table's padding. + /// + /// The captured standard output of stats. + /// The output without timing, for comparison. + private static string NormalizeStats(string output) + { + return string.Join('\n', output.ReplaceLineEndings("\n") + .Split('\n') + .Where(line => !line.TrimStart().StartsWith("Analysis time", StringComparison.Ordinal)) + .Select(line => RunOfSpaces().Replace(line, " "))); + } + + [GeneratedRegex(" {2,}")] + private static partial Regex RunOfSpaces(); +} diff --git a/tests/ProjGraph.Tests.Smoke.Aot/Fixtures/legacy.sln b/tests/ProjGraph.Tests.Smoke.Aot/Fixtures/legacy.sln new file mode 100644 index 0000000..9bd47cf --- /dev/null +++ b/tests/ProjGraph.Tests.Smoke.Aot/Fixtures/legacy.sln @@ -0,0 +1,26 @@ +Microsoft Visual Studio Solution File, Format Version 12.00 +# Visual Studio Version 17 +VisualStudioVersion = 17.0.31903.59 +MinimumVisualStudioVersion = 10.0.40219.1 +Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "modular-architecture", "modular-architecture", "{0A5E0000-0000-4000-8000-000000000001}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "App", "..\..\..\samples\visualize\modular-architecture\App\App.csproj", "{0A5E0000-0000-4000-8000-000000000002}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Core", "..\..\..\samples\visualize\modular-architecture\Core\Core.csproj", "{0A5E0000-0000-4000-8000-000000000003}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Infrastructure", "..\..\..\samples\visualize\modular-architecture\Infrastructure\Infrastructure.csproj", "{0A5E0000-0000-4000-8000-000000000004}" +EndProject +Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Services", "..\..\..\samples\visualize\modular-architecture\Services\Services.csproj", "{0A5E0000-0000-4000-8000-000000000005}" +EndProject +Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Shared", "..\..\..\samples\visualize\modular-architecture\Shared\Shared.csproj", "{0A5E0000-0000-4000-8000-000000000006}" +EndProject +Global + GlobalSection(SolutionConfigurationPlatforms) = preSolution + Debug|Any CPU = Debug|Any CPU + Release|Any CPU = Release|Any CPU + EndGlobalSection + GlobalSection(NestedProjects) = preSolution + {0A5E0000-0000-4000-8000-000000000002} = {0A5E0000-0000-4000-8000-000000000001} + {0A5E0000-0000-4000-8000-000000000003} = {0A5E0000-0000-4000-8000-000000000001} + EndGlobalSection +EndGlobal diff --git a/tests/ProjGraph.Tests.Smoke.Aot/Helpers/AotSmokeFactAttribute.cs b/tests/ProjGraph.Tests.Smoke.Aot/Helpers/AotSmokeFactAttribute.cs new file mode 100644 index 0000000..f6f38e5 --- /dev/null +++ b/tests/ProjGraph.Tests.Smoke.Aot/Helpers/AotSmokeFactAttribute.cs @@ -0,0 +1,22 @@ +namespace ProjGraph.Tests.Smoke.Aot.Helpers; + +/// +/// A fact that runs only when all four PROJGRAPH_SMOKE_* variables are set, so a plain +/// dotnet test ProjGraph.slnx reports the smoke suite as skipped instead of failing it. When +/// PROJGRAPH_SMOKE_REQUIRED=true, nothing is skipped and a missing variable fails the test. +/// +[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)] +public sealed class AotSmokeFactAttribute : FactAttribute +{ + /// + /// Initializes a new instance of the class. + /// + public AotSmokeFactAttribute() + { + var missing = SmokeEnvironment.GetMissingVariables(); + if (missing.Count > 0 && !SmokeEnvironment.IsRequired) + { + Skip = $"Native AOT smoke suite not configured; set {string.Join(", ", missing)}."; + } + } +} diff --git a/tests/ProjGraph.Tests.Smoke.Aot/Helpers/ProcessRunner.cs b/tests/ProjGraph.Tests.Smoke.Aot/Helpers/ProcessRunner.cs new file mode 100644 index 0000000..7555442 --- /dev/null +++ b/tests/ProjGraph.Tests.Smoke.Aot/Helpers/ProcessRunner.cs @@ -0,0 +1,65 @@ +using System.Diagnostics; +using System.Text; + +namespace ProjGraph.Tests.Smoke.Aot.Helpers; + +/// +/// The observable result of one CLI invocation. +/// +/// The process exit code. +/// Everything written to standard output. +/// Everything written to standard error. +internal sealed record ProcessResult(int ExitCode, string StandardOutput, string StandardError); + +/// +/// Runs a build of the CLI from the repository root and captures its output. +/// +internal static class ProcessRunner +{ + private static readonly TimeSpan Timeout = TimeSpan.FromMinutes(3); + + /// + /// Runs with and waits for it to exit. + /// + /// The build to run. + /// The CLI arguments. + /// The exit code and the captured output streams. + /// Thrown when the process cannot be started. + /// Thrown when the process does not exit in time; it is killed. + public static async Task RunAsync(SmokeCommand command, IReadOnlyList arguments) + { + var startInfo = new ProcessStartInfo(command.FileName) + { + WorkingDirectory = SmokeEnvironment.RepositoryRoot, + RedirectStandardOutput = true, + RedirectStandardError = true, + StandardOutputEncoding = Encoding.UTF8, + StandardErrorEncoding = Encoding.UTF8, + UseShellExecute = false + }; + foreach (var argument in command.LeadingArguments.Concat(arguments)) + { + startInfo.ArgumentList.Add(argument); + } + + using var process = Process.Start(startInfo) + ?? throw new InvalidOperationException($"Could not start {command.FileName}."); + using var timeout = new CancellationTokenSource(Timeout); + + // Both streams are drained concurrently so a full pipe buffer can never block the child. + var standardOutput = process.StandardOutput.ReadToEndAsync(timeout.Token); + var standardError = process.StandardError.ReadToEndAsync(timeout.Token); + try + { + await process.WaitForExitAsync(timeout.Token); + } + catch (OperationCanceledException) + { + process.Kill(entireProcessTree: true); + throw new TimeoutException( + $"'{command.FileName} {string.Join(' ', arguments)}' did not exit within {Timeout}."); + } + + return new ProcessResult(process.ExitCode, await standardOutput, await standardError); + } +} diff --git a/tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeCommand.cs b/tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeCommand.cs new file mode 100644 index 0000000..7ce4e45 --- /dev/null +++ b/tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeCommand.cs @@ -0,0 +1,27 @@ +namespace ProjGraph.Tests.Smoke.Aot.Helpers; + +/// +/// How to start one build of a tool: an executable is started directly, and a framework-dependent +/// .dll is started through the dotnet host. +/// +/// The program to start. +/// Arguments that precede the tool's own arguments. +internal sealed record SmokeCommand(string FileName, IReadOnlyList LeadingArguments) +{ + /// + /// Creates the command that starts the build at . + /// + /// The absolute path to an executable or a .dll. + /// The command. + public static SmokeCommand For(string path) + { + if (!path.EndsWith(".dll", StringComparison.OrdinalIgnoreCase)) + { + return new SmokeCommand(path, []); + } + + // dotnet test exports the host that runs the tests, so the reference build runs on the same SDK. + var host = Environment.GetEnvironmentVariable("DOTNET_HOST_PATH"); + return new SmokeCommand(string.IsNullOrEmpty(host) ? "dotnet" : host, [path]); + } +} diff --git a/tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeEnvironment.cs b/tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeEnvironment.cs new file mode 100644 index 0000000..7e4100e --- /dev/null +++ b/tests/ProjGraph.Tests.Smoke.Aot/Helpers/SmokeEnvironment.cs @@ -0,0 +1,96 @@ +namespace ProjGraph.Tests.Smoke.Aot.Helpers; + +/// +/// The smoke suite's configuration: the four builds under comparison, read from environment +/// variables, and the repository root that every command runs from. +/// +internal static class SmokeEnvironment +{ + private const string CliNativeVariable = "PROJGRAPH_SMOKE_CLI_NATIVE"; + private const string CliReferenceVariable = "PROJGRAPH_SMOKE_CLI_REFERENCE"; + private const string McpNativeVariable = "PROJGRAPH_SMOKE_MCP_NATIVE"; + private const string McpReferenceVariable = "PROJGRAPH_SMOKE_MCP_REFERENCE"; + private const string RequiredVariable = "PROJGRAPH_SMOKE_REQUIRED"; + + private static readonly string[] RequiredVariables = + [CliNativeVariable, CliReferenceVariable, McpNativeVariable, McpReferenceVariable]; + + private static readonly Lazy LazyRepositoryRoot = new(FindRepositoryRoot); + + /// + /// Gets the repository root: the nearest ancestor of the test output directory that contains + /// ProjGraph.slnx. + /// + public static string RepositoryRoot => LazyRepositoryRoot.Value; + + /// + /// Gets a value indicating whether the suite must run. CI sets PROJGRAPH_SMOKE_REQUIRED=true, + /// so a misspelled or missing variable fails the job instead of skipping every test. + /// + public static bool IsRequired => + string.Equals(Environment.GetEnvironmentVariable(RequiredVariable), "true", StringComparison.OrdinalIgnoreCase); + + /// Gets the native CLI executable. + public static SmokeCommand CliNative => Resolve(CliNativeVariable); + + /// Gets the JIT CLI build of the same commit. + public static SmokeCommand CliReference => Resolve(CliReferenceVariable); + + /// Gets the native MCP server executable. + public static SmokeCommand McpNative => Resolve(McpNativeVariable); + + /// Gets the JIT MCP server build of the same commit. + public static SmokeCommand McpReference => Resolve(McpReferenceVariable); + + /// + /// Lists the required variables that are unset or empty. + /// + /// The missing variable names, empty when the suite is fully configured. + public static IReadOnlyList GetMissingVariables() + { + return + [ + .. RequiredVariables.Where(name => string.IsNullOrWhiteSpace(Environment.GetEnvironmentVariable(name))) + ]; + } + + /// + /// Resolves an absolute path under the repository root. + /// + /// A path relative to the repository root, using / separators. + /// The absolute path. + public static string GetRootPath(string relativePath) + { + return Path.GetFullPath(Path.Combine(RepositoryRoot, relativePath)); + } + + private static SmokeCommand Resolve(string variable) + { + var value = Environment.GetEnvironmentVariable(variable); + if (string.IsNullOrWhiteSpace(value)) + { + throw new InvalidOperationException($"{variable} is not set."); + } + + // A configured path that does not exist is a broken CI setup, never a reason to skip. + var path = Path.GetFullPath(value, RepositoryRoot); + return File.Exists(path) + ? SmokeCommand.For(path) + : throw new FileNotFoundException($"{variable} points to a file that does not exist.", path); + } + + private static string FindRepositoryRoot() + { + for (var directory = new DirectoryInfo(AppContext.BaseDirectory); + directory is not null; + directory = directory.Parent) + { + if (File.Exists(Path.Combine(directory.FullName, "ProjGraph.slnx"))) + { + return directory.FullName; + } + } + + throw new InvalidOperationException($"No ProjGraph.slnx found above {AppContext.BaseDirectory}."); + } +} diff --git a/tests/ProjGraph.Tests.Smoke.Aot/ProjGraph.Tests.Smoke.Aot.csproj b/tests/ProjGraph.Tests.Smoke.Aot/ProjGraph.Tests.Smoke.Aot.csproj new file mode 100644 index 0000000..81c09eb --- /dev/null +++ b/tests/ProjGraph.Tests.Smoke.Aot/ProjGraph.Tests.Smoke.Aot.csproj @@ -0,0 +1,27 @@ + + + + true + + + + + + + + + + + + + + + + + + + + + + From cf3511b498143d42f4062f3650d3063862dff772 Mon Sep 17 00:00:00 2001 From: = Date: Mon, 14 Sep 2026 21:31:32 +0200 Subject: [PATCH 03/10] feat(cli): make the CLI publishable as Native AOT Co-Authored-By: Claude Sonnet 5 --- .../Infrastructure/DependencyInjection.cs | 5 +++++ src/ProjGraph.Cli/Program.cs | 5 +++++ src/ProjGraph.Cli/ProjGraph.Cli.csproj | 13 +++++++++++++ 3 files changed, 23 insertions(+) diff --git a/src/ProjGraph.Cli/Infrastructure/DependencyInjection.cs b/src/ProjGraph.Cli/Infrastructure/DependencyInjection.cs index 8f36f7c..b37d2d9 100644 --- a/src/ProjGraph.Cli/Infrastructure/DependencyInjection.cs +++ b/src/ProjGraph.Cli/Infrastructure/DependencyInjection.cs @@ -1,5 +1,6 @@ using Microsoft.Extensions.DependencyInjection; using Spectre.Console.Cli; +using System.Diagnostics.CodeAnalysis; namespace ProjGraph.Cli.Infrastructure; @@ -24,6 +25,10 @@ public ITypeResolver Build() /// /// The type of the service to register. /// The type of the implementation to register. + [UnconditionalSuppressMessage("Trimming", "IL2067:DynamicallyAccessedMembers", + Justification = "Spectre.Console.Cli registers the command and settings types it discovers. They live in " + + "ProjGraph.Cli and Spectre.Console.Cli, which are trimmer root assemblies, so their " + + "public constructors are kept; tests/ProjGraph.Tests.Smoke.Aot covers every command.")] public void Register(Type service, Type implementation) { builder.AddSingleton(service, implementation); diff --git a/src/ProjGraph.Cli/Program.cs b/src/ProjGraph.Cli/Program.cs index a60155d..a11c2e3 100644 --- a/src/ProjGraph.Cli/Program.cs +++ b/src/ProjGraph.Cli/Program.cs @@ -3,11 +3,16 @@ using ProjGraph.Cli.Infrastructure; using ProjGraph.Lib; using Spectre.Console.Cli; +using System.Diagnostics.CodeAnalysis; namespace ProjGraph.Cli; internal static class Program { + [UnconditionalSuppressMessage("AOT", "IL3050:RequiresDynamicCode", + Justification = "Spectre.Console.Cli reflects over the command and settings types. ProjGraph.Cli and " + + "Spectre.Console.Cli are trimmer root assemblies, so that metadata is kept, and " + + "tests/ProjGraph.Tests.Smoke.Aot runs every command natively.")] public static int Main(string[] args) { const string packageDiagramCommandName = "visualize"; diff --git a/src/ProjGraph.Cli/ProjGraph.Cli.csproj b/src/ProjGraph.Cli/ProjGraph.Cli.csproj index 588468a..f1d99da 100644 --- a/src/ProjGraph.Cli/ProjGraph.Cli.csproj +++ b/src/ProjGraph.Cli/ProjGraph.Cli.csproj @@ -5,6 +5,12 @@ LatestMajor + + true + true + false + true true @@ -38,4 +44,11 @@ + + + + + + From 36c5229b306d91360a19bd717d4e9207582b9638 Mon Sep 17 00:00:00 2001 From: = Date: Mon, 14 Sep 2026 21:42:29 +0200 Subject: [PATCH 04/10] feat(mcp): make the MCP server publishable as Native AOT and add MCP parity cases Co-Authored-By: Claude Sonnet 5 --- src/ProjGraph.Mcp/ProjGraph.Mcp.csproj | 6 + .../Helpers/McpServerPair.cs | 58 +++++++ .../McpParityTests.cs | 143 ++++++++++++++++++ 3 files changed, 207 insertions(+) create mode 100644 tests/ProjGraph.Tests.Smoke.Aot/Helpers/McpServerPair.cs create mode 100644 tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs diff --git a/src/ProjGraph.Mcp/ProjGraph.Mcp.csproj b/src/ProjGraph.Mcp/ProjGraph.Mcp.csproj index 48d879c..c8437ce 100644 --- a/src/ProjGraph.Mcp/ProjGraph.Mcp.csproj +++ b/src/ProjGraph.Mcp/ProjGraph.Mcp.csproj @@ -15,6 +15,12 @@ true + + true + true + false + false diff --git a/tests/ProjGraph.Tests.Smoke.Aot/Helpers/McpServerPair.cs b/tests/ProjGraph.Tests.Smoke.Aot/Helpers/McpServerPair.cs new file mode 100644 index 0000000..23cd163 --- /dev/null +++ b/tests/ProjGraph.Tests.Smoke.Aot/Helpers/McpServerPair.cs @@ -0,0 +1,58 @@ +using ModelContextProtocol.Client; + +namespace ProjGraph.Tests.Smoke.Aot.Helpers; + +/// +/// One native and one reference MCP server, shared by every test in a class. Stopping a stdio +/// client takes seconds, and one long-lived session per build is also how clients use the server. +/// The servers start on first use, so a skipped suite never launches them. +/// +public sealed class McpServerPair : IAsyncLifetime +{ + private McpClient? _native; + private McpClient? _reference; + + /// + /// Connects to both servers on the first call and returns the same clients afterwards. + /// + /// The native and the reference client. + public async Task<(McpClient Native, McpClient Reference)> GetClientsAsync() + { + _native ??= await ConnectAsync(SmokeEnvironment.McpNative); + _reference ??= await ConnectAsync(SmokeEnvironment.McpReference); + return (_native, _reference); + } + + /// + public Task InitializeAsync() + { + return Task.CompletedTask; + } + + /// + public async Task DisposeAsync() + { + if (_native is not null) + { + await _native.DisposeAsync(); + } + + if (_reference is not null) + { + await _reference.DisposeAsync(); + } + } + + private static async Task ConnectAsync(SmokeCommand command) + { + var transport = new StdioClientTransport(new StdioClientTransportOptions + { + Name = "ProjGraph AOT smoke", + Command = command.FileName, + Arguments = [.. command.LeadingArguments], + WorkingDirectory = SmokeEnvironment.RepositoryRoot + }); + + return await McpClient.CreateAsync(transport); + } +} diff --git a/tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs b/tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs new file mode 100644 index 0000000..26b16c3 --- /dev/null +++ b/tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs @@ -0,0 +1,143 @@ +using ModelContextProtocol; +using ModelContextProtocol.Client; +using ModelContextProtocol.Protocol; +using ProjGraph.Tests.Shared.Helpers; +using ProjGraph.Tests.Smoke.Aot.Helpers; +using System.Text.Json; +using System.Text.Json.Nodes; + +namespace ProjGraph.Tests.Smoke.Aot; + +/// +/// Starts the native MCP server and the JIT build of the same commit over stdio, sends both the +/// same requests, and requires identical protocol results. +/// +/// The native and reference servers shared by this class. +public sealed class McpParityTests(McpServerPair servers) : IClassFixture +{ + [AotSmokeFact] + public async Task Initialize_ShouldAdvertiseTheSameServer() + { + var (native, reference) = await servers.GetClientsAsync(); + + reference.ServerInfo.Name.Should().Be("ProjGraph"); + AssertSameJson(native.ServerInfo, reference.ServerInfo); + AssertSameJson(native.ServerCapabilities, reference.ServerCapabilities); + } + + [AotSmokeFact] + public async Task ListTools_ShouldMatchReference() + { + await AssertSameResultAsync(client => client.ListToolsAsync(new ListToolsRequestParams())); + } + + [AotSmokeFact] + public async Task ListPrompts_ShouldMatchReference() + { + await AssertSameResultAsync(client => client.ListPromptsAsync(new ListPromptsRequestParams())); + } + + [AotSmokeFact] + public async Task GetPrompt_ShouldMatchReference() + { + // Prompt results serialize IEnumerable through McpJsonContext. + var path = SmokeEnvironment.GetRootPath("samples/classdiagram/complex-hierarchy/Domain/Models/CEO.cs"); + + await AssertSameResultAsync(client => client.GetPromptAsync( + "class_structure_review", new Dictionary { ["path"] = path })); + } + + [AotSmokeFact] + public async Task ReadWelcomeResource_ShouldMatchReference() + { + await AssertSameResultAsync(client => client.ReadResourceAsync(new Uri("projgraph://welcome"))); + } + + [AotSmokeFact] + public async Task GetProjectGraph_ShouldMatchReference() + { + await AssertSameToolResultAsync("get_project_graph", new Dictionary + { + ["path"] = SmokeEnvironment.GetRootPath("samples/visualize/modular-architecture/ModularArchitecture.slnx") + }); + } + + [AotSmokeFact] + public async Task GetProjectStats_ShouldMatchReference() + { + await AssertSameToolResultAsync("get_project_stats", new Dictionary + { + ["path"] = SmokeEnvironment.GetRootPath("samples/visualize/modular-architecture/ModularArchitecture.slnx") + }); + } + + [AotSmokeFact] + public async Task GetProjectStats_SolutionWithMalformedProject_ShouldMatchReference() + { + // The warnings array is built with JsonNode APIs that fail without reflection-based JSON. + using var temp = new TestDirectory(); + temp.CreateFile("Good/Good.csproj", + "net10.0"); + temp.CreateFile("Bad/Bad.csproj", ""); + var solution = temp.CreateFile("Malformed.slnx", + ""); + + var reference = await AssertSameToolResultAsync( + "get_project_stats", new Dictionary { ["path"] = solution }); + + var warnings = JsonNode.Parse(JoinText(reference))?["warnings"]?.AsArray(); + warnings.Should().NotBeNullOrEmpty("the malformed project must surface as a warning"); + } + + [AotSmokeFact] + public async Task GetClassDiagram_ShouldMatchReference() + { + await AssertSameToolResultAsync("get_class_diagram", new Dictionary + { + ["path"] = SmokeEnvironment.GetRootPath("samples/classdiagram/complex-hierarchy/Domain/Models/CEO.cs"), + ["options"] = new Dictionary { ["includeInheritance"] = true } + }); + } + + [AotSmokeFact] + public async Task GetErd_ShouldMatchReference() + { + await AssertSameToolResultAsync("get_erd", new Dictionary + { + ["path"] = SmokeEnvironment.GetRootPath("samples/erd/complex-ecommerce/Data/MyDbContext.cs") + }); + } + + private async Task AssertSameResultAsync(Func> request) + { + var (native, reference) = await servers.GetClientsAsync(); + + var referenceResult = await request(reference); + AssertSameJson(await request(native), referenceResult); + return referenceResult; + } + + private async Task AssertSameToolResultAsync( + string toolName, Dictionary arguments) + { + var reference = await AssertSameResultAsync(client => client.CallToolAsync(toolName, arguments)); + + // Pins the reference outcome, so an error both builds return identically cannot pass as parity. + reference.IsError.Should().NotBe(true, JoinText(reference)); + return reference; + } + + private static void AssertSameJson(T native, T reference) + { + var nativeJson = JsonSerializer.SerializeToNode(native, McpJsonUtilities.DefaultOptions); + var referenceJson = JsonSerializer.SerializeToNode(reference, McpJsonUtilities.DefaultOptions); + + JsonNode.DeepEquals(nativeJson, referenceJson).Should().BeTrue( + $"the native server must return the reference result.\nNative:\n{nativeJson}\nReference:\n{referenceJson}"); + } + + private static string JoinText(CallToolResult result) + { + return string.Join("\n", result.Content.OfType().Select(block => block.Text)); + } +} From e8b158cc8a4599ee6c163594429e7671cf34b8e3 Mon Sep 17 00:00:00 2001 From: = Date: Mon, 14 Sep 2026 21:48:53 +0200 Subject: [PATCH 05/10] ci: add the aot-smoke job Co-Authored-By: Claude Sonnet 5 --- .github/workflows/ci.yml | 41 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1f39c60..8b29c5d 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -56,3 +56,44 @@ jobs: name: coverage-reports path: ./coverage/**/coverage.cobertura.xml retention-days: 14 + + aot-smoke: + runs-on: ubuntu-latest + + steps: + - uses: actions/checkout@v7 + + - name: Setup .NET + uses: actions/setup-dotnet@v6 + with: + global-json-file: global.json + cache: true + cache-dependency-path: Directory.Packages.props + + - name: Install Native AOT prerequisites + run: | + sudo apt-get update + sudo apt-get install -y --no-install-recommends clang zlib1g-dev + + - name: Restore dependencies + run: dotnet restore ProjGraph.slnx + + - name: Build + run: dotnet build ProjGraph.slnx --no-restore --configuration Release + + # --artifacts-path keeps the native publish out of src/*/bin and obj. ProjGraph.Mcp is + # self-contained, so an in-place publish would overwrite the JIT reference build's apphost + # and runtimeconfig.json in bin/Release/net10.0/linux-x64. + - name: Publish native executables + run: | + dotnet publish src/ProjGraph.Cli --configuration Release --runtime linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build --output artifacts/native/cli + dotnet publish src/ProjGraph.Mcp --configuration Release --runtime linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build --output artifacts/native/mcp + + - name: Smoke test native executables + env: + PROJGRAPH_SMOKE_REQUIRED: 'true' + PROJGRAPH_SMOKE_CLI_NATIVE: artifacts/native/cli/ProjGraph.Cli + PROJGRAPH_SMOKE_CLI_REFERENCE: src/ProjGraph.Cli/bin/Release/net10.0/ProjGraph.Cli.dll + PROJGRAPH_SMOKE_MCP_NATIVE: artifacts/native/mcp/ProjGraph.Mcp + PROJGRAPH_SMOKE_MCP_REFERENCE: src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp.dll + run: dotnet test tests/ProjGraph.Tests.Smoke.Aot --no-build --configuration Release --logger "console;verbosity=normal" From c8f54385e380b8b0a4f046bebd15aca4b525dabf Mon Sep 17 00:00:00 2001 From: = Date: Mon, 14 Sep 2026 21:53:29 +0200 Subject: [PATCH 06/10] docs: explain how to run the Native AOT smoke suite locally Co-Authored-By: Claude Sonnet 5 --- CONTRIBUTING.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 7a6f8b5..2086889 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -67,6 +67,24 @@ dotnet test tests/ProjGraph.Tests.Unit.ClassDiagram --filter "ClassAnalysisDepth | `Tests.Smoke.Aot` | Native AOT vs JIT parity for the CLI and MCP executables (skipped unless `PROJGRAPH_SMOKE_*` is set; runs in the `aot-smoke` CI job) | | `Tests.Shared` | Shared helpers (`TestDirectory`, `TestPathHelper`) | +### Native AOT smoke tests + +`Tests.Smoke.Aot` compares Native AOT builds of the CLI and MCP server with the JIT build of the +same commit. It is skipped unless the `PROJGRAPH_SMOKE_*` variables are set, and the `aot-smoke` +CI job runs it on every PR. To run it locally on Linux (requires `clang` or `gcc`; the `--artifacts-path` +keeps the self-contained MCP publish from overwriting the JIT build): + +```bash +dotnet build ProjGraph.slnx -c Release +dotnet publish src/ProjGraph.Cli -c Release -r linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build -o artifacts/native/cli +dotnet publish src/ProjGraph.Mcp -c Release -r linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build -o artifacts/native/mcp +PROJGRAPH_SMOKE_CLI_NATIVE=artifacts/native/cli/ProjGraph.Cli \ +PROJGRAPH_SMOKE_CLI_REFERENCE=src/ProjGraph.Cli/bin/Release/net10.0/ProjGraph.Cli.dll \ +PROJGRAPH_SMOKE_MCP_NATIVE=artifacts/native/mcp/ProjGraph.Mcp \ +PROJGRAPH_SMOKE_MCP_REFERENCE=src/ProjGraph.Mcp/bin/Release/net10.0/linux-x64/ProjGraph.Mcp.dll \ +dotnet test tests/ProjGraph.Tests.Smoke.Aot -c Release --no-build +``` + ## Adding a New Feature 1. Add domain models to `ProjGraph.Core` if needed. From 046b2b854c574af10c23a2d8fe61326f26f08d63 Mon Sep 17 00:00:00 2001 From: = Date: Mon, 14 Sep 2026 22:28:49 +0200 Subject: [PATCH 07/10] fix(smoke): strip ANSI when normalizing stats and bound MCP requests Spectre.Console's GitHub enricher turns on ANSI colour under GITHUB_ACTIONS=true, which children inherit, so the stats timing row survived NormalizeStats and Stats_* failed on every aot-smoke run; strip escapes only to decide which line to drop. Also give every MCP request in McpParityTests a 2-minute CancellationToken so a deadlocked server fails the test instead of hanging, add timeout-minutes: 30 to the aot-smoke CI job, require diagram parity cases to produce non-empty output, and add --help parity cases. Co-Authored-By: Claude Sonnet 5 --- .github/workflows/ci.yml | 1 + .../CliParityTests.cs | 41 ++++++++++++++++--- .../McpParityTests.cs | 31 ++++++++++---- 3 files changed, 59 insertions(+), 14 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8b29c5d..29afe5b 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -59,6 +59,7 @@ jobs: aot-smoke: runs-on: ubuntu-latest + timeout-minutes: 30 steps: - uses: actions/checkout@v7 diff --git a/tests/ProjGraph.Tests.Smoke.Aot/CliParityTests.cs b/tests/ProjGraph.Tests.Smoke.Aot/CliParityTests.cs index 769cfd2..2367001 100644 --- a/tests/ProjGraph.Tests.Smoke.Aot/CliParityTests.cs +++ b/tests/ProjGraph.Tests.Smoke.Aot/CliParityTests.cs @@ -165,6 +165,20 @@ public async Task Stats_UnknownOption_ShouldFailLikeReference() await AssertParityAsync(expectSuccess: false, "stats", "ProjGraph.slnx", "--tpo", "3"); } + [AotSmokeFact] + public async Task Help_Root_ShouldMatchReference() + { + // Help rendering is the most reflection-heavy Spectre path (descriptions and defaults on + // settings types). + await AssertParityAsync(expectSuccess: true, "--help"); + } + + [AotSmokeFact] + public async Task Help_Visualize_ShouldMatchReference() + { + await AssertParityAsync(expectSuccess: true, "visualize", "--help"); + } + private string CreateSolutionWithMalformedProject() { _temp.CreateFile("Good/Good.csproj", @@ -174,24 +188,33 @@ private string CreateSolutionWithMalformedProject() ""); } - private static async Task AssertParityAsync(bool expectSuccess, params string[] arguments) + /// + /// Runs both builds and requires they agree, pinning the reference outcome first. + /// + /// Whether the reference build must exit with 0. + /// The CLI arguments. + /// The reference build's result, for callers that need to inspect it further. + private static async Task AssertParityAsync(bool expectSuccess, params string[] arguments) { var (native, reference) = await RunBothAsync(arguments); AssertReferenceOutcome(reference, expectSuccess); AssertSameResult(native, reference); + return reference; } /// /// Compares a diagram command twice: printed to the console, then written with --output. /// Both builds write to the same path one after the other, because the "Saved to" line on /// standard error names the path and Spectre wraps it at the console width, so two different - /// paths could not be compared exactly. + /// paths could not be compared exactly. Both runs must produce non-empty output, so a case where + /// both builds silently print or write nothing cannot pass as parity. /// /// The CLI arguments, without --output. private async Task AssertDiagramParityAsync(params string[] arguments) { - await AssertParityAsync(expectSuccess: true, arguments); + var console = await AssertParityAsync(expectSuccess: true, arguments); + console.StandardOutput.Should().NotBeNullOrWhiteSpace("the console run must print a diagram"); var outputPath = Path.Combine(_temp.DirectoryPath, "output", "diagram.mmd"); string[] withOutput = [.. arguments, "--output", outputPath]; @@ -204,6 +227,7 @@ private async Task AssertDiagramParityAsync(params string[] arguments) var reference = await ProcessRunner.RunAsync(SmokeEnvironment.CliReference, withOutput); AssertReferenceOutcome(reference, expectSuccess: true); var referenceFile = await File.ReadAllBytesAsync(outputPath); + referenceFile.Should().NotBeEmpty("the --output run must write a non-empty diagram file"); AssertSameResult(native, reference); nativeFile.Should().Equal(referenceFile, "the native build must write the same diagram bytes"); @@ -253,7 +277,10 @@ private static void AssertSameResult(ProcessResult native, ProcessResult referen /// /// Drops the wall-clock "Analysis time" row and collapses runs of spaces, because the timing - /// value's width can shift the table's padding. + /// value's width can shift the table's padding. ANSI colour escapes (Spectre's GitHub Actions + /// enricher sets Capabilities.Ansi = true when GITHUB_ACTIONS=true, and child + /// processes inherit that variable) are stripped only to decide which line to drop, so colour + /// output on the other rows is still compared exactly. /// /// The captured standard output of stats. /// The output without timing, for comparison. @@ -261,10 +288,14 @@ private static string NormalizeStats(string output) { return string.Join('\n', output.ReplaceLineEndings("\n") .Split('\n') - .Where(line => !line.TrimStart().StartsWith("Analysis time", StringComparison.Ordinal)) + .Where(line => !AnsiEscape().Replace(line, string.Empty).TrimStart() + .StartsWith("Analysis time", StringComparison.Ordinal)) .Select(line => RunOfSpaces().Replace(line, " "))); } [GeneratedRegex(" {2,}")] private static partial Regex RunOfSpaces(); + + [GeneratedRegex(@"\x1B\[[0-9;?]*[A-Za-z]")] + private static partial Regex AnsiEscape(); } diff --git a/tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs b/tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs index 26b16c3..7c50c3f 100644 --- a/tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs +++ b/tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs @@ -15,6 +15,13 @@ namespace ProjGraph.Tests.Smoke.Aot; /// The native and reference servers shared by this class. public sealed class McpParityTests(McpServerPair servers) : IClassFixture { + /// + /// The time limit for one MCP request. The SDK's own timeout (2.2.0) only covers initialization, + /// so without this a deadlocked native or reference server would hang the test instead of + /// failing it. + /// + private static readonly TimeSpan RequestTimeout = TimeSpan.FromMinutes(2); + [AotSmokeFact] public async Task Initialize_ShouldAdvertiseTheSameServer() { @@ -28,13 +35,15 @@ public async Task Initialize_ShouldAdvertiseTheSameServer() [AotSmokeFact] public async Task ListTools_ShouldMatchReference() { - await AssertSameResultAsync(client => client.ListToolsAsync(new ListToolsRequestParams())); + await AssertSameResultAsync((client, cancellationToken) => + client.ListToolsAsync(new ListToolsRequestParams(), cancellationToken)); } [AotSmokeFact] public async Task ListPrompts_ShouldMatchReference() { - await AssertSameResultAsync(client => client.ListPromptsAsync(new ListPromptsRequestParams())); + await AssertSameResultAsync((client, cancellationToken) => + client.ListPromptsAsync(new ListPromptsRequestParams(), cancellationToken)); } [AotSmokeFact] @@ -43,14 +52,16 @@ public async Task GetPrompt_ShouldMatchReference() // Prompt results serialize IEnumerable through McpJsonContext. var path = SmokeEnvironment.GetRootPath("samples/classdiagram/complex-hierarchy/Domain/Models/CEO.cs"); - await AssertSameResultAsync(client => client.GetPromptAsync( - "class_structure_review", new Dictionary { ["path"] = path })); + await AssertSameResultAsync((client, cancellationToken) => client.GetPromptAsync( + "class_structure_review", new Dictionary { ["path"] = path }, + cancellationToken: cancellationToken)); } [AotSmokeFact] public async Task ReadWelcomeResource_ShouldMatchReference() { - await AssertSameResultAsync(client => client.ReadResourceAsync(new Uri("projgraph://welcome"))); + await AssertSameResultAsync((client, cancellationToken) => + client.ReadResourceAsync(new Uri("projgraph://welcome"), cancellationToken: cancellationToken)); } [AotSmokeFact] @@ -108,19 +119,21 @@ public async Task GetErd_ShouldMatchReference() }); } - private async Task AssertSameResultAsync(Func> request) + private async Task AssertSameResultAsync(Func> request) { var (native, reference) = await servers.GetClientsAsync(); + using var timeout = new CancellationTokenSource(RequestTimeout); - var referenceResult = await request(reference); - AssertSameJson(await request(native), referenceResult); + var referenceResult = await request(reference, timeout.Token); + AssertSameJson(await request(native, timeout.Token), referenceResult); return referenceResult; } private async Task AssertSameToolResultAsync( string toolName, Dictionary arguments) { - var reference = await AssertSameResultAsync(client => client.CallToolAsync(toolName, arguments)); + var reference = await AssertSameResultAsync((client, cancellationToken) => + client.CallToolAsync(toolName, arguments, cancellationToken: cancellationToken)); // Pins the reference outcome, so an error both builds return identically cannot pass as parity. reference.IsError.Should().NotBe(true, JoinText(reference)); From 849bff9872754e3f80b46d49ed1480a12ff48311 Mon Sep 17 00:00:00 2001 From: = Date: Mon, 14 Sep 2026 22:28:56 +0200 Subject: [PATCH 08/10] docs: align the AOT smoke docs and plan with the review fixes CONTRIBUTING.md's Native AOT prerequisite sentence was missing zlib1g-dev, which the CI job installs, and didn't note the commands are linux-x64. SmokeCommand.cs's "same SDK" comment was inaccurate for a self-contained .dll, which uses its app-local runtime. The plan document is updated to match: the git grep form that actually distinguishes intentional csproj comments from a real PublishAot property, the GITHUB_ACTIONS=true env needed to exercise Spectre's GitHub Actions ANSI path locally, the artifacts/native deviation, and the CliParityTests/McpParityTests code blocks as changed by the fixes. Co-Authored-By: Claude Sonnet 5 --- CONTRIBUTING.md | 5 +- ...09-14-native-aot-pr2-native-build-smoke.md | 80 +++++++++++++++---- .../Helpers/SmokeCommand.cs | 2 +- 3 files changed, 67 insertions(+), 20 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 2086889..c0f0d50 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -71,8 +71,9 @@ dotnet test tests/ProjGraph.Tests.Unit.ClassDiagram --filter "ClassAnalysisDepth `Tests.Smoke.Aot` compares Native AOT builds of the CLI and MCP server with the JIT build of the same commit. It is skipped unless the `PROJGRAPH_SMOKE_*` variables are set, and the `aot-smoke` -CI job runs it on every PR. To run it locally on Linux (requires `clang` or `gcc`; the `--artifacts-path` -keeps the self-contained MCP publish from overwriting the JIT build): +CI job runs it on every PR. Native AOT on Linux needs `clang` (or `gcc`) and `zlib1g-dev`, as the +CI job installs. The commands below are for linux-x64; the `--artifacts-path` keeps the +self-contained MCP publish from overwriting the JIT build: ```bash dotnet build ProjGraph.slnx -c Release diff --git a/docs/superpowers/plans/2026-09-14-native-aot-pr2-native-build-smoke.md b/docs/superpowers/plans/2026-09-14-native-aot-pr2-native-build-smoke.md index 7cd1348..492aba7 100644 --- a/docs/superpowers/plans/2026-09-14-native-aot-pr2-native-build-smoke.md +++ b/docs/superpowers/plans/2026-09-14-native-aot-pr2-native-build-smoke.md @@ -36,6 +36,7 @@ 8. **Parameter name.** The spec's MCP case says `options.inheritance`; the real parameter is `options.includeInheritance`. 9. **Shared MCP servers.** Disposing a stdio `McpClient` takes about 5 s in SDK 2.2.0 (the existing `McpTransportTests` pay this too; the servers themselves exit at once on stdin EOF). One native and one reference server are shared across the MCP test class. That brings the whole suite from 1 min 51 s to about 20 s, and it mirrors how clients hold one long session. 10. **Legacy `.sln` fixture.** Besides the five `modular-architecture` projects, it contains a solution folder, a `NestedProjects` section, and one classic C# project-type GUID, so the SolutionPersistence filtering path runs natively. +11. **Native outputs go to git-ignored `artifacts/native/*` (and `artifacts/aot-build`) instead of the spec's `out/*`.** `out/` isn't git-ignored, and committing the ~36.6 MB CLI and ~42.9 MB MCP binaries would bloat the repository. ## Verified facts this plan relies on @@ -587,6 +588,20 @@ public sealed partial class CliParityTests : IDisposable await AssertParityAsync(expectSuccess: false, "stats", "ProjGraph.slnx", "--tpo", "3"); } + [AotSmokeFact] + public async Task Help_Root_ShouldMatchReference() + { + // Help rendering is the most reflection-heavy Spectre path (descriptions and defaults on + // settings types). + await AssertParityAsync(expectSuccess: true, "--help"); + } + + [AotSmokeFact] + public async Task Help_Visualize_ShouldMatchReference() + { + await AssertParityAsync(expectSuccess: true, "visualize", "--help"); + } + private string CreateSolutionWithMalformedProject() { _temp.CreateFile("Good/Good.csproj", @@ -596,24 +611,33 @@ public sealed partial class CliParityTests : IDisposable ""); } - private static async Task AssertParityAsync(bool expectSuccess, params string[] arguments) + /// + /// Runs both builds and requires they agree, pinning the reference outcome first. + /// + /// Whether the reference build must exit with 0. + /// The CLI arguments. + /// The reference build's result, for callers that need to inspect it further. + private static async Task AssertParityAsync(bool expectSuccess, params string[] arguments) { var (native, reference) = await RunBothAsync(arguments); AssertReferenceOutcome(reference, expectSuccess); AssertSameResult(native, reference); + return reference; } /// /// Compares a diagram command twice: printed to the console, then written with --output. /// Both builds write to the same path one after the other, because the "Saved to" line on /// standard error names the path and Spectre wraps it at the console width, so two different - /// paths could not be compared exactly. + /// paths could not be compared exactly. Both runs must produce non-empty output, so a case where + /// both builds silently print or write nothing cannot pass as parity. /// /// The CLI arguments, without --output. private async Task AssertDiagramParityAsync(params string[] arguments) { - await AssertParityAsync(expectSuccess: true, arguments); + var console = await AssertParityAsync(expectSuccess: true, arguments); + console.StandardOutput.Should().NotBeNullOrWhiteSpace("the console run must print a diagram"); var outputPath = Path.Combine(_temp.DirectoryPath, "output", "diagram.mmd"); string[] withOutput = [.. arguments, "--output", outputPath]; @@ -626,6 +650,7 @@ public sealed partial class CliParityTests : IDisposable var reference = await ProcessRunner.RunAsync(SmokeEnvironment.CliReference, withOutput); AssertReferenceOutcome(reference, expectSuccess: true); var referenceFile = await File.ReadAllBytesAsync(outputPath); + referenceFile.Should().NotBeEmpty("the --output run must write a non-empty diagram file"); AssertSameResult(native, reference); nativeFile.Should().Equal(referenceFile, "the native build must write the same diagram bytes"); @@ -675,7 +700,10 @@ public sealed partial class CliParityTests : IDisposable /// /// Drops the wall-clock "Analysis time" row and collapses runs of spaces, because the timing - /// value's width can shift the table's padding. + /// value's width can shift the table's padding. ANSI colour escapes (Spectre's GitHub Actions + /// enricher sets Capabilities.Ansi = true when GITHUB_ACTIONS=true, and child + /// processes inherit that variable) are stripped only to decide which line to drop, so colour + /// output on the other rows is still compared exactly. /// /// The captured standard output of stats. /// The output without timing, for comparison. @@ -683,12 +711,16 @@ public sealed partial class CliParityTests : IDisposable { return string.Join('\n', output.ReplaceLineEndings("\n") .Split('\n') - .Where(line => !line.TrimStart().StartsWith("Analysis time", StringComparison.Ordinal)) + .Where(line => !AnsiEscape().Replace(line, string.Empty).TrimStart() + .StartsWith("Analysis time", StringComparison.Ordinal)) .Select(line => RunOfSpaces().Replace(line, " "))); } [GeneratedRegex(" {2,}")] private static partial Regex RunOfSpaces(); + + [GeneratedRegex(@"\x1B\[[0-9;?]*[A-Za-z]")] + private static partial Regex AnsiEscape(); } ``` @@ -862,7 +894,7 @@ Expected: `Passed: 16`. The binary is about 36.6 MB. unset PROJGRAPH_SMOKE_CLI_NATIVE PROJGRAPH_SMOKE_CLI_REFERENCE PROJGRAPH_SMOKE_MCP_NATIVE PROJGRAPH_SMOKE_MCP_REFERENCE dotnet build ProjGraph.slnx -c Release dotnet test tests/ProjGraph.Tests.Integration.Cli -c Release --no-build -git grep -n "PublishAot" -- '*.csproj' '*.props' +git grep -n "The native and reference servers shared by this class. public sealed class McpParityTests(McpServerPair servers) : IClassFixture { + /// + /// The time limit for one MCP request. The SDK's own timeout (2.2.0) only covers initialization, + /// so without this a deadlocked native or reference server would hang the test instead of + /// failing it. + /// + private static readonly TimeSpan RequestTimeout = TimeSpan.FromMinutes(2); + [AotSmokeFact] public async Task Initialize_ShouldAdvertiseTheSameServer() { @@ -988,13 +1027,15 @@ public sealed class McpParityTests(McpServerPair servers) : IClassFixture client.ListToolsAsync(new ListToolsRequestParams())); + await AssertSameResultAsync((client, cancellationToken) => + client.ListToolsAsync(new ListToolsRequestParams(), cancellationToken)); } [AotSmokeFact] public async Task ListPrompts_ShouldMatchReference() { - await AssertSameResultAsync(client => client.ListPromptsAsync(new ListPromptsRequestParams())); + await AssertSameResultAsync((client, cancellationToken) => + client.ListPromptsAsync(new ListPromptsRequestParams(), cancellationToken)); } [AotSmokeFact] @@ -1003,14 +1044,16 @@ public sealed class McpParityTests(McpServerPair servers) : IClassFixture through McpJsonContext. var path = SmokeEnvironment.GetRootPath("samples/classdiagram/complex-hierarchy/Domain/Models/CEO.cs"); - await AssertSameResultAsync(client => client.GetPromptAsync( - "class_structure_review", new Dictionary { ["path"] = path })); + await AssertSameResultAsync((client, cancellationToken) => client.GetPromptAsync( + "class_structure_review", new Dictionary { ["path"] = path }, + cancellationToken: cancellationToken)); } [AotSmokeFact] public async Task ReadWelcomeResource_ShouldMatchReference() { - await AssertSameResultAsync(client => client.ReadResourceAsync(new Uri("projgraph://welcome"))); + await AssertSameResultAsync((client, cancellationToken) => + client.ReadResourceAsync(new Uri("projgraph://welcome"), cancellationToken: cancellationToken)); } [AotSmokeFact] @@ -1068,19 +1111,21 @@ public sealed class McpParityTests(McpServerPair servers) : IClassFixture AssertSameResultAsync(Func> request) + private async Task AssertSameResultAsync(Func> request) { var (native, reference) = await servers.GetClientsAsync(); + using var timeout = new CancellationTokenSource(RequestTimeout); - var referenceResult = await request(reference); - AssertSameJson(await request(native), referenceResult); + var referenceResult = await request(reference, timeout.Token); + AssertSameJson(await request(native, timeout.Token), referenceResult); return referenceResult; } private async Task AssertSameToolResultAsync( string toolName, Dictionary arguments) { - var reference = await AssertSameResultAsync(client => client.CallToolAsync(toolName, arguments)); + var reference = await AssertSameResultAsync((client, cancellationToken) => + client.CallToolAsync(toolName, arguments, cancellationToken: cancellationToken)); // Pins the reference outcome, so an error both builds return identically cannot pass as parity. reference.IsError.Should().NotBe(true, JoinText(reference)); @@ -1236,7 +1281,7 @@ Append to `.github/workflows/ci.yml`, under `jobs:` after the `build` job (same - [ ] **Step 2: Run the job's commands locally from a clean clone** -This proves the YAML's commands work from scratch, with no leftovers from earlier tasks. A clone only sees committed state, so commit the workflow first: +This proves the YAML's commands work from scratch, with no leftovers from earlier tasks. A clone only sees committed state, so commit the workflow first. `GITHUB_ACTIONS=true` is exported alongside the four `PROJGRAPH_SMOKE_*` variables because Spectre.Console's GitHub enricher turns on ANSI output under that variable, and child processes inherit it; that behaviour must be exercised locally, not just on the runner: ```bash git add .github/workflows/ci.yml && git commit -m "ci: add the aot-smoke job" @@ -1246,6 +1291,7 @@ dotnet restore ProjGraph.slnx dotnet build ProjGraph.slnx --no-restore --configuration Release dotnet publish src/ProjGraph.Cli --configuration Release --runtime linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build --output artifacts/native/cli dotnet publish src/ProjGraph.Mcp --configuration Release --runtime linux-x64 -p:PublishAot=true --artifacts-path artifacts/aot-build --output artifacts/native/mcp +GITHUB_ACTIONS=true \ PROJGRAPH_SMOKE_REQUIRED=true \ PROJGRAPH_SMOKE_CLI_NATIVE=artifacts/native/cli/ProjGraph.Cli \ PROJGRAPH_SMOKE_CLI_REFERENCE=src/ProjGraph.Cli/bin/Release/net10.0/ProjGraph.Cli.dll \ @@ -1336,5 +1382,5 @@ This is a repository-settings change, so it belongs to the user (or run it only ## Exit criteria (spec, PR 2) - `aot-smoke` is green on the PR and marked required on `develop`. -- No csproj or props file sets `PublishAot` (`git grep -n PublishAot -- '*.csproj' '*.props'` is empty). +- No csproj or props file sets `PublishAot` (`git grep -n " Date: Mon, 14 Sep 2026 22:50:00 +0200 Subject: [PATCH 09/10] ci: build ProjGraph.slnx explicitly for CodeQL CodeQL autobuild selected the new tests/ProjGraph.Tests.Smoke.Aot/Fixtures/legacy.sln fixture instead of ProjGraph.slnx, compiled nothing, and failed. Manual build mode builds the real solution regardless of fixture files. Co-Authored-By: Claude Opus 5 (1M context) --- .github/workflows/codeql.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index b88a041..cf5b642 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -33,7 +33,12 @@ jobs: uses: github/codeql-action/init@v4 with: languages: csharp - build-mode: autobuild + build-mode: manual + + # Autobuild picks test fixture .sln files (e.g. tests/ProjGraph.Tests.Smoke.Aot/Fixtures/legacy.sln) + # over ProjGraph.slnx, so the real solution is built explicitly. + - name: Build + run: dotnet build ProjGraph.slnx --configuration Release - name: Perform CodeQL Analysis uses: github/codeql-action/analyze@v4 From 685a29ff925cdbc15c2d28bbf3e33af3a2cb706b Mon Sep 17 00:00:00 2001 From: = Date: Mon, 14 Sep 2026 22:55:26 +0200 Subject: [PATCH 10/10] fix(smoke): observe process reads on timeout and time MCP requests separately - ProcessRunner: the stdout/stderr reads no longer take the timeout token. On a timeout the process tree is killed, its exit awaited, and both reads awaited, so the tasks are observed and the captured stderr is included in the TimeoutException. - McpParityTests: the reference and native requests each get their own 2-minute CancellationTokenSource instead of sharing one budget. Addresses Copilot review comments on #192. Co-Authored-By: Claude Opus 5 (1M context) --- .../Helpers/ProcessRunner.cs | 15 ++++++++++---- .../McpParityTests.cs | 20 ++++++++++++++++--- 2 files changed, 28 insertions(+), 7 deletions(-) diff --git a/tests/ProjGraph.Tests.Smoke.Aot/Helpers/ProcessRunner.cs b/tests/ProjGraph.Tests.Smoke.Aot/Helpers/ProcessRunner.cs index 7555442..073e368 100644 --- a/tests/ProjGraph.Tests.Smoke.Aot/Helpers/ProcessRunner.cs +++ b/tests/ProjGraph.Tests.Smoke.Aot/Helpers/ProcessRunner.cs @@ -46,18 +46,25 @@ public static async Task RunAsync(SmokeCommand command, IReadOnly ?? throw new InvalidOperationException($"Could not start {command.FileName}."); using var timeout = new CancellationTokenSource(Timeout); - // Both streams are drained concurrently so a full pipe buffer can never block the child. - var standardOutput = process.StandardOutput.ReadToEndAsync(timeout.Token); - var standardError = process.StandardError.ReadToEndAsync(timeout.Token); + // Both streams are drained concurrently so a full pipe buffer can never block the child. The + // reads take no token: they finish when the process exits, including after a timeout kill. + var standardOutput = process.StandardOutput.ReadToEndAsync(); + var standardError = process.StandardError.ReadToEndAsync(); try { await process.WaitForExitAsync(timeout.Token); } catch (OperationCanceledException) { + // Killing the tree closes the pipes, so both reads complete and are observed here, and + // whatever the process wrote before it hung ends up in the failure message. process.Kill(entireProcessTree: true); + await process.WaitForExitAsync(); + await standardOutput; + var partialError = await standardError; throw new TimeoutException( - $"'{command.FileName} {string.Join(' ', arguments)}' did not exit within {Timeout}."); + $"'{command.FileName} {string.Join(' ', command.LeadingArguments.Concat(arguments))}' " + + $"did not exit within {Timeout}; stderr:\n{partialError}"); } return new ProcessResult(process.ExitCode, await standardOutput, await standardError); diff --git a/tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs b/tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs index 7c50c3f..194f464 100644 --- a/tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs +++ b/tests/ProjGraph.Tests.Smoke.Aot/McpParityTests.cs @@ -122,13 +122,27 @@ public async Task GetErd_ShouldMatchReference() private async Task AssertSameResultAsync(Func> request) { var (native, reference) = await servers.GetClientsAsync(); - using var timeout = new CancellationTokenSource(RequestTimeout); - var referenceResult = await request(reference, timeout.Token); - AssertSameJson(await request(native, timeout.Token), referenceResult); + var referenceResult = await SendWithTimeoutAsync(reference, request); + AssertSameJson(await SendWithTimeoutAsync(native, request), referenceResult); return referenceResult; } + /// + /// Sends one request with its own , so a slow reference call never + /// shortens the native call's budget. + /// + /// The protocol result type. + /// The server to send the request to. + /// The request to send. + /// The server's result. + private static async Task SendWithTimeoutAsync( + McpClient client, Func> request) + { + using var timeout = new CancellationTokenSource(RequestTimeout); + return await request(client, timeout.Token); + } + private async Task AssertSameToolResultAsync( string toolName, Dictionary arguments) {