[cdac] Pipeline refactor + StackWalk doc cleanup + DataGenerator build fix

docs/design/datacontracts/StackWalk.md

@@ -626,26 +626,7 @@ At each frame yielded by `Filter`, the walk determines whether to scan for GC re
 - **PrestubMethodFrame / CallCountingHelperFrame**: Use signature-based scanning.
 - Other frame types: No GC roots to report.
 
-See [GCRefMap Format and Resolution](#gcrefmap-format-and-resolution) for the GCRefMap scanning path and [Signature-Based Scanning](#signature-based-scanning) for the signature decoding path.
-
-### Signature-Based Scanning (currently deferred)
-
-When a transition frame's calling convention is not described by a precomputed GCRefMap (`PrestubMethodFrame`, `CallCountingHelperFrame`, and the fallback path for `StubDispatchFrame`/`ExternalMethodFrame`), the native runtime classifies caller-stack arguments by decoding the callee's method signature (`TransitionFrame::PromoteCallerStack` in `src/coreclr/vm/frames.cpp`).
-
-The cDAC does **not** currently port this scan. `GcScanner.PromoteCallerStack` is a stub that records the frame as deferred and returns without enumerating any refs:
-
-```csharp
-private static void PromoteCallerStack(TargetPointer frameAddress, GcScanContext scanContext)
-{
-    scanContext.RecordDeferredFrame(frameAddress);
-}
-```
-
-`RecordDeferredFrame` (on `GcScanContext`) appends a sentinel `StackRefData` entry with `Flags = GcScanFlags.CDAC_DEFERRED_FRAME (0x40000000)` and `Source = frameAddress`. The sentinel has no real GC ref payload; downstream consumers (e.g. the cDAC stress harness in `src/coreclr/vm/cdacstress.cpp`) can detect it and treat the missing refs at that frame as expected gaps rather than cDAC bugs. See [tests/StressTests/known-issues.md](../../../src/native/managed/cdac/tests/StressTests/known-issues.md) for the stress framework's handling and the tracking work to re-enable the scan.
-
-The `GcSignatureTypeProvider` class remains in the tree as the scaffolding the eventual port will use; it has no callers while `PromoteCallerStack` is stubbed.
-
-Tracking work to re-enable the scan: it requires porting `ArgIterator` behind an `ICallingConvention` contract. Once that lands, `PromoteCallerStack` will fan out into the signature-decoding algorithm (reserved-slot computation, signature walk, slot reporting) that mirrors the native version. See also [dotnet/runtime#127765](https://github.com/dotnet/runtime/issues/127765).
+See [GCRefMap Format and Resolution](#gcrefmap-format-and-resolution) for the GCRefMap scanning path. The signature-based scanning fallback (`PromoteCallerStack`) is currently a stub awaiting an `ICallingConvention` contract port; the stress harness handles the resulting gaps via a deferred-frame sentinel (see [tests/StressTests/known-issues.md](../../../src/native/managed/cdac/tests/StressTests/known-issues.md) and tracking issue [dotnet/runtime#127765](https://github.com/dotnet/runtime/issues/127765)).
 
 ### GCRefMap Format and Resolution
 

eng/Subsets.props

@@ -531,6 +531,7 @@
 
   <ItemGroup Condition="$(_subset.Contains('+tools.cdactests+'))">
     <ProjectToBuild Include="$(SharedNativeRoot)managed\cdac\tests\UnitTests\Microsoft.Diagnostics.DataContractReader.Tests.csproj" Test="true" Category="tools"/>
+    <ProjectToBuild Include="$(SharedNativeRoot)managed\cdac\tests\DataGenerator\Microsoft.Diagnostics.DataContractReader.DataGeneratorTests.csproj" Test="true" Category="tools"/>
   </ItemGroup>
 
   <ItemGroup Condition="$(_subset.Contains('+tools.cdacdumptests+'))">

eng/pipelines/cdac/cdac-helix-test-leg.yml

@@ -0,0 +1,106 @@
+# cdac-helix-test-leg.yml -- wraps the matrix + per-platform job + artifact
+# download + prepare + send-to-helix + fail-on-error boilerplate shared by
+# the three cDAC dump-style legs in runtime-diagnostics.yml:
+#
+#   CdacDumpTests      (single-leg mode) -- runs DumpTests directly
+#   CdacXPlatDumpGen   -- generates dumps only (DumpOnly=true)
+#   CdacXPlatDumpTests -- runs DumpTests against dumps from all source platforms
+#
+# All three: download the shared CdacBuild artifact, prepare a Helix payload,
+# send to Helix, fail the job if Helix reports failures. They differ in:
+#   - which platform list they fan out over
+#   - which Helix .proj they send
+#   - extra prepare-cdac-helix-steps parameters (skipDebuggeeCopy, etc.)
+#   - extra postBuildSteps inserted between prepare and send (xplat-test needs
+#     to download all source platforms' dumps and extract the tars first)
+#   - whether to publish dump artifacts at the end, and on what condition
+#
+# Stress tests are not collapsed into this template -- they use a different
+# prepare template (prepare-cdac-stress-helix-steps.yml) and have unique
+# crash-dump-symbolication artifacts, so they stay inline.
+
+parameters:
+- name: nameSuffix
+  type: string
+- name: platforms
+  type: object
+- name: sendDisplayName
+  type: string
+- name: sendParams
+  type: string
+- name: prepareParameters
+  type: object
+  default: {}
+- name: dependsOnCdacBuild
+  type: boolean
+  default: true
+- name: extraStepsBeforeSend
+  type: stepList
+  default: []
+- name: failDisplayName
+  type: string
+  default: 'Fail on test errors'
+- name: failErrorMessage
+  type: string
+  default: 'One or more cDAC Helix failures were detected. Failing the job.'
+- name: publishDumpsArtifactName
+  type: string
+  default: ''   # if empty, no dump-artifact publish step is emitted
+- name: publishDumpsCondition
+  type: string
+  default: 'and(always(), ne(variables[''Agent.JobStatus''], ''Succeeded''))'
+- name: extraDependsOn
+  type: object
+  default: []   # explicit job names to add to dependsOn (combined with dependsOnGlobalBuilds)
+
+jobs:
+- template: /eng/pipelines/common/platform-matrix.yml
+  parameters:
+    jobTemplate: /eng/pipelines/common/global-build-job.yml
+    buildConfig: release
+    platforms: ${{ parameters.platforms }}
+    shouldContinueOnError: true
+    jobParameters:
+      nameSuffix: ${{ parameters.nameSuffix }}
+      # Tiny rebuild that initializes .dotnet and links the test csproj
+      # against the downloaded CdacBuild artifact; coreclr + libs + tools.cdac
+      # are not rebuilt.
+      buildArgs: -s tools.cdacdumptests -c $(_BuildConfig) -rc checked -lc $(_BuildConfig) /p:SkipDumpVersions=net10.0
+      timeoutInMinutes: 180
+      ${{ if parameters.dependsOnCdacBuild }}:
+        dependsOnGlobalBuilds:
+        - nameSuffix: CdacBuild
+      ${{ if ne(length(parameters.extraDependsOn), 0) }}:
+        dependsOn: ${{ parameters.extraDependsOn }}
+      preBuildSteps:
+      - template: /eng/pipelines/cdac/download-cdac-build-artifact.yml
+      postBuildSteps:
+      - template: /eng/pipelines/cdac/prepare-cdac-helix-steps.yml
+        parameters: ${{ parameters.prepareParameters }}
+      - ${{ each step in parameters.extraStepsBeforeSend }}:
+        - ${{ step }}
+      - template: /eng/pipelines/common/templates/runtimes/send-to-helix-inner-step.yml
+        parameters:
+          displayName: ${{ parameters.sendDisplayName }}
+          sendParams: ${{ parameters.sendParams }}
+          environment:
+            _Creator: dotnet-bot
+            SYSTEM_ACCESSTOKEN: $(System.AccessToken)
+            NUGET_PACKAGES: $(Build.SourcesDirectory)$(dir).packages
+            ${{ if eq(parameters.nameSuffix, 'CdacXPlatDumpTest') }}:
+              SourcePlatforms: ${{ join(';', parameters.platforms) }}
+      - ${{ if ne(parameters.publishDumpsArtifactName, '') }}:
+        - task: PublishPipelineArtifact@1
+          inputs:
+            targetPath: $(Build.SourcesDirectory)/artifacts/helixresults
+            artifactName: ${{ parameters.publishDumpsArtifactName }}
+          displayName: 'Publish Dump Artifacts'
+          condition: ${{ parameters.publishDumpsCondition }}
+          continueOnError: true
+      - pwsh: |
+          if ("$(Agent.JobStatus)" -ne "Succeeded") {
+            Write-Error "${{ parameters.failErrorMessage }}"
+            exit 1
+          }
+        displayName: ${{ parameters.failDisplayName }}
+        condition: always()

eng/pipelines/cdac/download-cdac-build-artifact.yml

@@ -0,0 +1,13 @@
+# download-cdac-build-artifact.yml -- per-platform download of the
+# CdacBuildArtifacts_<plat> tarball published by the CdacBuild job.
+# Extracts into artifacts/bin/, so every test leg sees the same coreclr +
+# libs + cDAC + test-csproj outputs that CdacBuild produced. Used as a
+# preBuildStep by every test leg in the Cdac stage (and by CdacXPlatDumpTests).
+
+steps:
+- template: /eng/pipelines/common/download-artifact-step.yml
+  parameters:
+    artifactName: CdacBuildArtifacts_$(osGroup)$(osSubgroup)_$(archType)
+    artifactFileName: CdacBuildArtifacts_$(osGroup)$(osSubgroup)_$(archType)$(archiveExtension)
+    unpackFolder: $(Build.SourcesDirectory)/artifacts/bin
+    displayName: 'cDAC Shared Build Artifacts'

eng/pipelines/cdac/prepare-cdac-helix-steps.yml

@@ -7,23 +7,57 @@ parameters:
   buildDebuggees: true
   skipDebuggeeCopy: false
   runtimeConfiguration: 'Checked'
+  # Configuration the cDAC managed assemblies are built at during payload
+  # prep. The dump tests load the managed cDAC directly via ProjectReference,
+  # so passing /p:Configuration=Debug here makes MSBuild walk the
+  # DumpTests project graph and produce Debug-built cDAC dlls in
+  # DumpTests's bin folder, which PrepareHelixPayload then copies into
+  # the Helix payload. Defaults to $(_BuildConfig) for back-compat.
+  cdacTestConfig: '$(_BuildConfig)'
+  # Libraries configuration is pinned independently of cdacTestConfig so
+  # an /p:Configuration=Debug override here never causes MSBuild to look
+  # for Debug-built libraries (which we don't ship in CI).
+  librariesConfiguration: '$(_BuildConfig)'
 
 steps:
+# Pre-flight: confirm a libraries build exists at the expected configuration.
+# Detects misalignment between what CdacBuild produced and what we're about
+# to consume during payload prep.
+- pwsh: |
+    $libsDir = "$(Build.SourcesDirectory)/artifacts/bin/runtime"
+    if (-not (Test-Path $libsDir)) {
+        Write-Error "Libraries output directory not found at '$libsDir' -- did CdacBuild publish artifacts/bin?"
+        exit 1
+    }
+    $configMatches = Get-ChildItem -Directory $libsDir -ErrorAction SilentlyContinue |
+        Where-Object { $_.Name -like "*-${{ parameters.librariesConfiguration }}" }
+    if (-not $configMatches) {
+        Write-Error "No libraries built at configuration '${{ parameters.librariesConfiguration }}' under $libsDir. Found: $((Get-ChildItem -Directory $libsDir | Select-Object -ExpandProperty Name) -join ', ')"
+        exit 1
+    }
+    Write-Host "Libraries configuration '${{ parameters.librariesConfiguration }}' verified at:"
+    $configMatches | ForEach-Object { Write-Host "  $($_.FullName)" }
+  displayName: 'Verify libraries built at ${{ parameters.librariesConfiguration }}'
+
 - ${{ if parameters.buildDebuggees }}:
   - script: $(Build.SourcesDirectory)$(dir).dotnet$(dir)dotnet$(exeExt) msbuild
       $(Build.SourcesDirectory)/src/native/managed/cdac/tests/DumpTests/Microsoft.Diagnostics.DataContractReader.DumpTests.csproj
       /t:BuildDebuggeesOnly
-      /p:Configuration=$(_BuildConfig)
-      /p:TargetArchitecture=$(archType)
+      /p:Configuration=${{ parameters.cdacTestConfig }}
+      /p:ToolsConfiguration=${{ parameters.cdacTestConfig }}
+      /p:LibrariesConfiguration=${{ parameters.librariesConfiguration }}
       /p:RuntimeConfiguration=${{ parameters.runtimeConfiguration }}
+      /p:TargetArchitecture=$(archType)
       -bl:$(Build.SourcesDirectory)/artifacts/log/BuildDebuggees.binlog
     displayName: 'Build Debuggees'
 
 - script: $(Build.SourcesDirectory)$(dir).dotnet$(dir)dotnet$(exeExt) build
     $(Build.SourcesDirectory)/src/native/managed/cdac/tests/DumpTests/Microsoft.Diagnostics.DataContractReader.DumpTests.csproj
     /p:PrepareHelixPayload=true
     /p:SkipDebuggeeCopy=${{ parameters.skipDebuggeeCopy }}
-    /p:Configuration=$(_BuildConfig)
+    /p:Configuration=${{ parameters.cdacTestConfig }}
+    /p:ToolsConfiguration=${{ parameters.cdacTestConfig }}
+    /p:LibrariesConfiguration=${{ parameters.librariesConfiguration }}
     /p:HelixPayloadDir=$(Build.SourcesDirectory)/artifacts/helixPayload/cdac
     /p:SkipDumpVersions=net10.0
     -bl:$(Build.SourcesDirectory)/artifacts/log/DumpTestPayload.binlog

eng/pipelines/diagnostics/sos-test-leg.yml

@@ -0,0 +1,75 @@
+# sos-test-leg.yml -- wraps the matrix + job + dep + artifact download +
+# standard postBuildSteps pattern used by the SOS-style test legs (cDAC,
+# cDAC_no_fallback, DAC) that run on windows_x64. Each of those legs only
+# differs in its `name`, `useCdac`, and `noFallback` value.
+#
+# AzDO tasks workaround:
+#   PublishTestResults v2.270.0 garbles parameterized xUnit test names
+#   containing dots. Unsetting AllowPtrToDetectTestRunRetryFiles restores
+#   v2.262.0 behavior. Tracked at microsoft/azure-pipelines-tasks#21871.
+
+parameters:
+- name: name
+  type: string
+- name: useCdac
+  type: boolean
+  default: false
+- name: noFallback
+  type: boolean
+  default: false
+
+jobs:
+- template: /eng/pipelines/common/platform-matrix.yml
+  parameters:
+    jobTemplate: /eng/pipelines/diagnostics/runtime-diag-job.yml
+    buildConfig: release
+    platforms:
+    - windows_x64
+    variables:
+      - name: AllowPtrToDetectTestRunRetryFiles
+        value: false
+    jobParameters:
+      name: ${{ parameters.name }}
+      useCdac: ${{ parameters.useCdac }}
+      noFallback: ${{ parameters.noFallback }}
+      testInterpreter: true
+      methodFilter: SOS*
+      isOfficialBuild: ${{ variables.isOfficialBuild }}
+      liveRuntimeDir: $(Build.SourcesDirectory)/artifacts/runtime
+      timeoutInMinutes: 360
+      dependsOn:
+      - build_windows_x64_release_AllSubsets_CoreCLR
+      preBuildSteps:
+      - template: /eng/pipelines/common/download-artifact-step.yml
+        parameters:
+          artifactName: BuildArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)_coreclr
+          artifactFileName: BuildArtifacts_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)_coreclr$(archiveExtension)
+          unpackFolder: $(Build.SourcesDirectory)/artifacts/runtime
+          displayName: 'Runtime Build Artifacts'
+      postBuildSteps:
+      - task: PublishTestResults@2
+        inputs:
+          testResultsFormat: xUnit
+          testResultsFiles: '**/*.xml'
+          searchFolder: '$(Build.SourcesDirectory)/artifacts/TestResults'
+          testRunTitle: 'Tests $(_PhaseName)'
+          failTaskOnFailedTests: true
+          publishRunAttachments: true
+          mergeTestResults: true
+          buildConfiguration: $(_BuildConfig)
+        continueOnError: true
+        condition: always()
+      - task: PublishPipelineArtifact@1
+        inputs:
+          targetPath: '$(Build.SourcesDirectory)/artifacts/tmp/$(_BuildConfig)/dumps'
+          artifactName: 'Dumps_${{ parameters.name }}_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)_Attempt$(System.JobAttempt)'
+        displayName: 'Publish Crash Dumps'
+        continueOnError: true
+        condition: failed()
+      - task: PublishPipelineArtifact@1
+        inputs:
+          targetPath: '$(Build.SourcesDirectory)/artifacts/TestResults'
+          artifactName: 'TestResults_${{ parameters.name }}_$(osGroup)$(osSubgroup)_$(archType)_$(_BuildConfig)_Attempt$(System.JobAttempt)'
+        displayName: 'Publish Test Results and SOS Logs'
+        continueOnError: true
+        condition: failed()