Skip to content

Parallelize build-time shader compilation in the descriptor generators - #932

Open
rickbrew wants to merge 4 commits into
Sergio0694:mainfrom
rickbrew:feature/parallel-shader-compilation
Open

rickbrew wants to merge 4 commits into
Sergio0694:mainfrom
rickbrew:feature/parallel-shader-compilation

Conversation

@rickbrew

@rickbrew rickbrew commented Aug 31, 2026 •

Copy link
Copy Markdown
Collaborator

via Claude. I have not yet reviewed this code yet.

Closes #931

Description

Moves FXC/DXC compilation out of the ForAttributeWithMetadataName transform callbacks (which the incremental driver invokes sequentially, serializing all shader compilations) and into a dedicated node per generator that compiles every shader in the project in parallel. Applies to both D2DPixelShaderDescriptorGenerator and ComputeShaderDescriptorGenerator; the shared HlslBytecodeSyntaxProcessor carries the new logic for both.

Pipeline shape (identical in both generators):

  1. Transform: unchanged up to building HlslBytecodeInfoKey, but no longer compiles. It also captures the info needed to later synthesize the compile diagnostics (HlslBytecodeDiagnosticsInfo: type name, type location, [D2DRequiresDoublePrecisionSupport] presence + location), since symbols must not escape the transform.
  2. Warm-up node: Select(key).Collect() → CompileAllInParallel(keys, token), a Parallel.ForEach over the distinct keys into the existing DynamicCache (which is already thread-safe and needed no changes; the FXC and DXC wrappers are also both safe for concurrent use, DXC via its existing [ThreadStatic] instance).
  3. Join node: Combine with the warm-up node's output (this edge is what guarantees the ordering, via the dataflow graph rather than driver scheduling assumptions), then a Select that fetches the bytecode (a guaranteed cache hit), synthesizes the deferred diagnostics, and drops the captured diagnostics info from the model.

Locations are captured by value in the new LocationInfo model (file path + TextSpan + LinePositionSpan, materialized via Location.Create(path, span, lineSpan)), deliberately not as a SyntaxTree reference: this keeps transform outputs equatable across unrelated edits (comment-only edits still produce value-equal models), and avoids ever holding stale trees in cached models. All existing incrementality tests pass unchanged.

Numbers from Paint.NET (warm --no-incremental Release x64 rebuilds, second of two runs, vs. stock 3.2.0):

Project serial parallel delta
Effects.Gpu (51 shaders) 21.4s 13.5s -7.9s (-37%)
Windows.Direct2D1.Managed 12.6s 9.0s -3.6s (-29%)
Windows.Framework (546 shaders) 11.2s 7.6s -3.6s (-32%)

API breakdown

No public API changes — internal generator restructuring only.

Additional context

  • Generated output verified byte-for-byte identical to 3.2.0 across Paint.NET's ~660 shaders (329 emitted .g.cs files compared, generator version stamp aside), and the full Paint.NET solution builds clean against a locally packed build of this branch.
  • ComputeSharp.D2D1.Tests.SourceGenerators (113) and ComputeSharp.Tests.SourceGenerators (110) all pass. The WinUI source generator test project currently fails to restore for me on stock main as well (NU1603 warnings-as-errors from feed version drift), so I couldn't run that suite locally — but it only exercises CanvasEffectPropertyGenerator, which this PR doesn't touch.
  • D2DPixelShaderSourceGenerator (raw HLSL string literals) intentionally keeps compiling inline; it can get the same treatment in a follow-up if you want it.
  • As mentioned in Parallelize build-time shader compilation in the descriptor generators #931, I'd like to follow up separately with an opt-in persistent on-disk bytecode cache proposal, which composes with this change (parallelism helps the first compile; a disk cache would make unchanged CLI rebuilds nearly free).

🤖 Generated with Claude Code

Move FXC/DXC compilation out of the ForAttributeWithMetadataName transform
callbacks (which the incremental driver invokes sequentially) into a dedicated
node that warms the shared bytecode cache for all shaders in parallel. A join
node (ordered after the warm-up by its dataflow edge) then folds each compiled
shader back into its model and synthesizes the compile diagnostics, using
location info captured by value during the transform.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rickbrew and others added 3 commits September 18, 2026 09:21
The deferred compile diagnostics (CMPSD2D0033/0034/0080/0081 and their
D3D12 counterparts) were reported with no location, because their by-value
LocationInfo produced an external-file Location whose SyntaxTree is null,
which DiagnosticInfo then dropped entirely. This also broke #pragma and
[SuppressMessage] suppression for the one warning-severity diagnostic.

The transform now returns the shader model alongside a separate
HlslBytecodeDiagnosticsInfo holding the real Location values, which only
travels up to the join node that creates the diagnostics. The models are
back to their original shape, and the "Execute" tracking name moves to the
join, which is the node producing the complete model with diagnostics (all
existing incrementality expectations hold unchanged).

Also unwrap a single unexpected exception from the parallel warm-up (with
ExceptionDispatchInfo, preserving its stack trace) so that CS8785 reports
the actual exception type, and add tests asserting diagnostic locations.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@rickbrew
rickbrew marked this pull request as ready for review September 18, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parallelize build-time shader compilation in the descriptor generators

1 participant