Skip to content

Add TFM-aware buildTransitive->build forwarding guidance to msbuild skills#836

Open
YuliiaKovalova wants to merge 6 commits into
dotnet:mainfrom
YuliiaKovalova:ykovalova/msbuild-tfm-forwarding
Open

Add TFM-aware buildTransitive->build forwarding guidance to msbuild skills#836
YuliiaKovalova wants to merge 6 commits into
dotnet:mainfrom
YuliiaKovalova:ykovalova/msbuild-tfm-forwarding

Conversation

@YuliiaKovalova

Copy link
Copy Markdown
Member

Summary

Captures the lesson learned in microsoft/testfx#9431 about TFM-aware buildTransitive/build/ forwarding.

A NuGet build-extension buildTransitive/*.props should forward through the corresponding build/*.props (ownership chain buildTransitive → build → shared) rather than importing buildMultiTargeting/ directly. The subtlety: build/*.props are frequently packed per-TFM under build/<tfm>/, while buildMultiTargeting/ is not. So a forwarder living in buildTransitive/<tfm>/ must include the TFM segment — and must derive <tfm> from the file's own folder, not $(TargetFramework), because NuGet nearest-match can serve a consumer a different asset folder. Dropping the segment resolves to a non-existent package-root build/Pkg.props and fails with MSB4019 for transitive consumers.

Changes

  • extension-points/SKILL.md — new "Forwarding chain: buildTransitive/build/ → shared" section, including the TFM-segment requirement and the derive-from-own-folder expression.
  • msbuild-antipatterns/SKILL.md (AP-13) — added the forwarding/TFM note (the other relevant build rule).
  • msbuild-code-review.agent.md — added a Category-4 review check for the forwarding chain + TFM-correct forwarding.

Reference: microsoft/testfx#9431 (Rule E-1).

…kills

buildTransitive/*.props should forward through the corresponding build/*.props (ownership chain buildTransitive -> build -> shared) rather than importing buildMultiTargeting/ directly. When build/ is packed per-TFM (build/<tfm>/), the forwarder must include the TFM segment and derive it from the file own folder, not $(TargetFramework) (NuGet nearest-match can serve a different asset folder), otherwise transitive consumers hit MSB4019.

Updates extension-points (new Forwarding chain section), msbuild-antipatterns AP-13, and the msbuild-code-review agent. Lesson learned from microsoft/testfx#9431.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 26, 2026 10:26

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds guidance to the dotnet-msbuild skill set to prevent a real-world packaging/build break: when build/ assets are packed per-TFM, buildTransitive/<tfm>/ forwarders must include the <tfm> segment and derive it from the forwarder’s own folder (not $(TargetFramework)), preserving the intended ownership chain buildTransitive → build → shared.

Changes:

  • Documented the recommended forwarding chain (buildTransitive/build/ → shared) and the TFM-aware forwarding pattern, including a concrete MSBuild derivation expression.
  • Extended AP-13 (“Import Without Exists() Guard”) with a TFM-aware forwarding note to prevent false positives and real restore-time asset mismatches.
  • Added a Category-4 code review checklist item to ensure reviewers check for correct transitive forwarding behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
plugins/dotnet-msbuild/skills/msbuild-antipatterns/SKILL.md Adds AP-13 guidance on TFM-aware buildTransitive/<tfm>/build/<tfm>/ forwarding to avoid MSB4019.
plugins/dotnet-msbuild/skills/extension-points/SKILL.md Introduces a “Forwarding chain” section with rationale, examples, and a safe <tfm> derivation expression.
plugins/dotnet-msbuild/agents/msbuild-code-review.agent.md Adds a Category-4 review check for correct forwarding chain and TFM derivation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread plugins/dotnet-msbuild/agents/msbuild-code-review.agent.md Outdated
Address review feedback on dotnet#836: spell out ".props/.targets forwarders" instead of the ambiguous `buildTransitive/*.props|targets`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@YuliiaKovalova

Copy link
Copy Markdown
Member Author

Addressed the review feedback in d38fb2e: replaced the ambiguous buildTransitive/*.props|targets with the clearer ".props/.targets forwarders" wording.

@github-actions github-actions Bot added the waiting-on-author PR state label label Jun 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

👋 @YuliiaKovalova — this PR has 1 unresolved review thread(s). When you're ready, please address the feedback and push an update; the triage bot will pick up the next state automatically. (Add the no-stale label to silence further pings.)

@github-actions github-actions Bot added pr-state/ready-for-eval PR is mergeable and awaiting evaluation and removed waiting-on-author PR state label labels Jun 26, 2026
github-actions Bot added a commit that referenced this pull request Jun 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

Skill Scenario Quality Skills Loaded Overfit Verdict
msbuild-antipatterns Review MSBuild files for anti-patterns and style issues 5.0/5 → 4.7/5 🔴 ✅ msbuild-antipatterns; tools: skill, glob / ✅ msbuild-antipatterns; tools: task, read_agent, glob, skill ✅ 0.11 [1]
msbuild-antipatterns Add a module to an F# project 5.0/5 → 5.0/5 ⚠️ NOT ACTIVATED ✅ 0.11 [2]
msbuild-antipatterns Fix broken file order causing FS0039 4.0/5 → 4.0/5 ⚠️ NOT ACTIVATED ✅ 0.11 [3]
msbuild-antipatterns Add a signature file to define public API 5.0/5 → 5.0/5 ⚠️ NOT ACTIVATED ✅ 0.11 [4]
extension-points Diagnose build extension point failures 3.0/5 → 5.0/5 🟢 ✅ extension-points; tools: skill, edit ✅ 0.09
extension-points Diagnose NuGet package and repo extension conflicts 3.0/5 → 3.0/5 ✅ extension-points; tools: skill, edit ✅ 0.09 [5]
extension-points Fix extension point anti-patterns 5.0/5 → 4.3/5 🔴 ✅ extension-points; tools: skill / ⚠️ NOT ACTIVATED ✅ 0.09 [6]

[1] ⚠️ High run-to-run variance (CV=102%) — consider re-running with --runs 5
[2] ⚠️ High run-to-run variance (CV=58%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -3.8% due to: tokens (97636 → 167146)
[3] (Plugin) Quality unchanged but weighted score is -3.9% due to: tokens (67509 → 117300)
[4] (Plugin) Quality unchanged but weighted score is -3.9% due to: tokens (70847 → 123924)
[5] ⚠️ High run-to-run variance (CV=354%) — consider re-running with --runs 5
[6] ⚠️ High run-to-run variance (CV=81%) — consider re-running with --runs 5

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 836 in dotnet/skills, download eval artifacts with gh run download 28239731107 --repo dotnet/skills --pattern "skill-validator-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/d38fb2e76f4b756d0a1f2ca81434bf71b582d10b/eng/skill-validator/src/docs/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

Condense the new extension-points Forwarding chain section (+26 -> +11 lines) and the AP-13 note by dropping the redundant non-TFM example and self-evident derivation explanation, keeping the chain rule, MSB4019 cause, and the TFM derivation expression. Lower token footprint addresses the skill-validator weighted-score token penalty without losing substance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 26, 2026 13:26
@YuliiaKovalova

Copy link
Copy Markdown
Member Author

Eval analysis + quality iteration

I downloaded the eval artifacts (run 28239731107) and analyzed results.json. Summary of findings and the fix I pushed (b45beec).

The failing verdicts are noise/overhead-driven, not content regressions

The pairwise A/B judgments (the more reliable signal — direct comparison of baseline vs skilled output) favor the skill on both "failing" scenarios:

  • extension-points / "Fix extension point anti-patterns": overallWinner: "tie""Both responses identified the same issues, applied the same fixes… essentially identical." The absolute scores (baseline 5.0 → skilled 4.3) are judge scoring variance — the skilled run's own rubric reasoning is all positive ("canonical patterns", "clear and correct"). This is pattern Skill idea: Trimming & AOT Readiness #8 (baseline already good, no headroom) + Skill idea: Source Generator / Analyzer Authoring #7 (judge regression on close calls — "usually noise, re-run") from InvestigatingResults.md. perRunScores CV = 81%.
  • msbuild-antipatterns / "Review MSBuild files": overallWinner: "skill" — the skilled output actually won ("Response B is slightly better… checked the fsharp subdirectory for completeness"). CV = 102%.
  • The F# "NOT ACTIVATED" scenarios fail purely on the token penalty (pattern Add unit-testing agents & skill #4) — inherent skill-loading overhead, weighted only 0.05.

What I changed

Even though the dips are noise, the extension-points "Forwarding chain" addition is irrelevant to the anti-pattern-fixing scenario, so its length was pure overhead/distraction there. I tightened it from +26 to +11 net lines (and trimmed the AP-13 note) — dropping the redundant non-TFM example and the self-evident derivation explanation, while keeping the chain rule, the MSB4019 cause, and the TFM-derivation expression. This directly reduces the token overhead flagged by pattern #4 without losing substance.

Recommendation

Given CV of 81–102%, a single run is dominated by LLM non-determinism. Re-running with more runs should stabilize the signal (the guide suggests 7–10 for noisy scenarios).

/evaluate --runs 7

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comment thread plugins/dotnet-msbuild/skills/msbuild-antipatterns/SKILL.md Outdated
Comment thread plugins/dotnet-msbuild/skills/extension-points/SKILL.md Outdated
Update the additional anti-patterns range to AP-16 through AP-22 (the reference doc now includes AP-22), and use a forward-slash build/MyPackage.props in the forwarding-chain prose to match the build/<tfm>/ convention used in the section.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@YuliiaKovalova

Copy link
Copy Markdown
Member Author

Addressed both review nits in 0b7f356: updated the range to "AP-16 through AP-22" (the reference doc now includes AP-22), and switched the prose to a forward-slash build/MyPackage.props to match the build/<tfm>/ convention in that section.

github-actions Bot added a commit that referenced this pull request Jun 26, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Skill Validation Results

Skill Scenario Quality Skills Loaded Overfit Verdict
msbuild-antipatterns Review MSBuild files for anti-patterns and style issues 4.7/5 → 5.0/5 🟢 ✅ msbuild-antipatterns; tools: skill, glob / ⚠️ NOT ACTIVATED ✅ 0.09 [1]
msbuild-antipatterns Add a module to an F# project 5.0/5 → 5.0/5 ⚠️ NOT ACTIVATED ✅ 0.09 [2]
msbuild-antipatterns Fix broken file order causing FS0039 4.0/5 → 4.0/5 ⚠️ NOT ACTIVATED ✅ 0.09 [3]
msbuild-antipatterns Add a signature file to define public API 5.0/5 → 5.0/5 ⚠️ NOT ACTIVATED ✅ 0.09 [4]
extension-points Diagnose build extension point failures 3.3/5 → 4.7/5 🟢 ✅ extension-points; tools: skill, edit / ✅ extension-points; tools: skill, edit, glob ✅ 0.09 [5]
extension-points Diagnose NuGet package and repo extension conflicts 3.0/5 → 3.0/5 ✅ extension-points; tools: skill, edit ✅ 0.09 [6]
extension-points Fix extension point anti-patterns 5.0/5 → 4.0/5 🔴 ✅ extension-points; tools: skill / ⚠️ NOT ACTIVATED ✅ 0.09 [7]

[1] ⚠️ High run-to-run variance (CV=175%) — consider re-running with --runs 5
[2] ⚠️ High run-to-run variance (CV=183%) — consider re-running with --runs 5. (Plugin) Quality unchanged but weighted score is -0.6% due to: tokens (113227 → 158388)
[3] (Plugin) Quality unchanged but weighted score is -4.1% due to: tokens (67452 → 117334)
[4] (Isolated) Quality unchanged but weighted score is -4.3% due to: tokens (66511 → 116725)
[5] ⚠️ High run-to-run variance (CV=105%) — consider re-running with --runs 5
[6] ⚠️ High run-to-run variance (CV=625%) — consider re-running with --runs 5
[7] (Plugin) Quality unchanged but weighted score is -3.0% due to: tokens (105191 → 166565)

Model: claude-opus-4.6 | Judge: claude-opus-4.6

🔍 Full Results - additional metrics and failure investigation steps

To investigate failures, paste this to your AI coding agent:

For PR 836 in dotnet/skills, download eval artifacts with gh run download 28242987115 --repo dotnet/skills --pattern "skill-validator-results-*" --dir ./eval-results, then fetch https://raw.githubusercontent.com/dotnet/skills/0b7f356b217e31088715e6ad43a9eeb4497e4486/eng/skill-validator/src/docs/InvestigatingResults.md and follow it to analyze the results.json files. Diagnose each failure, suggest fixes to the eval.yaml and skill content, and tell me what to fix first.

▶ Sessions Visualisation -- interactive replay of all evaluation sessions
📊 Session Analytics (preview) -- aggregated metrics across evaluation sessions

…verhead

The full TFM-forwarding guidance lives in extension-points; AP-13 only needs a concise pointer. Removes duplication and trims the msbuild-antipatterns skill footprint flagged by the skill-validator token penalty.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 26, 2026 14:21
@YuliiaKovalova

Copy link
Copy Markdown
Member Author

Re: the three F# msbuild-antipatterns scenarios (footnotes [2]/[3]/[4])

These three are pre-existing, structural failures — not regressions from this PR, and not fixable by its content:

  • All three are NOT ACTIVATED with unchanged quality (5.0 → 5.0, 4.0 → 4.0). msbuild-antipatterns correctly does not activate for F# coding tasks (add a module / fix file order / add a signature file). That's the desired behavior.
  • They fail only on the token footnote (-0.6% / -4.1% / -4.3%) — the inherent cost of loading the plugin's skill menu in the skilled arm vs the no-skill baseline. The ~50k-token delta is the plugin/skill footprint; this PR's forwarding note is a negligible fraction of it.
  • They were already ❌ in the first eval on d38fb2e (before any of my tightening), with the identical NOT-ACTIVATED + token pattern. They fail the same way on main.

What I did do to minimize footprint (this lever is mine to pull, even though it won't flip the deterministic token-penalty rows):

  • Tightened the extension-points "Forwarding chain" section (+26 → +11 net lines).
  • Condensed the AP-13 note to a one-line cross-reference (8be4ea9), removing duplication with extension-points. Net result: scenario [2]'s skilled token count already dropped (167k → 158k, footnote moved -3.8% → -0.6%).

The remaining red on [3]/[4] is deterministic token overhead on control scenarios where the skill correctly stays dormant — it can only be moved by shrinking the overall plugin's skill-menu footprint (out of scope here) or by the harness's token weighting / run count. Re-running won't help these two (they're not variance-driven). Flagging for maintainer awareness.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.


### Forwarding chain: `buildTransitive/` → `build/` → shared

Forward `buildTransitive/*.props` through the sibling `build/*.props` (chain `buildTransitive → build → shared`) instead of importing `buildMultiTargeting/` directly — this keeps a clear ownership chain and makes explicit that transitive consumers get a subset of what direct consumers see.

```xml
<!-- buildTransitive/<tfm>/MyPackage.props -->
<Import Project="$(MSBuildThisFileDirectory)..\..\build\$([System.IO.Path]::GetFileName($([System.IO.Path]::GetDirectoryName($(MSBuildThisFileDirectory)))))\MyPackage.props" />
Comment thread plugins/dotnet-msbuild/agents/msbuild-code-review.agent.md Outdated

Before flagging an unguarded `<Import>` inside a `build/` or `buildTransitive/` folder, **resolve it against the packed layout** — read every `*.nuspec` in the project directory **and its immediate parent directory** (shared nuspecs are common in mono-repos; do not walk further up), and any `<PackagePath>` metadata on `<None>`/`<Content>` items in the `.csproj`. Only flag if the target path is missing from **both** the source tree *and* the projected package layout. The `dotnet-msbuild/extension-points` skill — *Source tree vs packed layout* — documents the full cross-check procedure.

**Forwarding `buildTransitive/` → `build/`:** forward through the sibling `build/*` file (not directly to `buildMultiTargeting/`); when `build/` is per-TFM (`build/<tfm>/`), include the TFM segment derived from the file's own folder (not `$(TargetFramework)`), or transitive consumers hit `MSB4019`. See the `extension-points` skill — *Forwarding chain* — for the rule and derivation expression.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 26, 2026 15:09

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.


### Forwarding chain: `buildTransitive/` → `build/` → shared

Forward `buildTransitive/*.props` through the sibling `build/*.props` (chain `buildTransitive → build → shared`) instead of importing `buildMultiTargeting/` directly — this keeps a clear ownership chain and makes explicit that transitive consumers get a subset of what direct consumers see.

```xml
<!-- buildTransitive/<tfm>/MyPackage.props -->
<Import Project="$(MSBuildThisFileDirectory)..\..\build\$([System.IO.Path]::GetFileName($([System.IO.Path]::GetDirectoryName($(MSBuildThisFileDirectory)))))\MyPackage.props" />
@github-actions github-actions Bot added waiting-on-author PR state label and removed pr-state/ready-for-eval PR is mergeable and awaiting evaluation labels Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

waiting-on-author PR state label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants