🏥 Repository Health Dashboard

[github-actions]

🏥 Daily Health Check — 2026-06-26

Status: 🔴 3 critical · 🟡 2 warnings · 🔵 1 info
Since yesterday: 🆕 3 new · ✅ 0 resolved · 📌 3 unchanged

📌 Maintainer action needed: please pin this issue as the canonical health dashboard and unpin/close any stale duplicate.

---

🆕 New Findings (3)

These appeared since the last health check (2026-06-25).

🔴 [P1] Evaluation build-validator job failed: GitHub.Copilot.SDK build error

• Fingerprint: pipeline:evaluation:build-validator:build-skill-validator:failure
• First seen: 2026-06-26
• Details: The build-validator job in the evaluation workflow failed on main at 2026-06-25T14:02Z for PR Bump GitHub.Copilot.SDK, MessagePack and Nerdbank.MessagePack #818 evaluation. The Build skill-validator step failed with multiple CS0234 errors: The type or namespace name 'SDK' does not exist in the namespace 'GitHub.Copilot'. Affected files: AgentRunner.cs, Judge.cs, LlmSession.cs, LocalSessionFsHandler.cs, PairwiseJudge.cs. The failure occurred against CLI 1.0.65. Note: Today's scheduled run (at 00:55 UTC) succeeded on the latest main commit, suggesting the incompatibility was fixed by a subsequent commit (PR Require environment input to pat_pool shared workflow. #832 merged 2026-06-25T22:35).
• Links: Failed run (build-validator, 5 min) · Today's successful scheduled run · evaluation.yml
• Suggested action: Verify that GitHub.Copilot.SDK namespace usage in AgentRunner.cs, Judge.cs, LlmSession.cs is compatible with the pinned CLI version. Check if PR skill-validator: restore 15K aggregate cap as the real Copilot CLI skill-menu budget #803 (restore 15K cap) introduced a reference to SDK APIs not yet available in CLI 1.0.65.

---

🔴 [P5] Evaluation failure rate critical: ~33% across all branches in last 24h

• Fingerprint: pipeline:evaluation:failure-rate:critical
• First seen: 2026-06-26
• Details: Of 4 evaluation runs in the last 24h, 1 failed, 2 succeeded, 1 was cancelled. Failure rate (excl. cancelled) = 33.3% — above the 30% Critical threshold. The single failure was run #4823 (Evaluate PR #818 @ fda2336, build-validator job, build error — same as P1 above). The failure rate is driven entirely by this one build-error run; the scheduled run and PR Bump the github-actions-dependencies group across 1 directory with 5 updates #833 evaluation both succeeded.
• Breakdown: 2 successes (scheduled + PR Bump the github-actions-dependencies group across 1 directory with 5 updates #833), 1 failure (PR Bump GitHub.Copilot.SDK, MessagePack and Nerdbank.MessagePack #818), 1 cancelled (issue_comment triggered)
• Links: Failed run #4823 · Successful PR #833 run · Successful scheduled run
• Suggested action: Correlated with P1 above — fixing the SDK build compatibility issue should resolve this metric.

---

🟡 [P2] Evaluation cancelled after 102 min on main (issue_comment trigger)

• Fingerprint: pipeline:evaluation:evaluate:timeout
• First seen: 2026-06-26
• Details: An issue_comment-triggered evaluation run ("Add eval coverage for dotnet-test/filter-syntax") was cancelled on main at 2026-06-25T08:34Z after running for ~102 minutes. No job failed — the run was cancelled externally (likely by a concurrency group eviction when a newer run started). This is consistent with the chronic eval duration issue (runs taking 99–130 min).
• Links: Cancelled run #4804
• Suggested action: The 102-min runtime is consistent with the chronic duration issue (P3/resource:eval-duration:critical). Reducing eval runtime (parallelizing dotnet-test plugin evaluation) would also reduce concurrency-group cancellations.

---

🔍 Investigation Results

Deep investigations are dispatched for new critical/warning findings.
The grooming workflow links results ~3 hours after this run.

Finding	Severity	Investigation	First Seen	Result
Evaluation avg duration critical (110+ min)	🔴 Critical	✅ Done	2026-06-03	The evaluate (dotnet-test) job is the sole critical bottleneck at 100 minutes, driven by 175 sequential eval scenarios; the regression traces directly to PR #707 (merged June 1) which added polyglot scenarios to three skills.
Evaluation build-validator failed (GitHub.Copilot.SDK CS0234)	🔴 Critical	🔄 Dispatched	2026-06-26	⏳ Investigation dispatched — results arriving shortly...
Evaluation failure rate critical (33% in 24h)	🔴 Critical	🔄 Dispatched	2026-06-26	⏳ Investigation dispatched — results arriving shortly...

---

✅ Resolved Since Yesterday (0)

No findings resolved since the last health check (2026-06-25).

---

📌 Existing Findings (3)

These have been present since before today. Sorted by severity then age.

🔴 Critical — Evaluation average duration critical (~99–130 min avg, threshold: 55 min) · first seen 2026-06-03 · 12 occurrences

• Fingerprint: resource:eval-duration:critical
• First seen: 2026-06-03 · Occurrences: 12 (chronic — 3+ weeks)
• Details: The 14-day average for substantial evaluation schedule runs remains well above the 55-min critical threshold. Today's scheduled run took 99 minutes (down from the 121–130 min recent average), but still 1.8× the threshold. The previous run (yesterday's) was 85 min, suggesting some variance.
• 7-day summary (schedule, main): Today: 99 min ✅ success; yesterday: 85 min ✅; Jun 23–24: 2 failures. Avg ~118 min (est.)
• Root cause (from investigation): The evaluate (dotnet-test) job is the bottleneck, driven by 175 sequential eval scenarios added by PR Make dotnet-test analysis skills and auditor agent polyglot #707 (merged 2026-06-01).
• Links: Today's run (99 min, success) · Investigation result · evaluation.yml
• Suggested action: Parallelize eval scenarios in evaluate (dotnet-test) — split 175 sequential scenarios across parallel jobs, or introduce a fast-path for PR-triggered evaluations vs. scheduled full runs.

🟡 Warning — Orphan plugin: dotnet-experimental not listed in marketplace.json · first seen 2026-05-14 · 31 occurrences

• Fingerprint: infra:orphan-plugin:dotnet-experimental
• First seen: 2026-05-14 · Occurrences: 31 (chronic — 6+ weeks)
• Details: plugins/dotnet-experimental/ exists on disk with a valid plugin.json and skills (exp-mock-usage-analysis, exp-test-maintainability, exp-simd-vectorization), but no entry in .github/plugin/marketplace.json (which lists 14 plugins). The plugin is not discoverable by consumers.
• Links: marketplace.json · plugins/dotnet-experimental/
• Suggested action: Either add { "name": "dotnet-experimental", "source": "./plugins/dotnet-experimental", "description": "..." } to marketplace.json when ready to publish, or remove the directory if not intended for publication.

🔵 Info — evaluation.yml uses --verdict-warn-only mode · first seen 2026-05-16 · 29 occurrences

• Fingerprint: infra:verdict-warn-only
• First seen: 2026-05-16 · Occurrences: 29 (intentional configuration)
• Details: evaluation.yml passes --verdict-warn-only to the skill-validator, treating skill validation failures as warnings rather than hard failures. This is intentional.
• Link: evaluation.yml

---

📊 Trends (7-day)

Metric	Today	7d Avg	Δ	Trend
Eval duration — schedule/main (min)	99	~118	-19	✅
Eval success rate — main schedule (7d)	100% (1/1)	~71%	+29%	✅
Eval success rate — all branches (24h)	67% (2/3)	100%	-33%	⚠️
Eval scheduled cancellation rate (24h)	0% (0/1)	0%	0%	➡️
Workflow failure rate — main (24h)	~3% (1 failure)	~0%	+3%	↗️
Compute hours/day	~3.5h	~2.0h	+1.5h	↗️

⚠️ Eval pipeline degraded today: 1 evaluation build failure + 1 cancellation on main in 24h.
✅ Scheduled eval succeeded (99 min, latest main commit). The build failure was on an older PR commit and appears resolved.
⚠️ P1 + P5 are correlated — the SDK build error in PR #818's skill-validator code drove both findings. The fix (PR #832, merged 22:35 UTC) appears to have resolved the incompatibility.
⚠️ Eval duration remains a chronic concern (118 min est. 7d avg). Concurrency cancellations will continue until runtime is reduced.
⚠️ Skipped I5 check (Pages deployment): GitHub Pages API not accessible via available tools.
i️ I3 (validate-skills): No validate-skills.yml workflow found — check not applicable.
i️ I6 (unpinned actions): All workflow action references use first-party (actions/*) or SHA-pinned third-party actions — no unpinned third-party actions detected.

---

_{🤖 Generated by DevOps Health Check agentic workflow · Run #28217288553 · 2026-06-26T04:34 UTC}

Generated by DevOps Daily Health Check · 1.2K AIC · ⊞ 36.5K · ◷

[github-actions]

🔍 Investigation: Evaluation average duration critical (110+ min)

Finding ID: resource:eval-duration:critical
Severity: Critical
Correlation: hc-2026-06-03-1
Executive Summary: The evaluate (dotnet-test) job is the sole critical bottleneck at 100 minutes, driven by 175 sequential eval scenarios; the regression traces directly to PR #707 (merged June 1) which added polyglot scenarios to three skills.

Root Cause

The evaluate (dotnet-test) job took 100 minutes in run #3963 (skill-validator step: 01:11:21Z → 02:51:30Z), making it the last job to unblock post-processing and the primary driver of total run duration. A full count of tests/dotnet-test/**/eval.yaml shows 175 scenarios running sequentially in a single job. The prior scheduled run (June 2, 107.7 min) started the same trend, confirming this is not a transient spike.

The growth directly follows PR #707 (680292e, merged 2026-06-01 14:46Z — "Make dotnet-test analysis skills and auditor agent polyglot"), which explicitly added new polyglot eval scenarios to three skills:

• test-anti-patterns — 7 scenarios (Python/pytest, TS/Jest, Java/JUnit added alongside C#)
• assertion-quality — 6 scenarios (polyglot scenarios added)
• test-smell-detection — 5 scenarios (polyglot scenarios added)

A secondary bottleneck is evaluate (dotnet-msbuild--shard-medium) at 79 minutes (parallel, so it doesn't add wall time on top of dotnet-test, but it does drive high compute usage across all 3 msbuild shards).

Job duration summary for run #3963 (all parallel from ~01:11):

Job	Duration
evaluate (dotnet-test) ← bottleneck	100 min
evaluate (dotnet-msbuild--shard-medium)	79 min
evaluate (dotnet-ai)	71 min
evaluate (dotnet-msbuild--shard-default)	64 min
evaluate (dotnet-blazor)	58 min
evaluate (dotnet-msbuild--shard-heavy)	54 min
evaluate (dotnet-upgrade)	47 min
evaluate (dotnet-diag)	40 min

Confidence: High — direct evidence from job timestamps; PR #707 commit message explicitly states "New polyglot eval scenarios added ... in test-anti-patterns, assertion-quality, test-smell-detection"; scenario count (175) corroborated by direct file count.

Blast Radius

• Every scheduled evaluation run exceeds the 55-min critical threshold; the 7-day trend is ↗️ Increasing (~120 min average).
• Feedback latency for PR authors using /evaluate is now ~2.5× the expected duration.
• If the schedule interval is ≤2 hours, cascading run cancellations are possible (the health check flagged this as P6 risk).
• The secondary msbuild bottleneck (3 shards, up to 79 min) adds significant compute cost in parallel.

Suggested Fix

1. Shard evaluate (dotnet-test) — dotnet-msbuild already splits its 3 shards (heavy/medium/default); apply the same pattern to dotnet-test. With 175 scenarios, 3–4 parallel shards would reduce the job to ~25–35 min. Group the heavy agentic scenarios (code-testing-agent, agent.test-*) in one dedicated shard and distribute the simpler analysis skills across the others.

2. Review the scenario density added by PR Make dotnet-test analysis skills and auditor agent polyglot #707 — The three newly-polyglot skills contributed ~10–15 new scenarios. If each scenario averages ~30–35 seconds of runtime, consolidating or reducing these could recover 5–9 minutes. Consider whether Python/TS/Java coverage can share fixtures or prompts to reduce scenario count without losing coverage breadth.

3. Reduce dotnet-msbuild--shard-medium runtime (79 min) — Audit which scenarios in the medium shard are slowest and consider whether any can be promoted to the heavy shard or simplified.

Evidence

# Job: evaluate (dotnet-test) — run #3963
# started_at: 2026-06-03T01:11:04Z
# completed_at: 2026-06-03T02:51:46Z  (100 min)
# Bottleneck step: Run skill-validator  01:11:21Z → 02:51:30Z

# dotnet-test eval scenario count (tests/dotnet-test/**/eval.yaml):
run-tests: 15, writing-mstest-tests: 15, migrate-mstest-v3-to-v4: 15,
migrate-vstest-to-mtp: 11, dotnet-test-frameworks: 11, migrate-xunit-to-xunit-v3: 12,
test-tagging: 8, mtp-hot-reload: 7, test-anti-patterns: 7, assertion-quality: 6,
detect-static-dependencies: 6, test-smell-detection: 5, migrate-mstest-v1v2-to-v3: 9,
migrate-static-to-wrapper: 3, coverage-analysis: 3, crap-score: 3,
test-gap-analysis: 4, generate-testability-wrappers: 4, agent.test-migration: 4,
agent.test-quality-auditor: 4, agent.testability-migration: 2,
code-testing-agent: 1  →  TOTAL: 175 scenarios

PR that introduced the regression:

• PR #707 — "Make dotnet-test analysis skills and auditor agent polyglot" — commit 680292e — merged 2026-06-01 by Evangelink

Related

• PR Make dotnet-test analysis skills and auditor agent polyglot #707 (680292e) — direct cause: added polyglot scenarios to test-anti-patterns, assertion-quality, test-smell-detection
• PR code-testing-agent: add polyglot pipeline examples for Python/TypeScript/Go/Java #708 (49a77a9) — coincident merge (June 2), adds documentation only, not a cause
• Trends table in this issue: 7d avg for scheduled eval = ~120 min (↗️ Increasing)
• No existing tracking issue found for the dotnet-test evaluation sharding work

---

_{🔍 Investigation Run #26864313000 · Dispatched by health check · hc-2026-06-03-1}

Generated by DevOps Health — Deep Investigation · ● 2.3M · ◷

[github-actions]

📋 Health Check — 2026-06-15

🆕 0 new · ✅ 2 resolved · 📌 3 unchanged

Status: 🔴 1 critical · 🟡 1 warning · 🔵 1 info

Resolved:

• 🔴 Evaluation failed: evaluate (dotnet-diag) — Run skill-validator — dotnet-diag token 429 issue cleared after June 14 run succeeded
• 🔴 Evaluation failure rate critical (50% in last 24h) — failure rate is now 0% (1 run, success)

Persistent (unchanged):

• 🔴 Evaluation average duration critical (~116 min, threshold 55 min) — 9 occurrences since 2026-06-03 · PR #707 bottleneck still unaddressed
• 🟡 Orphan plugin: dotnet-experimental not in marketplace.json — 28 occurrences since 2026-05-14
• 🔵 evaluation.yml uses --verdict-warn-only — intentional config

No investigation dispatches — no new qualifying findings this run.

Full report →

Generated by DevOps Daily Health Check · sonnet46 6.2M · ◷

[github-actions]

📋 Health Check — 2026-06-20

🆕 1 new · ✅ 2 resolved · 📌 3 unchanged

Status: 🔴 2 critical · 🟡 1 warning · 🔵 1 info

New:

• 🔴 Evaluation failed: build-validator — Build skill-validator — build-validator job failed at "Build skill-validator" step on main (run #4504, issue_comment trigger). All evaluate jobs skipped. Investigation dispatched.

Resolved:

• 🔴 Evaluation failed: evaluate (dotnet-diag) — Run skill-validator — token rate-limit cleared, Jun 19 scheduled run succeeded.
• 🔴 Evaluation failure rate critical (50% in last 24h) — failure rate now 8.3% (well below 15% threshold).

Notable (existing):

• 🔴 Eval duration still critical: 164 min on Jun 19 (up from 116 min on Jun 14, PR Make dotnet-test analysis skills and auditor agent polyglot #707 bottleneck unaddressed)

Full report →

Generated by DevOps Daily Health Check · sonnet46 7.3M · ◷

[github-actions]

📋 Health Check — 2026-06-21

🆕 0 new · ✅ 2 resolved · 📌 3 unchanged

New: (none — pipeline fully healthy today)

Resolved (from cache 2026-06-13; already resolved in 2026-06-20 dashboard):

• 🔴 Evaluation failed: evaluate (dotnet-diag) — Run skill-validator — all scheduled eval runs now completing successfully
• 🔴 Evaluation failure rate critical (50% in last 24h) — current rate: 0% (0/3 non-skipped runs)

Ongoing concerns:

• 🔴 Evaluation average duration critical (~145 min avg, 7+ weeks) — no actual eval ran Jun 20–21 (no new commits)
• 🟡 Orphan plugin: dotnet-experimental (not in marketplace.json) — 28 occurrences
• 🔵 --verdict-warn-only mode (intentional)

No new dispatches: 0 qualifying new findings.

Full report →

Generated by DevOps Daily Health Check · sonnet46 4.9M · ◷

[github-actions]

📋 Health Check — 2026-06-22

🆕 0 new · ✅ 0 resolved · 📌 3 unchanged

Status: 🔴 1 critical · 🟡 1 warning · 🔵 1 info

New:
No new findings — pipeline is fully healthy today (0 failures, 0 cancellations on main in last 24h; all 30 runs successful).

Resolved:
Nothing resolved since yesterday.

Existing (unchanged):

• 🔴 resource:eval-duration:critical — Eval avg ~138 min (10th consecutive occurrence; last actual run Jun 19)
• 🟡 infra:orphan-plugin:dotnet-experimental — Not in marketplace.json (29th occurrence)
• 🔵 infra:verdict-warn-only — Intentional configuration (27th occurrence)

i️ No new findings → no investigation dispatches this run.

Full report →

Generated by DevOps Daily Health Check · sonnet46 5.2M · ◷

[github-actions]

📋 Health Check — 2026-06-25

🆕 0 new · ✅ 0 resolved · 📌 3 unchanged

Status: 🔴 1 critical · 🟡 1 warning · 🔵 1 info

New: None — pipeline healthy today (0 failures, 0 cancellations on main).

Resolved: None.

Existing findings:

• 🔴 Eval avg duration critical (~130 min 14d avg, threshold: 55 min) — 11th occurrence since 2026-06-03. Today's run: 85 min ✅ (best in recent weeks, but still 1.5× threshold)
• 🟡 Orphan plugin dotnet-experimental not in marketplace.json — 30th occurrence since 2026-05-14
• 🔵 --verdict-warn-only in evaluation.yml — intentional config, 28th occurrence

Notable: Two consecutive eval schedule failures on Jun 23 and Jun 24 fell outside the 24h P1 window. Today's scheduled run recovered (85 min, success). No new or resolved findings since last check (2026-06-22). No investigation dispatches needed.

Full report →

Generated by DevOps Daily Health Check · sonnet46 3.3M · ◷