[skill] Generalize verification skill into 3 scenarios (module / release-PR / active-PR)

.github/skills/powertoys-module-verification/SKILL.md → .github/skills/powertoys-verification/SKILL.md

@@ -1,25 +1,47 @@
 ---
-name: powertoys-module-verification
-description: "Verify a single PowerToys module's release checklist items end-to-end. Drive each checkbox via UIA / Named Events / settings.json edits / clipboard inspection / GPO / SendInput. Output a structured PASS / FAIL / BLOCKED verdict per item with evidence (FAIL distinguishes product defects from stale/ambiguous checklist items). Combine standard winapp ui mechanics (see references/winapp-ui-testing.md) with PT-specific recipes and the helper .ps1 files shipped with this skill."
+name: powertoys-verification
+description: "Verify PowerToys behavior end-to-end with the winapp CLI across three scenarios: (A) a single module's full release checklist; (B) sign-off of the PRs in a release or hotfix (derive each PR's checklist from its description + diff, drive the installed build); (C) an active/unmerged PR (build + sideload the affected module, then validate). Drive each item via UIA invoke / Named Events / settings.json edits / clipboard / GPO / SendInput, and emit a structured PASS / FAIL / BLOCKED verdict per item with evidence (FAIL distinguishes product defects from stale/ambiguous checklist items). Use when asked to verify a PowerToys module checklist, sign off a release or hotfix's PRs, validate a PowerToys PR, or QA installed/sideloaded PowerToys bits. Combines generic winapp ui mechanics (references/winapp-ui-testing.md) with PT-specific recipes, per-scenario playbooks (references/scenarios/), and the helper .ps1 files shipped with this skill."
 license: Complete terms in LICENSE.txt
 ---
 
 ## When to use this skill
 
-Use this skill when you need to **verify every checklist item for a single PowerToys module** for a release sign-off — e.g. "verify all 18 Color Picker items", "verify all 88 Command Palette items". Each item produces a PASS / FAIL / BLOCKED verdict with evidence (UIA enumeration, log line, settings.json diff, screenshot, etc.).
+Use this skill whenever you need to **verify PowerToys behavior with the winapp CLI** and emit a
+structured PASS / FAIL / BLOCKED verdict with evidence (UIA enumeration, log line, settings.json
+diff, screenshot, etc.). It runs in **three scenarios** that share the same drive techniques,
+helpers, taxonomy and report format — and differ only on what the checklist is, what bits you test,
+and the test discipline:
 
-The **checklist to verify is supplied with the task** (the calling prompt points you at the module's checklist file). This skill is the *how* — the drive techniques, helpers, taxonomy, and reporting format — independent of any specific checklist.
+| Scenario | Trigger | Checklist source | Bits under test |
+|---|---|---|---|
+| **A — Module checklist** | "verify all `<Module>` items", "sign off Color Picker" | Supplied file (`references/release-checklist/<module>.md`) | Installed shipped artifact (read-only) |
+| **B — Release/hotfix PR sign-off** | "verify the PRs in this release/hotfix", "sign off 0.X.Y" | **Derived** from each PR's description + diff | Installed shipped artifact (read-only) |
+| **C — Active PR validation** | "validate PR #N", "build it and test the fix" | **Derived** from one PR's description + diff | **Your local build, sideloaded** |
+
+> **Step 0 for every run — pick the scenario.** Read **`references/scenarios/index.md`** (the
+> router + the per-scenario "bits under test" contract + the verdict-vocabulary mapping), then read
+> the one matching scenario doc (`references/scenarios/{module-checklist,release-pr-signoff,active-pr-validation}.md`).
+> This `SKILL.md` is the *shared engine* (drive techniques, helpers, taxonomy, pitfalls) common to
+> all three.
+
+Each item produces a PASS / FAIL / BLOCKED verdict with evidence. For **A** the checklist is
+supplied; for **B / C** you derive it from the PR(s). The skill is the *how* — independent of any
+specific checklist.
 
 ## Required reads (in order)
 
-1. **`references/winapp-ui-testing.md`** — the **prerequisite** UIA mechanics doc (winapp ui verbs, scripted batch testing, file pickers, accessibility audits, screenshots, click-vs-invoke, PostMessage, SendInput cb=40, stunted-UIA recovery, settings-mutation safety contract). **Read this first** — this skill assumes you know its content and only adds PT-specific extensions.
-2. **This `SKILL.md`** — the PT-specific playbook: the 3-bucket drive-technique selector (Step 2), classification taxonomy, critical pitfalls, helper-script catalog.
-3. **`references/modules/<module>.md` IF IT EXISTS** — per-module entry-paths, item-by-item recipes, common BLOCKED traps, fixture lists, source citations. **Always check `references/modules/` first.** If no profile exists, fall back to this SKILL.md and create one after you finish (template in `references/modules/README.md`).
-4. **`references/explorer-context-menu-flow.md` IF your module registers an Explorer right-click entry** (PowerRename, File Locksmith, Image Resizer, New+, Preview Pane, RegistryPreview) — shared synthetic-right-click + UIA-invoke + multi-file-selection flow + module-caption table. Helper: `scripts/pt-explorer-contextmenu.ps1`.
-5. **`references/pre-flight.md`** — pre-flight checks, bootstrap snippet, state-hygiene cleanup, final wrap-up, hard rules.
-6. **`references/reporting-format.md`** — per-item table template, top-of-report summary, step-table rules, anti-patterns, worked example.
-7. **`references/environment-setup.md`** — RDP/sleep/screensaver/session-attachment gotchas. Cite in BLK-ENV verdicts.
-8. **`references/release-checklist/<module>.md`** — the checklist for the module under test (one file per module; see `references/release-checklist/index.md` for the full list). Each item carries `[ADMIN: …]` + `[CLARITY: …]` metadata. **This file IS the set of items to verify.**
+1. **`references/scenarios/index.md`** — **read FIRST**: the scenario router (A/B/C), the
+   per-scenario **"bits under test" contract** (installed-and-immutable for A/B vs build-and-sideload
+   for C), and the verdict-vocabulary mapping. Then read the **one** matching scenario doc:
+   `references/scenarios/module-checklist.md` (A) · `release-pr-signoff.md` (B) · `active-pr-validation.md` (C).
+2. **`references/winapp-ui-testing.md`** — the **prerequisite** UIA mechanics doc (winapp ui verbs, scripted batch testing, file pickers, accessibility audits, screenshots, click-vs-invoke, PostMessage, SendInput cb=40, stunted-UIA recovery, settings-mutation safety contract). **Read this first** — this skill assumes you know its content and only adds PT-specific extensions.
+3. **This `SKILL.md`** — the shared engine: the 3-bucket drive-technique selector (Step 2), classification taxonomy, critical pitfalls, helper-script catalog.
+4. **`references/modules/<module>.md` IF IT EXISTS** — per-module entry-paths, item-by-item recipes, common BLOCKED traps, fixture lists, source citations. **Always check `references/modules/` first.** If no profile exists, fall back to this SKILL.md and create one after you finish (template in `references/modules/README.md`).
+5. **`references/explorer-context-menu-flow.md` IF your module registers an Explorer right-click entry** (PowerRename, File Locksmith, Image Resizer, New+, Preview Pane, RegistryPreview) — shared synthetic-right-click + UIA-invoke + multi-file-selection flow + module-caption table. Helper: `scripts/pt-explorer-contextmenu.ps1`.
+6. **`references/pre-flight.md`** — pre-flight checks, bootstrap snippet, state-hygiene cleanup, final wrap-up, hard rules.
+7. **`references/reporting-format.md`** — per-item table template, top-of-report summary, step-table rules, anti-patterns, worked example.
+8. **`references/environment-setup.md`** — RDP/sleep/screensaver/session-attachment gotchas. Cite in BLK-ENV verdicts.
+9. **`references/release-checklist/<module>.md` — SCENARIO A ONLY** — the supplied checklist for the module under test (one file per module; see `references/release-checklist/index.md`). Each item carries `[ADMIN: …]` + `[CLARITY: …]` metadata. **This file IS the set of items to verify.** For B/C the checklist is derived from the PR(s) instead (see the scenario doc).
 
 ## Helper scripts shipped with this skill
 
@@ -39,7 +61,7 @@ The **checklist to verify is supplied with the task** (the calling prompt points
 
 Dot-source them **all** at once in your bootstrap (the `Get-ChildItem` loop loads every helper — see **Step 1 — Bootstrap**):
 ```powershell
-$skill = '<this skill folder>'   # the folder containing SKILL.md, e.g. <PT-repo>\.github\skills\powertoys-module-verification
+$skill = '<this skill folder>'   # the folder containing SKILL.md, e.g. <PT-repo>\.github\skills\powertoys-verification
 Get-ChildItem "$skill\scripts" -Filter '*.ps1' | ForEach-Object { . $_.FullName }
 ```
 
@@ -63,8 +85,11 @@ Get-ChildItem "$skill\scripts" -Filter '*.ps1' | ForEach-Object { . $_.FullName
 $rn = Test-PtRunnerAdmin
 "PT runner: PID=$($rn.Pid) Elevated=$($rn.Elevated)" | Tee-Object $report -Append
 
-# The checklist items to verify are supplied with the task (see the calling prompt).
-# Read that module's checklist file and iterate its items (see Step 6 — Verifier loop).
+# The checklist source depends on the scenario (see references/scenarios/):
+#   A - read the supplied references/release-checklist/<module>.md
+#   B - derive 1-3 items from each PR's `gh pr view/diff` (release-pr-signoff.md)
+#   C - derive items from the PR, after build+sideload (active-pr-validation.md)
+# Then iterate the items (see Step 6 - Verifier loop).
 ```
 
 ## Step 2 — Drive techniques
@@ -247,20 +272,28 @@ Print the **moved** report path (under `…\PowerToys\Module-Signoff\`) as the l
 
 ## Invocation & placeholders
 
-This skill auto-activates when you ask to verify a PowerToys module's checklist (e.g. "verify all Color Picker items"). **One module per run** — never chain multiple modules into one report. Resolve these placeholders for the module under test:
+This skill auto-activates for any of the three scenarios above (verify a module checklist, sign off
+a release/hotfix's PRs, or validate a PR). **Step 0: resolve the scenario via
+`references/scenarios/index.md` and set the `BITS:` contract** before driving anything. Scope rule:
+**one module per run for Scenario A**, **one PR per report folder for B/C**. Resolve these
+placeholders:
 
 | Placeholder | Substitute with |
 |---|---|
 | `<Module>` | Exact display name, e.g. `Color Picker`, `Command Palette`, `PowerToys Run`, `FancyZones` (see `references/release-checklist/index.md`). |
-| `<module>` | Lowercase-kebab-case for file lookup, e.g. `color-picker`, `command-palette`, `power-rename` — used for BOTH `references/release-checklist/<module>.md` (checklist) and `references/modules/<module>.md` (profile, if any). |
+| `<module>` | Lowercase-kebab-case for file lookup, e.g. `color-picker`, `command-palette`, `power-rename` — used for BOTH `references/release-checklist/<module>.md` (A's checklist) and `references/modules/<module>.md` (profile, if any). |
 | `<ModuleDir>` | settings.json sub-dir under `%LOCALAPPDATA%\Microsoft\PowerToys\` (e.g. `AdvancedPaste`, `FancyZones`, `PowerToys Run` (with space)). |
-| `<N>` | Total item count for this module. |
+| `<N>` | Scenario A: total item count for the module. Scenario B/C: the GitHub PR number. |
 
-**Execution order:** `references/pre-flight.md` → per item, the §2 drive-stack (this file) → `references/reporting-format.md` per-item table → Step 6 verifier loop → `references/pre-flight.md` §Final wrap-up → Step 7 archive → print the final report path.
+**Execution order:** `references/scenarios/index.md` (pick scenario + set `BITS`) → the matching
+`references/scenarios/<scenario>.md` (inputs, discipline, deploy steps for C) → `references/pre-flight.md`
+→ per item, the §2 drive-stack (this file) → `references/reporting-format.md` per-item table →
+Step 6 verifier loop → `references/pre-flight.md` §Final wrap-up → Step 7 archive → print the final report path.
 
 ## What NOT to do
 
-- Do NOT chain multiple modules in one report — one module per run.
+- Do NOT skip Step 0 — drive nothing until you've set the scenario and the `BITS:` contract. A run that mutates the wrong bits (sideloads in B, or drives the installed binary in C) is invalid regardless of the verdict.
+- Do NOT chain multiple modules in one report (Scenario A) — one module per run. For B/C, one PR per report folder.
 - Do NOT mark an item BLOCKED without a concrete, named obstacle (see §3 and `references/pre-flight.md` §Hard rules).
 - Do NOT invent steps for a VAGUE checklist item — if the spec is too ambiguous to judge, that is FAIL (cause=checklist), not a guess.
 - All other rules (foreground guard, always restore mutated state, etc.) live in `references/pre-flight.md` §Hard rules — follow them.
@@ -293,7 +326,7 @@ This skill auto-activates when you ask to verify a PowerToys module's checklist
 | Module | Module-owned file (under `…\PowerToys\<Module>\`) | Key style | PT-store `settings.json` keys (UI/`Get-PtModuleSettings`) |
 |---|---|---|---|
 | PowerRename | `power-rename-settings.json` (+ `power-rename-last-run-data.json`, `search-mru.json`, `replace-mru.json`) | `ShowIcon`, `ExtendedContextMenuOnly`, `PersistState`, `MRUEnabled`, `MaxMRUSize`, `UseBoostLib` | `bool_show_icon_on_menu`, `bool_show_extended_menu`, `bool_persist_input`, `bool_mru_enabled`, `int_max_mru_size`, `bool_use_boost_lib` |
-| File Locksmith | `file-locksmith-settings.json` | `ShowInExtendedContextMenu` | `bool_show_extended_context_menu` |
+| File Locksmith | `file-locksmith-settings.json` | `showInExtendedContextMenu` | `bool_show_extended_context_menu` |
 | Image Resizer | `image-resizer-settings.json` | (resize sizes/encoder/etc.) | mirrored `imageresizer*` keys |
 | New+ | `NewPlus\settings.json` (sub-folder **`NewPlus`**, verified on disk + `constants.h` `powertoy_name=L"NewPlus"`) | `HideFileExtension`, `HideStartingDigits`, `TemplateLocation`, `ReplaceVariables`, `BuiltInNewHidePreference` | mirrored `newplus*` keys |
 

.github/skills/powertoys-module-verification/references/explorer-context-menu-flow.md → .github/skills/powertoys-verification/references/explorer-context-menu-flow.md

@@ -29,7 +29,7 @@ Net: for a context-menu module, **most items are behavior → CLI-first**; the *
 **Hard prerequisite — unlocked interactive desktop.** Synthetic right-click injects into the session input stream, so it requires foreground. If the workstation is locked / RDP minimized (`GetForegroundWindow()=0`), this flow is `BLK-ENV` — there is no foreground-free way to open a context menu. `Open-PtExplorerContextMenu` throws a clear BLK-ENV error in that case. (A 4-hour idle auto-lock is the common culprit — see `references/environment-setup.md`.)
 
 **Other constraints:**
-- **Settings for these modules live in a module-OWNED file, not the PT-store `settings.json`** — see `SKILL.md` pitfall #18. The context-menu handler reads e.g. `power-rename-settings.json` / `file-locksmith-settings.json` / `image-resizer-settings.json` / `New\settings.json` at launch; editing the PT-store `<Module>\settings.json` (what `Get-PtModuleSettings` reads) often has **no effect** on the live handler. Drive icon/extended-menu/feature toggles via the module-owned file + relaunch (restart runner+Explorer for the menu handlers), then restore.
+- **Settings for these modules live in a module-OWNED file, not the PT-store `settings.json`** — see `SKILL.md` pitfall #18. The context-menu handler reads e.g. `power-rename-settings.json` / `file-locksmith-settings.json` / `image-resizer-settings.json` / `NewPlus\settings.json` at launch; editing the PT-store `<Module>\settings.json` (what `Get-PtModuleSettings` reads) often has **no effect** on the live handler. Drive icon/extended-menu/feature toggles via the module-owned file + relaunch (restart runner+Explorer for the menu handlers), then restore.
 - This is the **Win11 packaged** context menu (`Microsoft.UI.Content.PopupWindowSiteBridge` / "PopupHost"). The packaged module commands appear **only** here — not in classic `Shell.Application.Verbs()` and not via `CoCreate` of the command CLSID (`REGDB_E_CLASSNOTREGISTERED`). On Win10, or under "Show more options", you'd get the classic menu instead (different structure).
 - The menu exists in the UIA tree **only while open** — you must open it with real input first; you can't enumerate it cold.
 - A menu-launched module UI runs **non-elevated** (Explorer's integrity), even if your agent shell is elevated. Mind elevation-visibility (e.g. a non-elevated File Locksmith can't see higher-IL processes — match locker integrity with `scripts/pt-nonelevated.ps1`).
@@ -79,7 +79,7 @@ Match the **visible caption**, not the AutomationId (Explorer assigns per-sessio
 |---|---|---|
 | File Locksmith | `PowerToys.FileLocksmithUI.exe` | ✓ `Unlock with File Locksmith` (NB: **not** the checklist's "What's using this file?") |
 | PowerRename | `PowerToys.PowerRename.exe` | ✓ `Rename with PowerRename` |
-| Image Resizer | `PowerToys.ImageResizer.exe` | `Resize images` (verify via `Get-PtContextMenuItems` — caption shifted across versions) |
+| Image Resizer | `PowerToys.ImageResizer.exe` | ✓ `Resize with Image Resizer` (source string `ImageResizer_Context_Menu_Entry`; verify via `Get-PtContextMenuItems` — caption shifted across versions) |
 | New+ | (creates from template) | `New+` (submenu) |
 
 > Tip: if a module's caption is unknown, enable the module, open the menu on an applicable file, and run `Get-PtContextMenuItems` to read the exact string — then hard-match it for present/absent assertions.

.github/skills/powertoys-module-verification/references/modules/power-rename.md → .github/skills/powertoys-verification/references/modules/power-rename.md

@@ -37,15 +37,23 @@ Use the canonical flow from `references/explorer-context-menu-flow.md` Recipe. T
 
 ```powershell
 . "$skill\scripts\pt-explorer-contextmenu.ps1"
-$hwnd = Open-PtExplorerContextMenu -FolderPath 'D:\fixtures' -FileNames 'a.txt'
-$items = Get-PtContextMenuItems -MenuHwnd $hwnd
-$has = $items | Where-Object Name -match 'Rename with PowerRename'
+# Disposable fixtures folder (same convention as Entry-path 1)
+$fx = New-Item -ItemType Directory -Path "$env:TEMP\pr-fixture-$(Get-Random)"
+'x' | Set-Content "$($fx.FullName)\a.txt"
+# Open Explorer on it and grab its CabinetWClass HWND
+Start-Process explorer.exe $fx.FullName; Start-Sleep 4
+$hwnd = (winapp ui list-windows --json | ConvertFrom-Json |
+    Where-Object { $_.className -eq 'CabinetWClass' -and $_.title -match [regex]::Escape($fx.Name) } |
+    Select-Object -First 1).hwnd
+$menu  = Open-PtExplorerContextMenu -ExplorerHwnd $hwnd -FileName 'a.txt'
+$items = Get-PtContextMenuItems -MenuHwnd $menu        # returns MenuItem name strings
+$has   = $items | Where-Object { $_ -match 'Rename with PowerRename' }
 # assert $has -> entry present
 ```
 
 ### 3. Shell COM classic verb (does NOT work on Win11 stock install)
 ```powershell
-Invoke-PtShellVerb -Path 'D:\fixtures\a.txt' -NamePattern 'PowerRename'  # -> False
+Invoke-PtShellVerb -Path "$($fx.FullName)\a.txt" -NamePattern 'PowerRename'  # -> False (reuses $fx from Entry-path 2)
 ```
 Returns False on Win11 because PT registers PR only via IExplorerCommand, not as a classic HKCR shell verb. **Use only for negative checks** (and prefer the synthetic-menu enumeration above, which observes the actual Tier-1 menu).