[UITests] AI migration skill + ScreenRuler tests example port. by khmyznikov · Pull Request #48842 · microsoft/PowerToys

khmyznikov · 2026-06-24T20:47:17Z

This PR is build on top of #48717 and #48467

It adds the ui-tests-migration skill which should be used to migrate the old UITest module by module to use new .Next UI framework.

As an example, the screen ruller tests was ported here.

…ensions Add ComboBox, CheckBox, RadioButton, Slider, TextBlock wrappers (all driven via winapp ui invoke/get-property/get-value/set-value). Element: add DoubleClick (click --double), ScrollIntoView (scroll-into-view), live properties IsEnabled/IsOffscreen/Displayed/Selected/AutomationId + GetAttribute (get-property), WaitForValue with --contains. Make GetProperty tolerant of non-string/error output and expose EnsureBound to subclasses. Session: add WaitForElement (wait-for appear). Fix: TextBox set-value/get-value hardcoded -w <hwnd>, which targeted window 0 under process-scoped (-a) sessions; now uses the session's TargetFlag/TargetValue.

…stics ModulePaths: expand PowerToysModule enum to 10 modules; resolve exe via POWERTOYS_INSTALL_DIR override -> installed build -> repo dev-build output (x64/ARM64, Debug/Release). useInstallerForTest forces installed layout. Lets tests run against either an installed PowerToys or a local dev build. EnvironmentConfig: IsInPipeline / UseInstallerForTest / Platform (ported from legacy harness). SettingsConfigHelper: dependency-free (System.Text.Json.Nodes) ConfigureGlobalModuleSettings + UpdateModuleSettings, writing the per-user settings JSON directly (no Settings.UI.Library coupling). Session diagnostics (CLI-first): Screenshot element-crop + --capture-screen + non-asserting TryScreenshot; Inspect --interactive/--hide-disabled/--hide-offscreen; InspectAncestors; GetFocused/GetFocusedName. UITestBase: capture a --capture-screen PNG and attach it on test failure. SessionHelper: RestartScope (kill -> wait exit -> relaunch + wait window).

…dow helpers) MouseHelper: add GetMousePosition, LeftDown/Up, RightDown/Up, MiddleDown/Up/Click, DoubleClick, ScrollWheel/Up/Down, and a stepped Drag(from,to). winappcli has no drag/wheel/raw-cursor verbs, so these stay Win32. KeyboardHelper: extend Key enum with digits, F1-F12, arrows, Home/End/PageUp/PageDown/Insert; add PressKey/ReleaseKey/SendKey/SendKeySequence with extended-key handling for nav keys. Element: add CLI-first Scroll(direction)/ScrollToEdge (winapp ui scroll) plus Win32 Drag/DragTo/KeyDownAndDrag using the element's search-reported center. Elements: add Pane/Thumb/Custom/Tab wrappers (drag inherited from Element). WindowHelper (new): WindowSize enum + SetWindowSize/SetMainWindowSize, GetWindowBounds/Center, GetDisplaySize/GetScreenCenter, GetPixelColor/GetPixelColorHex (GDI) — lets ColorPicker-style tests read on-screen pixels without a hidden XAML peer. IsWindowOpen stays CLI-based via WindowsFinder. Session: add Attach(module, size) — window-scoped session with optional preset resize.

ElevationHelper (new): IsCurrentProcessElevated / IsProcessElevated via OpenProcessToken + TokenElevation. Session.IsElevated surfaces the target process's elevation (null when no PID). MonitorInfo (new): GetAll / GetPrimary / Count via EnumDisplayMonitors + GetMonitorInfo, returning per-display bounds, work area, and primary flag for multi-monitor utility tests. Session.Status(): winapp ui status --json for connection diagnostics. Intentionally deferred (heavy / external-dep, and existing primitives already cover the common cases): perceptual-hash VisualAssert (GetPixelColor + Screenshot cover basic visual checks) and FFmpeg ScreenRecording (failure --capture-screen screenshots cover diagnostics).

Migrate the PowerToys module release-checklist verification skill into .github/skills/ following the agent-skills layout: SKILL.md + LICENSE.txt, scripts/ (12 PT helper .ps1), and references/ (UIA mechanics doc, per-module profiles, flow/format/setup docs). Folds the former entry-prompt into SKILL.md and demotes the winapp UI-testing doc to references/winapp-ui-testing.md (frontmatter stripped, MIT provenance retained). Only checklists for modules already verified end-to-end (with a Module-Signoff report) are included for now: Environment Variables, File Locksmith, Image Resizer, New+, Peek, PowerRename. Remaining modules' checklists will be added as each is verified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…on-skill' into pt-team/ui-tests-conversion

…ersion

Copilot

Pull request overview

Adds a new winappcli-based UI test migration skill and ports an example module (Screen Ruler) to the new .Next UI test harness, while also updating CI to provision/run the new runner style and aligning some repo-wide build/test infrastructure for .NET 10.

Changes:

Introduces/extends Microsoft.PowerToys.UITest.Next (winappcli-driven) helpers and adds new .Next UI test projects (Settings smoke nav + Screen Ruler port).
Updates pipelines to install winappcli, run UI tests via Microsoft.Testing.Platform runners, and improves WinAppDriver reuse for legacy tests.
Updates solution wiring/docs/spellcheck entries and moves fuzzing guidance/projects to net10.

Reviewed changes

Copilot reviewed 74 out of 76 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
src/settings-ui/Settings.UITests/SettingsTests.md	Adds a manual Settings UI test checklist doc.
src/settings-ui/Settings.UITests/SettingsNavigationSmokeTests.cs	New winappcli-based Settings navigation smoke test.
src/settings-ui/Settings.UITests/Settings.UITests.csproj	Adds Settings UI tests project (MTP runner + harness ref).
src/modules/MeasureTool/Tests/ScreenRuler.UITests.Next/TestSpacingVertical.cs	Ports vertical spacing UI test to `.Next` harness.
src/modules/MeasureTool/Tests/ScreenRuler.UITests.Next/TestSpacingHorizontal.cs	Ports horizontal spacing UI test to `.Next` harness.
src/modules/MeasureTool/Tests/ScreenRuler.UITests.Next/TestSpacing.cs	Ports spacing UI test to `.Next` harness.
src/modules/MeasureTool/Tests/ScreenRuler.UITests.Next/TestShortcutActivation.cs	Ports shortcut activation UI test to `.Next` harness.
src/modules/MeasureTool/Tests/ScreenRuler.UITests.Next/TestBounds.cs	Ports bounds UI test to `.Next` harness.
src/modules/MeasureTool/Tests/ScreenRuler.UITests.Next/ScreenRuler.UITests.Next.csproj	Adds Screen Ruler `.Next` UI tests project.
src/modules/MeasureTool/Tests/ScreenRuler.UITests.Next/app.manifest	Adds PerMonitorV2 DPI awareness for coordinate-precise tests.
src/modules/colorPicker/UITest-ColorPicker/UITest-ColorPicker.csproj	Removes legacy WinAppDriver/Appium ColorPicker UI test stub project.
src/modules/colorPicker/UITest-ColorPicker/ColorPickerUITest.cs	Removes legacy stub test class.
src/modules/colorPicker/ColorPickerUI/Views/MainView.xaml	Adds a hidden AutomationId hook for reading HEX value in UIA.
src/modules/colorPicker/ColorPicker.UITests/ColorPickerUITest.md	Adds Color Picker manual test checklist doc.
src/modules/colorPicker/ColorPicker.UITests/ColorPicker.UITests.csproj	Adds ColorPicker `.Next` UI tests project.
src/modules/colorPicker/ColorPicker.UITests/AssemblyInfo.cs	Forces sequential execution for UI tests via DoNotParallelize.
src/common/UITestAutomation/SessionHelper.cs	Improves WinAppDriver lifecycle (reuse server; stdin redirection; readiness wait).
src/common/UITestAutomation/ModuleInfo.cs	Adjusts legacy dev-path resolution to handle RID output folders.
src/common/UITestAutomation.Next/Windows.cs	Adds window discovery/wait helpers via `winapp ui list-windows --json`.
src/common/UITestAutomation.Next/WindowHelper.cs	Adds Win32 window sizing/pixel helpers for tests.
src/common/UITestAutomation.Next/WindowControl.cs	Adds tolerant window/process cleanup helpers for finally blocks.
src/common/UITestAutomation.Next/WinappCli.cs	Adds winapp.exe wrapper with timeouts, JSON handling, and alias-launch retry.
src/common/UITestAutomation.Next/UITestAutomation.Next.csproj	Adds the `.Next` harness library project.
src/common/UITestAutomation.Next/SettingsConfigHelper.cs	Adds JSON-only settings writer helpers (no Settings UI lib dependency).
src/common/UITestAutomation.Next/ScreenRecording.cs	Adds FFmpeg-based desktop recording for CI diagnostics (pipeline-gated).
src/common/UITestAutomation.Next/ScreenCapture.cs	Adds timer-based desktop screenshots for CI diagnostics (pipeline-gated).
src/common/UITestAutomation.Next/MouseHelper.cs	Adds global mouse input helpers (SetCursorPos/mouse_event + drag).
src/common/UITestAutomation.Next/MonitorInfo.cs	Adds multi-monitor enumeration helpers via Win32.
src/common/UITestAutomation.Next/ModuleConfigData.cs	Adds module scope enum and exe/process/title resolution logic for `.Next`.
src/common/UITestAutomation.Next/KeyboardHelper.cs	Adds global hotkey injection helpers (hybrid keybd_event + SendKeys).
src/common/UITestAutomation.Next/FRAMEWORK-PARITY-PLAN.md	Documents parity plan/status between legacy and `.Next` harnesses.
src/common/UITestAutomation.Next/EnvironmentConfig.cs	Adds env var helpers for pipeline/installer targeting.
src/common/UITestAutomation.Next/ElevationHelper.cs	Adds elevation detection helpers for tests.
src/common/UITestAutomation.Next/Element/Window.cs	Adds typed Window element wrapper.
src/common/UITestAutomation.Next/Element/ToggleSwitch.cs	Adds ToggleSwitch wrapper with ToggleState read/toggle semantics.
src/common/UITestAutomation.Next/Element/Thumb.cs	Adds Thumb wrapper for drag scenarios.
src/common/UITestAutomation.Next/Element/TextBox.cs	Adds TextBox wrapper for get/set value.
src/common/UITestAutomation.Next/Element/TextBlock.cs	Adds TextBlock wrapper for reading displayed text.
src/common/UITestAutomation.Next/Element/Tab.cs	Adds Tab wrapper.
src/common/UITestAutomation.Next/Element/Slider.cs	Adds Slider wrapper for get/set range values.
src/common/UITestAutomation.Next/Element/RadioButton.cs	Adds RadioButton wrapper for selection.
src/common/UITestAutomation.Next/Element/Pane.cs	Adds Pane wrapper.
src/common/UITestAutomation.Next/Element/NavigationViewItem.cs	Adds NavigationViewItem wrapper.
src/common/UITestAutomation.Next/Element/Custom.cs	Adds Custom control wrapper.
src/common/UITestAutomation.Next/Element/ComboBox.cs	Adds ComboBox wrapper (select + set-value).
src/common/UITestAutomation.Next/Element/CheckBox.cs	Adds CheckBox wrapper.
src/common/UITestAutomation.Next/Element/Button.cs	Adds Button wrapper.
src/common/UITestAutomation.Next/DisplayHelper.cs	Adds CI resolution normalization + monitor topology logging.
src/common/UITestAutomation.Next/ClipboardHelper.cs	Adds STA-safe clipboard helper methods with polling.
src/common/UITestAutomation.Next/By.cs	Adds selector abstraction matching winappcli selector types.
src/Common.Dotnet.FuzzTest.props	Moves fuzz test TFM from net8 to net10 and updates guidance comment.
PowerToys.slnx	Registers new projects and reorders some project entries/dependencies.
doc/devdocs/tools/fuzzingtesting.md	Updates fuzz testing docs to .NET 10 and new artifact path examples.
Cpp.Build.props	Adds MSVC/VS2026 coroutine deprecation suppression define.
AGENTS.md	Updates fuzz test prerequisite version to .NET 10.
.pipelines/v2/templates/job-test-project.yml	Updates UI test job to install .NET 10 + winappcli and run MTP UI test runners; adds shared WinAppDriver start/stop.
.pipelines/v2/templates/job-build-project.yml	Moves common-props audit earlier in build job.
.pipelines/InstallWinAppCli.ps1	Adds script to download/hash-verify/install pinned winappcli in CI.
.github/skills/ui-tests-migration/templates/TestHelper.cs	Adds migration template helper for `.Next` test projects.
.github/skills/ui-tests-migration/templates/ModuleEndToEndTests.cs	Adds migration template test class scaffold.
.github/skills/ui-tests-migration/templates/Module.UITests.Next.csproj	Adds migration template csproj scaffold.
.github/skills/ui-tests-migration/templates/app.manifest	Adds migration template PerMonitorV2 manifest.
.github/skills/ui-tests-migration/references/project-setup.md	Adds docs on naming, placement, csproj setup, solution registration, running.
.github/skills/ui-tests-migration/references/porting-workflow.md	Adds step-by-step workflows for porting vs greenfield test creation.
.github/skills/ui-tests-migration/references/framework-differences.md	Documents conceptual differences between legacy and `.Next` harness.
.github/skills/ui-tests-migration/LICENSE.txt	Adds Apache 2.0 license for the skill bundle.
.github/actions/spell-check/expect.txt	Adds `winapp` / `winappcli` to spellcheck allowlist.

+    <Nullable>enable</Nullable>
+    <IsPackable>false</IsPackable>
+    <TreatWarningsAsErrors>false</TreatWarningsAsErrors>
+    <RootNamespace>Microsoft.Settings.UITests</RootNamespace>


+    [TestMethod]
+    [TestCategory("Settings")]
+    [TestCategory("winappcli-POC")]
+    [DynamicData(nameof(NavigationCases), DynamicDataDisplayName = nameof(GetNavCaseDisplayName))]
+    public void NavigationItem_NavigatesWithoutCrashing(string parentGroupSlug, string navItemSlug)


+      injection in KeyboardHelper. (Same approach as the legacy harness.)
+    -->
+    <UseWindowsForms>true</UseWindowsForms>
+    <TreatWarningsAsErrors>false</TreatWarningsAsErrors>
+    <RootNamespace>Microsoft.PowerToys.UITest.Next</RootNamespace>
+    <AssemblyName>Microsoft.PowerToys.UITest.Next</AssemblyName>


+    <Nullable>enable</Nullable>
+    <IsPackable>false</IsPackable>
+    <TreatWarningsAsErrors>false</TreatWarningsAsErrors>
+    <RootNamespace>Microsoft.ColorPicker.UITests</RootNamespace>
+    <AssemblyName>ColorPicker.UITests</AssemblyName>


+    <Nullable>enable</Nullable>
+    <IsPackable>false</IsPackable>
+    <TreatWarningsAsErrors>false</TreatWarningsAsErrors>
+    <RootNamespace>Microsoft.ScreenRuler.UITests</RootNamespace>
+    <AssemblyName>ScreenRuler.UITests.Next</AssemblyName>


+    <TreatWarningsAsErrors>false</TreatWarningsAsErrors>
+    <RootNamespace>Microsoft.ScreenRuler.UITests</RootNamespace>
+    <AssemblyName>ScreenRuler.UITests.Next</AssemblyName>
+


+    [TestMethod]
+    [TestCategory("Activation")]
+    public void TestScreenRulerShortcutActivation()
+    {
+        var activationKeys = TestHelper.InitializeTest(this, "activation test");


+        private static void WaitForWinAppDriverReady(int timeoutMs = 30000)
+        {
+            var deadline = DateTime.UtcNow.AddMilliseconds(timeoutMs);
+            while (DateTime.UtcNow < deadline)
+            {
+                if (IsWinAppDriverListening())
+                {
+                    return;
+                }
+
+                System.Threading.Thread.Sleep(500);
+            }
        }


+      foreach ($rc in ($entries | Sort-Object FullName -Unique)) {
+        $base = $rc.Name -replace '\.runtimeconfig\.json$', ''
+        $dir = $rc.DirectoryName
+        $exe = Join-Path $dir "$base.exe"
+        $dll = Join-Path $dir "$base.dll"
+        Write-Host "##[group]Run UI tests: $base"
+        Push-Location $dir
+        try {
+          if (Test-Path $exe) {
+            & $exe --report-trx --results-directory $resultsDir
+          } elseif (Test-Path $dll) {
+            & dotnet $dll --report-trx --results-directory $resultsDir
+          } else {
+            Write-Warning "No runner (exe/dll) found for $base in $dir"
+          }
+          if ($LASTEXITCODE -ne 0) {
+            Write-Warning "UI tests reported failures for $base (exit $LASTEXITCODE)"
+            $failed++
+          }
+        } finally {
+          Pop-Location
+          Write-Host "##[endgroup]"
+        }
+      }


khmyznikov and others added 30 commits June 10, 2026 11:18

initial commit

3bc472b

remove smaller tests

c37eaf0

move n rename

cf3d132

Align the settings test with framework

7ee6bc6

Rename settings test

63bd903

add more stability

5740651

add plat for later

715e1bd

add winappcli installation

6ddfbde

Fix the audit step

45da84b

Move the audit to before the full build

1208c01

workarounds for winappcli install

bfd089f

disable onefuzz

6183b99

fuzzing bump to 10

eea7029

fix the tests output

0a93c4d

update framework

892fe24

fix the build location

40044ae

fix env config

7ead886

try to fix the settings launch

858b9ea

try to resolve the project startup

0b0a698

try to fix the ColorPicker UI pickup

5c487a5

address the PR review.

f13dd6e

Try to fix sessions helper

96d8d70

add shell execute for old framework

446009d

fix the nudge

9c9566d

try to fix old test workflow

8dcd2d4

khmyznikov added 6 commits June 23, 2026 13:45

fix the settings window sizing

8900ae0

Merge branch 'gleb/ui-tests-2' into pt-team/ui-tests-conversion

9340473

Merge remote-tracking branch 'origin/LegendaryBlair/module-verificati…

76b2046

…on-skill' into pt-team/ui-tests-conversion

migration skill

f84c4bb

port ScreenRuler tests

901383a

add test helper template

10db8a3

khmyznikov requested a review from a team as a code owner June 24, 2026 20:47

github-actions Bot added the Product-Screen Ruler Refers to the Screen Ruler PowerToy label Jun 24, 2026

Merge remote-tracking branch 'origin/main' into pt-team/ui-tests-conv…

7cf145b

…ersion

khmyznikov marked this pull request as draft June 24, 2026 20:59

khmyznikov requested review from LegendaryBlair and Copilot June 24, 2026 20:59

Copilot started reviewing on behalf of khmyznikov June 24, 2026 20:59 View session

Copilot AI reviewed Jun 24, 2026

View reviewed changes

khmyznikov added 7 commits June 24, 2026 15:50

fix the settings window offscreen and cursor park

004fd63

supress OOBE

fb88583

make the single test config more explicit

45e912a

add retries for ScreenRuler capture

3e8240a

additional workaround

512c979

unified workaround

76d707c

test fixes

623d505

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[UITests] AI migration skill + ScreenRuler tests example port.#48842

[UITests] AI migration skill + ScreenRuler tests example port.#48842
khmyznikov wants to merge 44 commits into
mainfrom
pt-team/ui-tests-conversion

khmyznikov commented Jun 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

khmyznikov commented Jun 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants