avifenesh · Oct 24, 2026 · Oct 24, 2026 · Oct 24, 2026 · Oct 24, 2026 · Oct 24, 2026
@@ -12,7 +12,7 @@
       "name": "computer-use-linux",
       "source": "./plugins/computer-use-linux",
       "description": "Native Linux desktop control: semantic AT-SPI accessibility trees, screenshots, compositor window targeting (GNOME, KDE, Hyprland, niri, COSMIC, i3, X11), Wayland/X11 input, and an on-screen activity indicator.",
-      "version": "0.7.22",
+      "version": "0.7.23",
       "category": "productivity"
     }
   ]

@@ -7,13 +7,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.7.23] - 2026-10-24
+
 ### Fixed
 
 - Report failed desktop actions and intentional refusals as MCP tool errors with `isError: true`, while preserving successful results when only post-action observation fails. Thanks @OnlyHa1bara. (#280)
+- Read stable machine-readable AT-SPI action names so Qt buttons use native click/press/toggle activation instead of pointer fallback. Thanks @sinantux for the detailed reports. (#281)
+- Map KWin element clicks using coherent frame, client, output and scale geometry. Reject ambiguous window ownership, unknown desktop visibility, mixed output scales and stale pointer dimensions before sending input. (#281)
+- Keep KWin Save/Open dialogs, confirmation prompts and verified interactive transients targetable, including dialogs that skip the taskbar. (#286)
+- Refresh stale semantic AT-SPI targets once before action dispatch, preserving the original app scope and never replaying numeric-index or already-dispatched actions. (#282)
+- Scope `get_app_state` by short or reverse-DNS `app_id` values, reject ambiguous aliases, and avoid whole-desktop fallback for unresolved explicit app IDs. (#283)
+- Verify editable-text `set_value` results with exact full-content readback, report mismatches or verification failures instead of false success, and mark password fields unverifiable. (#284)
+- Reject unsupported X11 PointerRoot keyboard monitoring before typing, preserve one xdotool dispatch with an explicit monitoring warning, and never replay the text. (#272)
 
 ### Security
 
-- Update Tokio to 1.53.3 and register the GNOME indicator socket with the I/O-safe `AsyncFd::register` API, addressing RUSTSEC-2026-0386. (#280)
+- Update Tokio to 1.53.3 and register the GNOME indicator socket with the I/O-safe `AsyncFd::register` API, addressing RUSTSEC-2026-0386. Thanks @OnlyHa1bara. (#280)
+- Update Pi and schema-check dependencies, including patched Undici, Vitest, Zod, body-parser, fast-uri and ip-address. Regenerate the bundled MCP client. (#285)
 
 ## [0.7.22] - 2026-10-20
 
@@ -857,7 +867,8 @@ pages; also bumps the MCP server's advertised version string to match.
 - Validated against GNOME 50.1 on Wayland (Ubuntu 25.10).
 - KDE / Sway / Hyprland untested — see README support matrix.
 
-[Unreleased]: https://github.com/agent-sh/computer-use-linux/compare/v0.7.22...HEAD
+[Unreleased]: https://github.com/agent-sh/computer-use-linux/compare/v0.7.23...HEAD
+[0.7.23]: https://github.com/agent-sh/computer-use-linux/compare/v0.7.22...v0.7.23
 [0.7.22]: https://github.com/agent-sh/computer-use-linux/compare/v0.7.21...v0.7.22
 [0.7.21]: https://github.com/agent-sh/computer-use-linux/compare/v0.7.20...v0.7.21
 [0.7.20]: https://github.com/agent-sh/computer-use-linux/compare/v0.7.19...v0.7.20

@@ -1,6 +1,6 @@
 [package]
 name = "computer-use-linux"
-version = "0.7.22"
+version = "0.7.23"
 edition = "2021"
 authors = ["Avi Fenesh <avifenesh@users.noreply.github.com>"]
 license = "MIT"

@@ -106,16 +106,22 @@ The preference does not substitute an arbitrary action name for a coordinate
 click. Explicit `x`/`y`, right clicks, and double/multiple clicks retain pointer
 semantics. Re-check application state after either kind of activation.
 
+On KDE Wayland, element clicks that need pointer dispatch match the cached AT-SPI frame to a coherent KWin window/output snapshot, then map the target separately to physical capture pixels (uinput) and logical desktop coordinates (portal). Client-local and decorated frames, actual decoration offsets, and nonzero or negative output origins are supported when the geometry and uniform output scale can be verified. Ambiguous windows, mixed or unknown scales, overlapping outputs, or stale pointer geometry fail before input; explicit coordinates keep their existing behavior.
+
 For coordinate `click` or `scroll` with `relative: true`, select a target window
 and measure from its clipped screenshot crop origin. Divide preview `x` and
 `y` by the returned screenshot `scale` before passing them. These are not raw
 GDK surface or widget-local coordinates; decorations and clipping can change
 the origin. A missing window target is rejected.
 
-**X11 text fidelity:** `xdotool`/XTEST can silently substitute symbols under non-US XKB layouts (reported with Brazilian `br`/ABNT2) or changing keymap state, even when `type_text` returns `ok: true`. Every successful xdotool text dispatch includes a layout warning. XKB layout replacement events, persistent mapping/group changes, and keyboard focus/grab transitions detected across a call return an error after dispatch, without replaying the text. Keyboard grabs can divert characters without changing the polled focused-window ID. If monitoring cannot start, xdotool still dispatches and the response explicitly warns that layout-change and keyboard interception detection were unavailable. This also detects a `setxkbmap` layout changed and restored during one call. Temporary raw remaps or group switches restored within a call and dropped text with a stable map remain unverified. Re-observe the actual text before pressing Enter, submitting it, or treating it as delivered. Use `include_tree: true` for AT-SPI readback when the application exposes the text, or `screenshot_after: true` to inspect it. These observations do not automatically compare the field to the requested text. Do not retry blindly.
+**X11 text fidelity:** `xdotool`/XTEST can silently substitute symbols under non-US XKB layouts (reported with Brazilian `br`/ABNT2) or changing keymap state, even when `type_text` returns `ok: true`. Every successful xdotool text dispatch includes a layout warning. XKB layout replacement events, persistent mapping/group changes, and keyboard focus/grab transitions detected across a call return an error after dispatch, without replaying the text. Keyboard grabs can divert characters without changing the polled focused-window ID. If monitoring cannot start, xdotool still dispatches and the response explicitly warns that layout-change and keyboard interception detection were unavailable. PointerRoot focus is unsupported because pointer motion can redirect input without a detectable focus event. This also detects a `setxkbmap` layout changed and restored during one call. Temporary raw remaps or group switches restored within a call and dropped text with a stable map remain unverified. Re-observe the actual text before pressing Enter, submitting it, or treating it as delivered. Use `include_tree: true` for AT-SPI readback when the application exposes the text, or `screenshot_after: true` to inspect it. These observations do not automatically compare the field to the requested text. Do not retry blindly.
 
 Targeted `press_key`/`type_text` results append focused-element feedback from AT-SPI (role, name, editable) and warn when a concrete non-terminal focus is not editable. Terminals accept keyboard input without an EditableText interface; re-observe the contents to confirm input. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. `get_app_state` returns a compact readiness block by default; pass `verbose: true` for the full diagnostics report. It also reports `tree_scoped` (false when no app target narrowed the AT-SPI tree, with a warning in `message`) and `accessibility_tree_truncated` (true when the node, depth, or read budget stopped traversal with unread elements left).
 
+An `app_id` may be a full desktop ID or its final dot-separated segment; matching is case-insensitive, ambiguous aliases are rejected, and an unresolved explicit `app_id` returns an empty scoped tree rather than the desktop.
+
+`set_value` on editable text reads the full contents back after dispatch. `verified: true` means an exact match; mismatches and readback failures return `ok: false` without replaying the mutation. Password fields return `verified: false` because masked content cannot prove the write. Re-observe after any mismatch; the contents may have changed.
+
 On X11, text input sends LF and CRLF line endings as Return, with one newline per CRLF pair.
 
 On GNOME, a read-only keymap preflight rejects the whole string before input if a character is absent or requires a modifier level Mutter cannot synthesize. The background reader cannot establish the active layout group, so every configured group must support the text. This can reject text that works in one of several groups; use `set_value` on an editable field instead. The check requires `libxkbcommon.so.0` at runtime and stops after a two-second keymap read deadline. It does not cover held modifiers or a keymap changed concurrently with typing.

@@ -1,6 +1,6 @@
 {
   "name": "@agent-sh/computer-use-linux",
-  "version": "0.7.22",
+  "version": "0.7.23",
   "description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
   "license": "MIT",
   "type": "commonjs",

@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
 	annotations: Record<string, unknown>;
 }
 
-export const GENERATED_SERVER_VERSION = "0.7.22";
-export const GENERATED_TOOL_CATALOG_HASH = "b9a6ef2736cd165b97982ec6ccf1e6cfb47712f0d4a8ae778ecac47e211b15ec";
-export const GENERATED_SHELL_TOOL_CATALOG_HASH = "113fc01364844db492475ec7ad00f1d03fe8e2d7acedc8a0aa3b48d245f99105";
+export const GENERATED_SERVER_VERSION = "0.7.23";
+export const GENERATED_TOOL_CATALOG_HASH = "3e2d6cbbf31957fcdc8d05e66f959ad1c3b277685dfa30ed3c3b2f4df451eec6";
+export const GENERATED_SHELL_TOOL_CATALOG_HASH = "052c430ada325bd74fd764fc3f9d1139f4e6e401b6a5dbc6ef8a3de538214241";
 export const GENERATED_MCP_TOOLS =
 [
   {
@@ -1794,7 +1794,7 @@ export const GENERATED_MCP_TOOLS =
       "openWorldHint": true,
       "readOnlyHint": false
     },
-    "description": "Set the value of a settable accessibility element selected by index, identifier, or semantic selector. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
+    "description": "Set the value of a settable accessibility element selected by index, identifier, or semantic selector. Editable-text writes are verified by exact readback; password fields are explicitly unverifiable. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
     "inputSchema": {
       "$defs": {
         "ScreenshotOutputFormat": {

@@ -1,6 +1,6 @@
 {
   "name": "computer-use-linux",
-  "version": "0.7.22",
+  "version": "0.7.23",
   "description": "Native Linux desktop control: semantic AT-SPI accessibility trees, screenshots, compositor window targeting (GNOME, KDE, Hyprland, niri, COSMIC, i3, X11), Wayland/X11 input, and an on-screen activity indicator.",
   "author": {
     "name": "agent-sh",

@@ -1,6 +1,6 @@
 {
   "name": "computer-use-linux",
-  "version": "0.7.22",
+  "version": "0.7.23",
   "description": "Native Linux desktop control: semantic AT-SPI accessibility trees, screenshots, compositor window targeting (GNOME, KDE, Hyprland, niri, COSMIC, i3, X11), Wayland/X11 input, and an on-screen activity indicator.",
   "author": {
     "name": "agent-sh",

@@ -5,7 +5,7 @@
 # message goes to stderr.
 set -eu
 
-version=0.7.22
+version=0.7.23
 
 say() {
   echo "computer-use-linux plugin: $*" >&2

@@ -134,7 +134,7 @@ Pi native tools skip this MCP `command` config; see [Pi setup](references/pi-set
 ## Procedure
 
 1. In Pi, call `computer_use_linux_tools` with the exact tools or capability you need. Enabled tools use the `computer_use_linux_<name>` prefix, appear starting on the next model turn, and remain active for the session.
-2. Begin every desktop-control turn with `get_app_state`, scoped to the app you are working in: pass `app_name_or_bundle_identifier` or a window target (`window_id`, `pid`, `app_id`, `wm_class`, `title`). Without a target the result is the whole desktop AT-SPI tree, `tree_scoped` is `false`, and `message` warns; that can flood context. Use `include_screenshot: false` when the accessibility tree is sufficient. If `accessibility_tree_truncated` is `true`, the tree is incomplete: scope to a narrower target and raise `max_nodes` or `max_depth` (hard caps 2000 and 64) rather than lowering them. The compact readiness block identifies missing setup.
+2. Begin every desktop-control turn with `get_app_state`, scoped to the app you are working in: pass `app_name_or_bundle_identifier` or a window target (`window_id`, `pid`, `app_id`, `wm_class`, `title`). An `app_id` may be a full desktop ID or its final dot-separated segment; matching is case-insensitive, ambiguous aliases are rejected, and an unresolved explicit `app_id` returns an empty scoped tree rather than the desktop. Without a target the result is the whole desktop AT-SPI tree, `tree_scoped` is `false`, and `message` warns; that can flood context. Use `include_screenshot: false` when the accessibility tree is sufficient. If `accessibility_tree_truncated` is `true`, the tree is incomplete: scope to a narrower target and raise `max_nodes` or `max_depth` (hard caps 2000 and 64) rather than lowering them. The compact readiness block identifies missing setup.
 3. Use `doctor` only when you need the full diagnostic report.
 4. If `can_build_accessibility_tree` is false, run `setup_accessibility` and restart the target app.
 5. If `can_query_windows` is false on GNOME Wayland, run `setup_window_targeting` and ask the user to log out and back in if setup says the shell extension needs a reload.
@@ -143,6 +143,7 @@ Pi native tools skip this MCP `command` config; see [Pi setup](references/pi-set
 8. Use coordinates only when the UI surface has no useful accessibility tree.
 9. For text input, prefer `type_text` with a target selector (`window_id`, `pid`, `app_id`, `wm_class`, `title`, `tty`, `terminal_pid`, `terminal_command`, or `terminal_cwd`) rather than relying on current focus.
 10. After mutating actions (`click`, `drag`, `scroll`, `press_key`, `type_text`, `perform_action`, `set_value`, `activate_window`, `move_window`, `resize_window`), either pass `screenshot_after: true | "screen" | "window"` (with optional `format`, `quality`, `scale`, `max_width`, `max_height`, `max_bytes`) and/or `include_tree: true` on the action call itself to observe the result in the same tool call, or re-check state with `get_app_state`, `focused_window`, or an app-specific readback.
+11. For `set_value`, `verified: true` is the only exact-content confirmation. `verified: false` requires fresh observation; password fields are deliberately unverifiable. The field is omitted for numeric values and early failures.
 
 On Hyprland, `type_text` passes one literal wtype argument after `--` for a
 classified nonsecret field, avoiding stdin keymap-refresh corruption in Electron.

@@ -2,7 +2,8 @@
 """Opt-in Xvfb/MCP regression for issue #272, isolated from the real desktop.
 
 Requires Xvfb, xterm, xdotool, xkbcomp, setxkbmap and dbus-run-session. Checks byte-exact
-stable input, restored layout changes, keyboard grabs and restored focus changes.
+stable input, restored layout changes, keyboard grabs, restored focus changes, and
+PointerRoot/no-fixed-focus dispatch behavior.
 """
 import json
 import os
@@ -100,6 +101,12 @@ def bind(name, result, args):
 def focus():
     value=W(); revert=I();get_focus(d,ctypes.byref(value),ctypes.byref(revert));return value.value
 before=focus()
+if sys.argv[1] in ("pointer-root", "focus-none"):
+    focus_id = 1 if sys.argv[1] == "pointer-root" else 0
+    set_focus(d,focus_id,1,0);sync(d,0)
+    assert focus() == focus_id
+    close(d)
+    sys.exit(0)
 w=create(d,root(d),0,0,10,10,0,0,0); map_window(d,w);sync(d,0)
 if sys.argv[1] == "grab": assert grab(d,w,0,1,1,0) == 0
 else: set_focus(d,w,1,0);sync(d,0)
@@ -130,6 +137,8 @@ def focus():
                 ("layout-restored-3", "TESTE:;'\"[]{}<>~^\\|!@#%&*()_+=?/abcXYZ0198" * 3, "layout"),
                 ("keyboard-grab-stable-layout-and-focus", "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ" * 2, "grab"),
                 ("focus-restored", "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ" * 2, "focus"),
+                ("focus-none", "No fixed focus still dispatches\n", "focus-none"),
+                ("pointer-root", "Pointer root still dispatches\n", "pointer-root"),
             ]):
                 capture = Path(scratch) / f"{index}.txt"
                 marker = Path(scratch) / f"{index}.dispatch"
@@ -157,8 +166,12 @@ def focus():
                         "protocolVersion": "2025-11-25", "capabilities": {},
                         "clientInfo": {"name": "x11-type-guard-test", "version": "1"}}})
                     rpc(mcp, {"jsonrpc": "2.0", "method": "notifications/initialized"})
+                    focus_mode = disturbance in ("pointer-root", "focus-none")
+                    if focus_mode:
+                        subprocess.run([sys.executable, str(interrupt), disturbance],
+                                       env=env, check=True, timeout=5)
                     before_map = subprocess.check_output(["xkbcomp", "-xkb", env["DISPLAY"], "-"], env=env, stderr=subprocess.DEVNULL, timeout=5)
-                    if disturbance:
+                    if disturbance and not focus_mode:
                         def replace_layout():
                             try:
                                 if not select.select([notify], [], [], 15)[0]:
@@ -188,7 +201,15 @@ def replace_layout():
                     output = result.get("structuredContent") or json.loads(
                         next(item["text"] for item in result["content"] if item["type"] == "text"))
                     assert marker.read_text().splitlines() == ["type"], "text was replayed"
-                    if disturbance:
+                    if focus_mode:
+                        assert output["ok"], output
+                        assert not result.get("isError", False), result
+                        assert "Action sent through xdotool (X11 XTEST)." in output["message"], output
+                        if disturbance == "pointer-root":
+                            assert "X11 keyboard monitoring does not support PointerRoot" in output["message"], output
+                        else:
+                            assert "layout-change detection was unavailable" not in output["message"], output
+                    elif disturbance:
                         assert not output["ok"], output
                         if disturbance == "layout":
                             assert "layout changed during type_text" in json.dumps(output), output