Conversation
Agents can list the desktop's applications and windows and read the windows a person shares with them (screenshot, accessibility tree, bounded predicate checks) through five computer_* tools on codeg-mcp. codeg never holds or uses the macOS permissions involved: every agent's shell inherits codeg's TCC identity. The executor is a new codeg-computer-helper binary, launched disclaimed over an inherited socketpair, which checks codeg's code signature before serving and runs the pinned cua-driver 0.28.2 as its child. The driver is started suspended and checked as a running image (designated requirement, per-architecture cdhash, hardened runtime, exact entitlements) before it runs, with a scrubbed environment. Grants are per window, never automatic, re-checked after every read, bound to the process start time, and lapse when unused. codeg's own windows and credential managers can never be shared. The group is off by default and re-read at call time; switching it off ends every grant and stops the helper.
A status-bar item (shown only while computer use is on) lists the shared windows with a stop button each, recent agent activity, the helper's state and its macOS permissions with request and open-settings actions, and warns when codeg itself holds a permission its agents would inherit. The window picker shows every window with a helper-captured thumbnail and shares or unshares one for reading. Collaboration settings gain the computer-use switch and a section for the grant timeout and the user-added blocklist, each written through its own narrow endpoint.
…elease The helper is built and staged next to codeg-mcp as a second externalBin sidecar. Release builds pin both trust anchors (codeg's and the helper's designated requirements) from the Team ID before compiling, and the macOS job now fails unless codeg, codeg-mcp and the helper carry no entitlements, use the hardened runtime, satisfy each other's pinned requirement, and codeg and codeg-mcp reference no Accessibility, event-posting or screen-capture API.
Helper boundary: - bind the helper to its launcher: the verified peer must be the parent process, and every request is checked against the verified audit token (LOCAL_PEERTOKEN names the last process to use the socket, not its owner) - spawn the driver under an AMFI launch requirement built from the pins (team, identifier, cdhashes), and the release helper under one naming this build's Team ID; a suspended child can be resumed by any process of the user, so the kernel check is the gate and the suspended check a second look. No launch requirement API (macOS < 14.4): no driver - register the child's exit watch before handing out a handle and never block under its lock; exit the helper when codeg goes away even with a request parked on a permission prompt - bound every write to the helper and to the driver, and give up on a peer that stopped reading Grant model: - re-check the switch (including an off-and-on the watcher would see as one change), the blocklist and the process identity after every read - windows without a process start time or an application key cannot be shared - end lapsed and newly blocklisted grants before anything is listed and on every settings change; never restart the helper after switch-off - make target retirement idempotent so a late "window gone" cannot unmap the id the same window has since been given - end a grant when the wall clock steps back by more than a minute Driver ops: - redact secrets per tree node, including multi-line values and the value="…" form the Windows and Linux trees use - treat a listing or snapshot answer without its payload as malformed Settings and UI: - write the grant timeout and blocklist field by field, lock the form until the stored values are read and while saving - drop fetched state an event has overtaken in the status bar and picker - keep the driver's cache directory out of the agents' namespace
- end the grants a settings change takes away inside the write itself, once per change, instead of in a watcher that sees only the latest value: a blocklist entry added and removed again still ends its grants, and no read admitted after a switch-off can use a pre-off grant - leave the helper's runtime without waiting on a request parked on a permission prompt, so the helper exits when codeg does - send driver calls one at a time, as the driver serves them, so a write only stalls on a driver that stopped reading; a call that runs past its time retires the driver instead of queueing the next call behind it - judge a tree node's secrecy on its role and labels, never its value, and tell node lines from value lines by each platform's exact node shape - show a grant the frontend store has not heard of yet as shared in the picker
…rkers A macOS title holding Linux's value=" or a macOS document holding a Windows " id= must not move where a node's value is taken to start or end: pick the value and after-value markers per platform.
…size The driver ends a caller's implicit session after five idle minutes and refuses every call after that, while the helper's driver lives as long as computer use is on: five quiet minutes left every read failing until the helper restarted. Its environment now sets the idle timeout to ten years, and a driver that still answers session_ended is dropped so the next call starts a fresh one. The driver maps a click's pixel coordinates by the scale of the last capture of that window, whoever took it at whatever size, so a picker thumbnail would move every later click. The driver now runs with no ceiling on a capture (a config file in its private home, confirmed with get_config at launch), the helper shrinks images itself, and each capture reports its full-size dimensions and whether they are the window's own pixels. Helper protocol v2.
Five action tools join the reads on codeg-mcp: computer_click, computer_scroll, computer_type, computer_press_key and computer_set_value. Each needs the window shared for control, and goes to it in the background only; the driver's foreground routes do not hit-test where their input lands. Every ref and point is resolved against what the agent last read of the window: a ref only from the latest snapshot and only if its line was in the text the agent was given, a point only from the latest screenshot and mapped back to the window's own pixels by codeg. Keys are judged for a window grant (editing and navigation, and the chords that stay in a text field), pastes are refused, and characters are typed only into an element the agent names. Text never goes into a field the tree redaction judged secret. The helper checks again at delivery what only it can see: the pid is still the process the window was shared from, the session is not locked, the window is still the size a point was read at, and nothing has been stopped. It rebuilds each driver call from the closed action type and maps the driver's refusals to its own codes and words, and it refuses setting Safari pop-ups by value, which the driver does through AppleScript against Safari's front document. Driver calls now queue in codeg, so an action's checks run when its turn comes rather than before a long read ahead of it. The person's Stop pauses every agent call, ends every sharing and has the helper kill the driver mid-action; Resume is explicit.
The window picker and the status-bar panel offer two levels for a shared window, read or read-and-act, as the two entries of one menu, and move a shared window between them. While any window is shared for control a red stop sits beside the status-bar glyph: one click refuses every agent call, ends every sharing and cuts off what is in progress. The panel then says so and offers Resume, and sharing waits until then. The store carries the paused state with the shared list, from the state event and the status fetch alike. The activity list names the new actions, and the settings copy says the group can now act. All ten locales.
Stop now holds without waiting for anything: the helper marks itself halted the moment it reads the frame, the local backend keeps its own halted flag so an action cannot start a fresh helper after a Stop that found none running, a share cannot land between a Stop and its revocation, and a Stop and a Resume pressed close together leave codeg and the helper agreeing. Listings in flight when Stop came answer computer_paused, and an action cut off by it says it may have happened. The helper checks, just before each driver call, that nothing was stopped, the pid is still the shared process and the session is affirmatively unlocked; a platform that cannot say is not acted on, which leaves Linux read-only for now. Typing then return is held to that check for both calls and reports the weaker of their effects. Refs come from the driver's structured element list, placed by the tree: an index a value line forges cannot add an element, take a real one's place or clear a secret. Shift+Delete and Cmd+Backspace are no longer window chords, a Safari pop-up is not set by value when the application cannot be named, and the action tools refuse arguments they do not take. A codeg window that loads after something was shared or stopped asks for that state at once, and the picker offers nothing to share while stopped.
A Stop that set `paused` between Resume's "no Stop since?" and its "not stopped" was undone by the write, and then skipped halting the helper. Resume now decides under the lock Stop takes. The window listing now sweeps before its last "stopped?" check, so nothing it waits on comes after that check.
A global stop shortcut (Ctrl+Command+Escape on a Mac, Ctrl+Alt+Escape elsewhere, configurable or off) does what the Stop button does. It is held with the OS only while computer use is on, on a closed list of keys: a media key would take an event tap and Input Monitoring with it, which codeg must never hold. Settings say whether the OS actually took it. While any window is shared, a strip above every window names what agents may do and where, flashes what one just did, and carries Stop. After an action lands, a short mark shows where: the element's frame from its snapshot, or the point in the window the helper measured just before. Both are codeg windows of their own, never focused, the mark click- through, and kept out of the window-state plugin's restore.
- Tell the strip, the marker and the panels in the order the state was read: two changes told at once could leave the strip hidden while a window is shared. - Hide the marker window rather than close it, so a share straight after an unshare never takes a closing window for a ready one; drop a mark that comes back after nothing is shared for control. - Size the strip to its words, not to the window it is in, and keep the last action shown when a read follows it. - Ask for the stop shortcut's status only once its listener is in place; end recording when a save locks the row; say nothing of a shortcut the form has not read. - Clear a refused shortcut's status once it is no longer wanted, and mark a point only when the driver said where the window is.
- Settings > Computer Use: the switch, cua-driver (release in use, what the cache holds, install / upgrade to the pinned release / uninstall, download progress), the helper's macOS permissions and the sharing and stop settings, moved off the Collaboration page. - codeg itself holding Accessibility or Screen Recording no longer turns computer use off; it is a notice, and the permission texts are shorter. - The share picker shares or unshares every shareable window at once, and folds unshareable windows away with the reason, saying why codeg's own windows are never shared. - macOS: the helper tells which app owns a window from the process itself instead of the driver's app list, which froze at the driver's first call and could lend a reused pid a quit app's identity. - macOS: the helper asks TCC in a fresh probe process (the pinned driver's own probe, under the same launch requirement) instead of in itself, since a process keeps its first "not granted"; a permission granted since the driver started restarts the driver. - The helper sweeps the per-launch driver homes dead helpers left behind.
…-all - Shares are decided by the switch and blocklist as the last settings change left them, under the lock that change revokes under, so neither a single nor a bulk share can outlive a switch-off. - Uninstalling kills the driver before the files go, and the backend reads the switch itself before starting a helper, so a settings watcher a change behind cannot fetch the driver again. - Installing no longer prunes other releases: the cache is shared with any other codeg on the machine. - macOS: an app is identified by its bundle identifier or not at all, a helper app is the application it sits in, and identities are read afresh on every listing. - The helper's run-directory sweep follows no links, the permission probe rejects oversized output, and a grant seen while a driver was starting still restarts it. - The settings page and popover act only in the local desktop window, the switch keeps a newer broadcast over its own older answer, and the picker takes one change at a time.
…un sweep - Uninstalling waits for the helper's answer to the halt within the helper's own bounds on starting a driver, so a driver still starting is stopped before the files go rather than outliving its helper. - The helper no longer sweeps other helpers' run directories: each run directory is removed by the helper that made it, as before. - The permission recheck interval exists on macOS only.
… included - Switching computer use off, and uninstalling the driver, close the helper's input and wait for it to exit instead of signalling it: the helper treats that like a Stop, so a driver still starting stops once it has started, and only then does the helper exit. - Closes are serialized, so an uninstall that finds the helper already being closed by the switch-off waits for that close to finish before the driver files go.
The helper checks for a Stop, or codeg leaving, once more right before it spawns a driver — after hashing the cached file, which takes as long as the disk makes it — so what is left of a start once a driver exists is bounded by the driver's handshake and configuration, well inside the time the helper waits for its driver when codeg goes.
Collaboration now follows General, and Quick Messages sits above the two Use pages, which stay side by side.
An application is called by its bundle's file name unless it names itself in the person's language, so Visual Studio Code is no longer "Code" and the Finder stays 访达. A bundle that says it is an application (CFBundlePackageType APPL) is one whatever its name, so Chromium browsers running from their code-sign clone (Foo.app.bundle) are identified again; the blocklist also knows such a clone by the file name it was cloned from.
build.rs fingerprints the helper's sources into both binaries and the helper reports its fingerprint when it introduces itself. A debug codeg refuses a helper whose fingerprint differs and says to restart pnpm tauri dev, which only rebuilds codeg after an edit.
Without it there are no window titles or pictures; the picker now says so and offers to grant it. Back from System Settings with the permission granted, it lists the windows and fetches their pictures again, and Refresh fetches the pictures again too.
A bundle that is an application only by its own CFBundlePackageType is one only standing by itself: inside another application it would pass for that one and lose its own place on a blocklist.
pnpm tauri dev does not build sidecars under CODEG_SKIP_SIDECAR=1, so restarting it alone never replaces the helper; the refusal now says to run pnpm tauri:prepare-sidecars first, and the skipped step says why it matters for computer use.
…e list Granting: the helper now asks macOS from a fresh process each time, as it already checks - macOS takes one request per process, so a second ask from the long-lived helper (after a stale entry was removed) reached nobody and the helper was never listed in System Settings. One Grant button asks and, while the permission is still missing, opens its pane; Show in Finder helps drag the helper in by hand. Never-share list: the defaults are listed in Settings and can be taken off, except System Settings and the system's password prompts, which guard computer use itself. Entries can be added and the defaults restored. Only the removals are stored, so defaults added later apply. Development-build notes are shorter: one status tag, one line under the permissions, and no mention in the codeg-window explanation.
…e list A default is found by the name the list shows as well as by its identifiers, one permission request runs at a time, and the rebuild hint no longer repeats the development-build tag.
On macOS a helper the system has not seen before can wait at launch on a question of its own - reading from a removable volume, when codeg lives on another disk - so the timeout names that and asks to answer it and try again.
… fold path case only on macOS and Windows
…dows does The pinned driver's Windows window list names each owner by its executable's file name alone, with no path and no bundle identifier, so every window was unidentified and none could be shared. The helper now reads the owner's full executable path off the process itself, through the same handle as its start time, and that executable is the application; the application list is made from the windows' owners, as on macOS. Settings' own window is on the default never-share list by its executable. A host's windows, drawn for other applications, stay unidentified: ApplicationFrameHost.exe frames every packaged application's window (the application's own window, listed beside the frame, is known by its executable), and msedgewebview2.exe draws the inspector and dialogs of every WebView2 application, codeg among them. So do the system's own agents under SystemApps: the Start menu, the lock screen, the PIN and password prompts. Any executable named codeg.exe (or codeg) is codeg, so one codeg cannot share another's windows, a development build beside the installed one included. An application is named as Windows names it to the person: a packaged one (Terminal, Settings, Notepad) by its Start menu entry, found through its application user model id, in the person's language; any other by the localized description in its executable's version resource, as Task Manager shows it; by its file name only where neither says. Names are kept per executable, and a listing's owners are read off the runtime's threads, since the Start menu can take a fifth of a second to answer. Identity and the never-share list still go by path.
…nd window When the driver will not deliver a key or typing in the background, the refusal now says that no retry gets it through, by ref or not, and points at what still reaches the application: computer_set_value for a field, a click by ref on the button the key would have pressed, or the user. For a window the driver names as Chromium's, it adds that no Chromium-based application on Windows takes keys in the background (Edge, Chrome, VS Code and other Electron apps) and that codeg's own browser does for a web page. Refused pointer input keeps its wording.
…on it Some applications take no keys or typing while they are in the background — on Windows every one built on Chromium, such as Edge, Chrome and VS Code — and the driver refuses such input outright, so an agent could not, say, press return in Edge's address bar. The click, scroll, type and press-key tools now take an optional `delivery`: "background" leaves the window where it is; "foreground" has the driver bring it to the front for that one action, send real input and switch back to the window the user was in. Left out, the user's default applies. Setting a value and restoring a window always stay in the background. Computer use settings gain "Let agents bring windows to the front", on by default, and "Default input mode", Background by default, which can be Foreground only while the front is allowed and keeps its choice while it is not. Delivery is resolved against the settings as they are at each press, so switching the front off stops the next one, and asking for it while it is off is refused with computer_foreground_not_allowed. What agents are told follows the settings: a computer_background_unavailable note ends with the next step they leave — retry with `delivery: "foreground"`, or ask the user to switch it back on — computer_list_windows reports the input policy, and each result says whether the input went in the background or with the window brought to the front. An elevated application or an ARM console, which would refuse the front as well, is reported as a failure rather than a background refusal, and a front that could not be had gets words of its own. The tool descriptions say so, and so does the Agent tools copy in all ten locales. Helper protocol v6.
…n windows On Windows a packaged application's window, such as Settings or Calculator, is a frame owned by ApplicationFrameHost.exe with the application drawing inside it from its own process, so each was listed as unidentified and could not be shared. The helper now reads, by the frame's handle, which process draws inside it: the owner of the core window set into the frame, or, while the frame is minimized and that window stands outside it, the one process running the application the frame names by its application user model id. The frame is then that application's, named as Windows names it and matched against the never-share list by its executable, so Settings stays on it by default. No such process, or more than one, leaves the frame unidentified, as WebView2's windows and the system's own agents are. The run of the process drawing inside a frame is part of the window's identity, checked again before every read and action: a grant on the frame holds only while that same run is inside, so an application relaunched into the same frame is a window nobody has shared. computer_list_apps lists each application the frame host shows on its own. A window the compositor cloaks is off the screen. One on another virtual desktop is still listed, like a window on another Space on macOS; one on the current desktop is something nobody can see or use, such as the input experience panel or a minimized frame's content, and is no longer listed. A window whose desktop the system cannot tell stays as the driver listed it. Helper protocol v7.
… clippy on Windows
- An action on a packaged application's window is held, at the moment it goes out, to the run of the process drawing inside the frame as well as to the frame's owner (helper protocol v8). A frame's run is read through the handle it was found by, so a pid passed to another process in between cannot lend the frame that process's identity. - Keys and typing sent with the window brought to the front wait while the user holds a modifier (Ctrl, Alt, Shift or the Windows key; Command, Option, Control or Shift on macOS), which they would combine with; the agent is told which. The return after typing says why when it is held back. - A front the driver reports losing after the input went out no longer reads as "nothing was sent, try again": the action may have happened, and the agent is told to read the window before repeating it. - Only the system's own SystemApps folder hides its agents. The applications listed are the owners of windows a person could mean, from the same listing as the windows, so a minimized packaged application is named once. Application names are read with a two-second wait and at most four readers at once, so a shell that does not answer cannot hold a listing up.
The stop shortcut registers through tauri-plugin-global-shortcut, whose global-hotkey links CGEventTapCreate to watch media keys. codeg never asks it to: StopShortcut accepts no media key, and a test now holds that no webview is given the plugin's commands, reading the capabilities and the app's permission sets the way tauri-build picks and decodes them. The macOS release gate lets codeg alone import that one symbol; codeg-mcp and every other banned API still fail it.
macOS looks permissions up under the app bundle an executable sits in, so the helper in codeg.app/Contents/MacOS was checked as codeg: granting it did nothing, and granting codeg - and with it every agent's shell - is what made computer use work. The helper now ships as Contents/Helpers/codeg-computer-helper.app, with bundle and signing identifier app.codeg.computer-helper and no Dock icon. A bundled codeg launches it from there and nowhere else, and the Finder button shows that app. No bare copy is left in the bundle. On macOS the helper is no longer a sidecar, and its binary target needs the new computer-helper feature, which `tauri build` never enables: the CLI bundled its own default-feature build of the helper, which on macOS landed over the staged sidecar. prepare-sidecars builds the helper with the feature and, for macOS, assembles the app; release.yml signs the app before the bundle is built, and the release gate fails on a helper in Contents/MacOS. Development builds still find the staged helper beside codeg.
Rust 1.99's clippy rejects code that was clean under 1.98. async-trait 0.1.89 marks every generated method #[must_use] on top of a future that already is (double_must_use); AtomicUsize::fetch_update is deprecated in favour of try_update, stable since 1.95; and on Windows, which has no short scratch root, sweep_roots loops over a single element. async-trait 0.1.92 no longer adds the attribute, the two calls use try_update, and the loop keeps its one element on Windows.
macOS charges Screen Recording to the outermost app around an executable that the same team signed. The helper app nested in codeg.app therefore still asked for, and captured on, codeg's Screen Recording grant, which every agent's shell shares. Accessibility was already the helper's own. codeg now keeps a byte-identical copy of the shipped helper app in the helper's data directory, brought up to date before every launch and swapped in whole, and runs the serving helper, the one-shot permission request and the Finder reveal from that copy. The copy carries the same signature, so the launch requirement, the peer check and the helper's existing grants all still apply.
…he app The Tauri CLI bundles every binary target whose required features are all among the ones it builds with — its own --features, tauri/custom-protocol and the config's build.features, never the manifest's defaults — and then adds the default-run one. codeg-server and codeg-mcp required none, so the desktop app shipped the standalone server it never runs (about 70 MB) and a codeg-mcp compiled with the desktop features, WebKit and all, in place of the lean sidecar prepare-sidecars stages for it. Each now requires a feature of its own, server-bin and mcp-bin, off by default and on under test-utils; every build site passes it. The macOS release gate fails on a codeg-server in the bundle or a codeg-mcp that links WebKit, and a test holds that no binary target but codeg is one tauri build would bundle.
A running codeg-computer-helper.exe holds its file like a stray codeg-mcp.exe does, and it is not certain to have exited by the time the updater starts writing. The installer hooks now stop it too, with the cua-driver it runs.
macOS: a window whose application is hidden with Command-H is now listed, marked hidden, and can be shared; Accessibility tells it apart from the windows applications keep out of sight. computer_restore shows such an application again, then takes the window out of the Dock if it is minimized, without bringing it to the front — once the driver has confirmed the window is still there. Windows: computer_restore takes a minimized window off the taskbar without making it the active window. Linux: the X11 window manager says which windows are minimized and which on another desktop, which the driver's listing does not. computer_restore brings a minimized window back the only way there is, by bringing it to the front — so only where the person allows that. Actions are no longer refused outright: the helper acts only when logind says the session is active and unlocked and the desktop's own screen saver says it is not on, and refuses whenever that cannot be established. Keys and typing at the front wait for held modifiers, read from the X server, and are refused where they cannot be read (Wayland). Helper protocol v9.
Agents can now drag in a shared window (computer_drag: two points from the latest screenshot, a button, how long the path takes), hold a key down for a while (computer_hold_key), and hold Shift, Control or Command over a click. - A held key is the key pressed at once, then after half a second about every 50 ms until its time is up, as a held key repeats; no driver can press and hold one. Its time runs in real time from the first press, and a press that waits its turn past the end is not sent. - Every press of a held or repeated key takes its own turn at the driver and is checked again, held to the Stop count and the sharing its first press went out under: Stop, or the window taken back and shared again, ends the presses. - Keys over the pointer stop where the window does: Option is refused on macOS, since an Option-click reaches every window of the application, and the Windows and Super keys elsewhere. Only macOS's driver holds keys over a drag; on Windows and Linux a drag with keys is refused rather than sent without them. - A drag at the front on macOS goes only to its application's front window. The driver brings the application forward, not the window, so a drag on a window behind another of the same application would land on the other. Helper protocol v10.
A person can now share an application as a whole from the picker, where the windows are grouped by application: every window of it, the ones it opens later too, at one level and on one idle clock. Its windows change with it, and ending it, a Stop, the blocklist or the application quitting ends them all. The status popover and the strip list it once, open windows or not. - An application shared as a whole takes its own shortcuts (closing, quitting, its menu commands' keys), Option over the pointer on macOS, and its menus through the new computer_invoke_menu (macOS and Linux; the application is brought to the front for it, where the user allows that). The desktop's shortcuts stay out of reach: switching applications, the launcher, screenshots, locking the screen, logging out. - The Apple menu and the application menu stay out of reach too, by title: they restart, log out, run Services and hide the others. - A window shared on its own no longer reaches its application's menu bar on macOS. The driver's window tree carries it, so the helper now leaves out every menu bar item with its open menu, and refuses their elements. - A paste stays refused by any route that can be told apart: a menu command named for pasting or whose shortcut is Command-V, and a menu item or button named for pasting. Helper protocol v11.
With "Let agents open applications and move windows" switched on in the Computer use settings (off by default), agents get two more tools: - computer_launch_app starts an installed application in the background, named by its key or by its name as the system lists it. The helper starts it only by what the driver listed it under, never by anything the agent wrote, and sends the start once. codeg and the applications that are never shared are refused, judged by who the application is and by every word of the command that starts it. Starting it shares none of its windows. - computer_set_window_frame moves and sizes a window shared for control, in desktop units. What is left out stays as the window is just before, read then rather than from an older listing. Helper protocol v12.
…own switch A paste used to be refused outright: the clipboard is the person's, and holds what they copied from a password manager as readily as anything else. Now an agent may paste what it put there itself, and only that. - The helper stamps the clipboard around an agent's copy or cut (the key, or a menu command named for it). macOS counts the pasteboard's changes, Windows numbers them, and on Linux the stamp is a digest of plain text only. A copy that changed the clipboard is the agent's, for as long as the window it came from stays shared under the same sharing. - A paste of any kind (the key, a menu command named for it or whose shortcut is Command-V, a button or menu item named for it) goes only while the clipboard still holds that, checked as late as the helper can. Content an application marked concealed is never the agent's. Stop forgets it. - With "Let agents use the clipboard" on (off by default), computer_clipboard_read reads back only what the agent put there, and computer_clipboard_write puts text there. A write is the agent's only once read back as that plain text, alone, with nothing copied in between. Helper protocol v13.
Where the person turns it on (Settings → Computer use; macOS and Windows), the share picker offers the entire screen. Every window the rules allow is shared with it, the ones that come up later too, together with the desktop's own shortcuts; agents read the screen as one picture (target `d1`) and click, drag and scroll at points of it. Sharing it takes over whatever was shared before, and nothing else changes on its own while it lasts. The helper judges every on-screen window by its owner when it captures or acts: codeg's own windows, the never-share list, owners it cannot identify, and the system's views of other windows (Mission Control and the Dock off its level, Stage Manager, notifications, Task View, the switcher, taskbar previews) are painted over, with their edges, and a point on them is refused. A picture is handed over only when those windows stood still across it. The keys that lock the screen, log out or show every window at once, the screen's corners on macOS, and Mission Control itself stay out of reach. An action goes only at the front, where the person allows it, on a screen still the size and scale its picture was taken at. The driver is had running first, and the sharing, the switches and the never-share list are asked about once more just before the action is sent.
…ws it codeg-server shares the screen of the machine it runs on only when it is started with CODEG_COMPUTER_USE=1 (or true/yes/on): the person running it says so, not a web client. Its web clients — each holding the server's token — then share windows, the whole application or the entire screen, and press Stop, from the same panel the desktop app has; agents are offered the computer tools only where such a service runs. The service no longer needs a desktop window: what changes is told to the desktop app's own webviews as before (never to its web service), or to the server's web clients; the strip, the action marker and the stop shortcut stay the desktop app's. The panel's commands are shared cores behind the desktop commands and new HTTP routes, which refuse everywhere no service runs — the desktop app's own web service included — and say so through `computer_available`, which a web window asks (again after a failed answer, and on every reconnect) before it shows any of it. The server releases ship codeg-computer-helper beside codeg-server; the installers put it in place (renamed over one still exiting), and the server's self-update swaps and rolls it back with the rest.
The desktop app installs itself for one user in %LOCALAPPDATA%\codeg, with codeg-mcp.exe, codeg-computer-helper.exe and its frontend in web\. install.ps1 put the server in the same folder, so each install replaced the other's files, and each installer stopped the other's helper and companion processes by name. install.ps1 now installs to %LOCALAPPDATA%\codeg-server. A server an earlier version left in %LOCALAPPDATA%\codeg stays there unless the desktop app is there too; then it moves, and only its codeg-server.exe and the PATH entry leave the old folder. A folder holding codeg.exe is refused, the PATH cleanup leaves the desktop app's files alone, and a process is stopped only when it runs a file this install replaces or removes, found through CIM so that a 32-bit PowerShell sees 64-bit processes too. A relative -InstallDir resolves from PowerShell's location. The desktop installer's hook stops only the sidecars running from its own folder, and stops them by name as before when PowerShell cannot. codeg-server warns at startup when it still shares a folder with the desktop app.
Pin cua-driver 0.32.0: new archive and executable digests for every platform, taken once each archive's Sigstore bundle checked out against trycua's release workflow, and the two new macOS cdhashes. The signer, designated requirement and entitlements are unchanged. 0.32.0 refuses an argument its tool does not name, addresses elements only by token, and keeps one snapshot per window, whose capture aims every point: - An element goes as its element_token. macOS's set_value gets no delivery mode, and a double-click in a window no longer takes modifier keys (macOS and Linux refuse them; Windows dropped them). - On macOS and Windows a screenshot is read through verify_state, which leaves the window's snapshot alone, so refs survive it as they did. Its bounds are read before the capture and after it, and a capture the window changed size around aims no point. - On Linux a capture stays a snapshot of its own, at full size, and one the driver took from the screen is refused. - A snapshot captures the window too and drops the image, so points still work after one. A point the driver holds no capture for gets one, the window is measured again, and it goes once more: nothing went out the first time. - A snapshot walks the tree for up to 20 s, as macOS did before. - The new refusal codes are read, keeping "may have happened" apart from "nothing was sent". Foreground delivery no longer says it switches back on Linux, where the driver now leaves the window in front, and says a click may move the pointer on any platform.
xintaofei
marked this pull request as ready for review
October 2, 2026 12:28
xintaofei
added a commit
that referenced
this pull request
Oct 2, 2026
This one lets agents see and operate your desktop. With the new Computer Use preview, you share a window, a whole app or the entire screen, and agents can read it and click, type and drag in it, with Stop sharing always one click away. Codex moves up to its 2.1.1 adapter and Claude Code to 0.85.1, and seven more bundled agents get new versions. Alongside that: the desktop app reopens the workspace windows you had open when you quit, the on-screen keyboard stays down when you switch conversations on a touch device, icons draw a little lighter, a to-do runs the agent its picker shows, and on Windows codeg-server installs into a folder of its own. ## New - **Agents can see and operate the windows you share with them (preview)** — switch it on under Settings → Computer Use, share a window from the status bar's Computer use panel as "Read only" or "Read and act", and agents can take screenshots, read what the window shows, and click, scroll, type and press keys in it on macOS, Windows and Linux, while the panel lists what they just did; nothing is readable until you share it. (#870) - **One click or one shortcut stops everything** — "Stop sharing" in the Computer use panel or on the floating stop bar, or Ctrl+⌘+Esc (Ctrl+Alt+Esc on Windows and Linux), ends every sharing and cuts off whatever agents are doing, and a shared window no agent has used for 30 minutes (adjustable) stops being shared by itself. (#870) - **codeg's own windows and your never-share list stay off limits** — password managers and system settings are on the list by default, and you can add or remove apps. (#870) - **You can share a whole app or the entire screen** — an app shared as a whole includes the windows it opens later, its menus and its own shortcuts, and "Offer the entire screen" (macOS and Windows, off by default) lets agents see one picture of the screen and click anywhere on it, with codeg's windows and never-share apps painted over. (#870) - **Agents can do more than click** — drag, hold keys, restore minimized or hidden windows and, for apps that take no keys in the background (such as Edge, Chrome and VS Code on Windows), bring the window to the front, which you can switch off; opening apps, moving windows and using the clipboard have their own switches, off by default, and an agent can only paste what it put on the clipboard itself. (#870) - **codeg-server can offer computer use too** — start it with `CODEG_COMPUTER_USE=1` and its web clients can share the server machine's windows and stop them from the same panel. (#870) - **The desktop app reopens the workspace windows you had open when you quit** — the local one and each remote workspace, with the one you were using back in front, and a remote workspace that can't be reached is reported in the local window. ## Improved - **Updated bundled agents:** Claude Code ACP 0.85.1 (Claude Code 2.1.286), Codex ACP 2.1.1 (Codex CLI 0.159.3), OpenCode 1.18.34, Cline 3.0.67, CodeBuddy 2.161.0, Grok 1.0.46, Qoder 1.1.65, OpenClaw 2026.9.7, Cursor 2026.09.28. - **Forking a conversation no longer leaves the original session held by the agent** — codeg closes it after the fork, so Codex lets go of it within about a minute and Claude Code stops keeping a process running for every fork, and the "open elsewhere" banner now says to reload in about a minute if you just forked. - **Icons draw a little lighter** — those at the icon set's default weight of 2 now use 1.75, which sits better beside small text, while icons given a deliberate weight keep it. (#866, @SousekiL) - **Settings → "Browser" is now "Browser Use"**, next to the new "Computer Use" page, and "Collaboration" moves up to follow "General". (#870) - **On Windows, codeg-server installs into its own folder**, `%LOCALAPPDATA%\codeg-server`, so neither install overwrites the other's files or stops the other's processes, and re-running the install script moves a server that shares the desktop app's folder out of it. (#870) - **The desktop app drops about 70 MB it never used** — the standalone server binary is no longer bundled with it. (#870) - **The built-in model catalog is refreshed** — 8,339 models from 225 providers, up from 8,281 models. ## Fixed - **On a touch device, switching to another conversation no longer pops up the on-screen keyboard over it** — tapping the message box still brings it up, and desktop behavior is unchanged. (#865, @Flyneen) - **A to-do runs the agent its picker shows** — on a fresh install, or when the usual agent is disabled, the picker could show an agent that wasn't actually saved, and the to-do then failed to start with "no agent configured"; now the agent shown is saved with the to-do, and the hint under the picker says whether it is inherited or saved. (#864 reported by @qq974969638) - **Quitting with ⌘Q, from the Dock or by logging out now runs the full quit cleanup** — agent processes, terminals and the web service are stopped as they are when you quit from inside codeg, and logging off on Windows does the same. - **Cline runs on macOS 27** — macOS 27 killed Cline 3.0.65 and earlier at launch, while 3.0.67 ships with a valid signature, so upgrade it in Settings → Agents; OpenCode 1.18.34 is now properly signed too. - **A background command in a Claude Code conversation no longer has its card overwritten with internal marker text when it finishes**, and one that moved to the background because you sent a message or it timed out shows the Background badge. - **Codex conversations reopen more faithfully** — a question you answered with your own text under "Other" comes back as that text rather than as two picks, a failed compaction shows the service's error message instead of raw JSON, and a conversation started in the Codex desktop app shows its attachments as links followed by your request, titled after what you wrote. - **A search pipeline that finds nothing no longer shows as a failed command** — a command like `rg --files src | rg foo` that ends in `rg` or `grep` and matches nothing now reads as completed with its exit code still shown, instead of a red failure counted as an error in its tool group, while real errors such as exit code 2 or an error message still show as failed. (#877 reported by @Ryn-Mic) - **Grok's model picker picks up the full model list** — in a session opened before Grok had fetched its model catalog, such as after your sign-in expired, the picker kept the short built-in list for the rest of the connection; it now updates as soon as Grok announces the catalog, and the model and reasoning effort you chose stay selected. (#876 reported by @goon-13) ## Note - **Computer use downloads the open-source cua-driver (0.32.0) the first time it is needed**, or when you press Install in Settings → Computer Use. - **On macOS, grant Accessibility and Screen Recording to `codeg-computer-helper`, not to codeg** — only codeg can use that helper, so agents' own shells can't borrow the permissions; Windows and Linux have no such separation, so there the settings decide what agents get through codeg. - **On Windows, codeg-server's web files now live in `%LOCALAPPDATA%\codeg-server\web`** — if your server moves there, update a `CODEG_STATIC_DIR` that still points at `%LOCALAPPDATA%\codeg\web`. - **With Claude Code 2.1.286, sending a message moves a running command, MCP call or sub-agent to the background** instead of waiting for it or interrupting it — codeg marks such a command with the Background badge, and a question card still open is dropped when you send, as before. - **Behind a custom API address, Claude Code's model list no longer has separate "(1M context)" entries** — since 2.1.285 those models use their 1M window by default, and a conversation saved on such an entry carries on with the plain model; if your gateway stops at 200K, Claude Code's advice is `/autocompact 200k`. - **Cline 3.0.66 and later count output tokens without reasoning tokens**, so reasoning models show lower output token counts. Thanks to @Flyneen and @SousekiL for contributing to this release, and to @qq974969638, @Ryn-Mic and @goon-13 for the reports. ----------------------------- # 发布版本 0.33.0 这一版让智能体能看见并操作你的桌面：在新的「电脑操作」（预览）里，你可以共享一个窗口、一个应用或整个屏幕，智能体就能读取画面，并在其中点击、输入、拖拽，「停止共享」则随时一键就能按下。Codex 升级到 2.1.1 适配器，Claude Code 升级到 0.85.1，另有七个内置智能体更新了版本。 同期还有：桌面版会重新打开你退出时开着的工作区窗口，触屏设备切换会话时不再弹出软键盘，图标线条细了一点，待办任务会运行选择器里显示的智能体，Windows 上的 codeg-server 也装进了自己的文件夹。 ## 新增 - **智能体可以查看并操作你共享给它的窗口（预览）**——在「设置 → 电脑操作」里启用后，从状态栏的「电脑操作」面板把窗口共享为「仅读取」或「读取并操作」，智能体就能在 macOS、Windows 和 Linux 上截图、读取窗口内容，并在窗口里点击、滚动、输入和按键，面板里还会列出它们刚做了什么；没有共享的窗口一律读不到。（#870） - **一键或一个快捷键就能停下全部**——「电脑操作」面板或悬浮停止条上的「停止共享」，以及快捷键 Ctrl+⌘+Esc（Windows、Linux 上是 Ctrl+Alt+Esc），都会立即结束所有共享并中断智能体正在做的事；共享的窗口 30 分钟没有智能体用过（时长可调）也会自动停止共享。（#870） - **codeg 自己的窗口和「永不共享」名单里的应用始终无法共享**——名单默认包含密码管理器、系统设置等，可以自行增删。（#870） - **可以共享整个应用或整个屏幕**——共享整个应用时，它之后打开的窗口、菜单和自己的快捷键都包含在内；「允许共享整个屏幕」（仅 macOS 和 Windows，默认关闭）开启后，智能体能看到整屏画面并在任意位置点击，codeg 自己的窗口和永不共享的应用会被遮盖。（#870） - **智能体能做的不止点击**——还能拖拽、按住按键、恢复被最小化或隐藏的窗口，遇到在后台收不到按键的应用（比如 Windows 上的 Edge、Chrome、VS Code），还能把窗口切到前台（可关闭）；打开应用、移动窗口和使用剪贴板各有单独的开关，默认关闭，智能体也只能粘贴它自己放进剪贴板的内容。（#870） - **codeg-server 也可以提供电脑操作**——启动时设置 `CODEG_COMPUTER_USE=1`，网页端就能在同一个面板里共享服务器所在机器的窗口，也能随时停止共享。（#870） - **桌面版会重新打开你退出时开着的工作区窗口**——本地的和每个远程工作区都一样，退出时正在用的那个回到最前面；连不上的远程工作区会在本地窗口里提示。 ## 改进 - **内置智能体版本更新：** Claude Code ACP 0.85.1（Claude Code 2.1.286）、Codex ACP 2.1.1（Codex CLI 0.159.3）、OpenCode 1.18.34、Cline 3.0.67、CodeBuddy 2.161.0、Grok 1.0.46、Qoder 1.1.65、OpenClaw 2026.9.7、Cursor 2026.09.28。 - **分叉会话后，原会话不再被智能体一直占着**——分叉之后 codeg 会关掉原会话，Codex 约一分钟内就会放开它，Claude Code 也不再为每次分叉多留一个进程；「会话在别处打开」的提示现在也会说明：刚分叉过的话，约一分钟后重新加载即可。 - **图标线条略微变细**——原本使用图标库默认粗细 2 的图标改为 1.75，与小号文字搭配更协调，特意指定过粗细的图标保持不变。（#866，@SousekiL） - **「设置」里的「浏览器」更名为「浏览器操作」**，与新增的「电脑操作」并排，「协作」也挪到了「常规」后面。（#870） - **Windows 上的 codeg-server 装进自己的文件夹** `%LOCALAPPDATA%\codeg-server`，两边安装时互不覆盖文件，也不再互相结束对方的进程；重新运行安装脚本，会把与桌面版共用目录的服务器挪出来。（#870） - **桌面版去掉了约 70 MB 用不上的内容**——不再捆绑独立服务器程序。（#870） - **内置模型目录已更新**——共 225 个供应商、8339 个模型，此前为 8281 个。 ## 修复 - **触屏设备上切换到另一个会话时，不再弹出软键盘遮住对话**——点一下输入框仍会弹出，桌面端行为不变。（#865，@Flyneen） - **待办任务会运行选择器里显示的智能体**——全新安装，或常用智能体被禁用时，选择器里显示的智能体其实没有被保存，任务启动时报「no agent configured」；现在显示的智能体会随任务保存，选择器下方的提示也会说明它是继承来的还是随任务保存的。（#864 由 @qq974969638 反馈） - **用 ⌘Q、Dock 菜单退出或注销登录时，也会执行完整的退出清理**——结束智能体进程、终端和 Web 服务，与在 codeg 里点退出一样；Windows 上注销时同样如此。 - **Cline 能在 macOS 27 上运行了**——macOS 27 会在启动时杀掉 3.0.65 及更早的 Cline，而 3.0.67 带有有效签名，请在「设置 → 智能体」里升级；OpenCode 1.18.34 也补上了正式签名。 - **Claude Code 会话里的后台命令结束时，卡片不再被改写成内部标记文字**；因为你发了消息或超时而转入后台的命令，会标上「后台运行」徽标。 - **Codex 会话重新打开后更贴近原样**——在「其他」里输入自己的文字作答的提问，回来时还是那段文字，不再变成两个选项；压缩失败时显示服务返回的错误信息，而不是原始 JSON；在 Codex 桌面版里发起、带附件的会话，附件显示为链接，后面跟着你的请求，标题也取自你写的内容。 - **以搜索收尾的管道命令没有匹配项时，不再显示成失败**——像 `rg --files src | rg foo` 这样以 `rg` 或 `grep` 结尾的命令没有匹配时，现在显示为已完成，退出码仍保留在卡片上，不再是红色失败，也不会被计入工具组的错误数；退出码 2、错误信息这类真正的错误仍显示为失败。（#877 由 @Ryn-Mic 反馈） - **Grok 的模型选择器会补全完整的模型列表**——在 Grok 拉取到模型目录之前打开的会话（比如登录已过期的情况），选择器原本在整个连接期间都只有内置的几个模型；现在 Grok 一发出模型目录更新，选择器就会随之刷新，你选好的模型和推理强度保持不变。（#876 由 @goon-13 反馈） ## 注意 - **电脑操作首次使用时会下载开源的 cua-driver（0.32.0）**，也可以在「设置 → 电脑操作」里点「安装」提前装好。 - **在 macOS 上，请把「辅助功能」和「屏幕录制」授予 `codeg-computer-helper`，而不是 codeg**——只有 codeg 能使用这个 helper，智能体自己运行的命令借不到这些权限；Windows 和 Linux 没有这层隔离，那里由设置决定智能体能通过 codeg 拿到什么。 - **Windows 上 codeg-server 的网页文件现在位于 `%LOCALAPPDATA%\codeg-server\web`**——如果你的服务器被挪到了那里，而 `CODEG_STATIC_DIR` 还指向 `%LOCALAPPDATA%\codeg\web`，请改成新路径。 - **Claude Code 2.1.286 起，发消息会把正在运行的命令、MCP 调用或子智能体挪到后台**，不再等它结束或直接中断；codeg 会给这类命令标上「后台运行」徽标，仍开着的提问卡片会像以前一样随发送被收起。 - **通过自定义 API 地址使用 Claude Code 时，模型列表里不再有单独的「(1M context)」条目**——2.1.285 起这些模型默认就用 1M 窗口，原本选了这类条目的会话会继续使用对应的普通模型；如果你的网关只支持 200K，Claude Code 的建议是用 `/autocompact 200k`。 - **Cline 3.0.66 及以上版本统计输出 Token 时不再包含推理 Token**，所以推理模型显示的输出量会偏低。 感谢 @Flyneen、@SousekiL 为本次发布做出的贡献，也感谢 @qq974969638、@Ryn-Mic 和 @goon-13 的反馈。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Computer use, as a preview that is off by default. Agents in any conversation can read the windows the person shares with them, and act on the ones the person allows: one window, a whole application, or (where the person switches it on) the entire screen.
Draft: opened to run CI on all three platforms. It is not ready to merge; see Before release below.
What agents get
computer_list_apps,computer_list_windows,computer_screenshot,computer_snapshot(the accessibility tree, with element refs) andcomputer_verify.computer_click,computer_drag,computer_scroll,computer_type,computer_press_key,computer_hold_key,computer_set_valueandcomputer_restore(puts a minimized or hidden window back on the screen, on every platform).computer_invoke_menu, macOS and Linux) and its own shortcuts. The desktop's shortcuts stay out of reach.computer_launch_app,computer_set_window_frame); the clipboard (computer_clipboard_read,computer_clipboard_write). A paste goes only while the clipboard holds what an agent copied from a window it may read, or wrote itself.d1(macOS and Windows, on its own switch): one picture of it and clicks, drags and scrolls at points of it, sent at the front. codeg's windows, the never-share list, windows whose application cannot be told and the system's views of other windows (Mission Control, Stage Manager, notifications, Task View, previews) are painted over, and a point on them is refused; so are the keys that lock the screen, log out or show every window at once, and the screen's corners on macOS.computer_control_required,computer_stale_ref,computer_occluded,computer_blockedorcomputer_stopped.What the person controls
How it is built
codeg-computer-helper, is its own TCC principal.AXIsProcessTrusted: no other Accessibility, event-posting or screen-capture call. The release workflow checks this withnm -uon the signed bundle. The same step checks each executable's hardened runtime, that none carries entitlements, and the signing requirement each side compiles in for the other.codeg-serveroffers computer use only when the person running it starts it withCODEG_COMPUTER_USE=1, for the screen of the machine it runs on: its web clients (each holding the server's token) share windows and press Stop from the same panel. There is no floating strip, action mark or stop shortcut there. The desktop app's own web service never offers it. The server releases shipcodeg-computer-helper, and the installers and the server's self-update keep it in step. On Windows the server installs into its own folder,%LOCALAPPDATA%\codeg-server, apart from the desktop app's:install.ps1moves a server out of the desktop app's folder, and each installer stops only the processes running from its own files.Verified
Every step was reviewed with Codex until approved. At the head of this branch:
cargo test --features test-utils: 5022 passed.cargo test --no-default-features --features server-bin --bin codeg-server --lib: 4709 passed.-D warningsfor desktop, server,codeg-mcpandcodeg-computer-helper, on Rust 1.99 and 1.98.1 for macOS, and on 1.99 for Windows (x86_64-pc-windows-gnu) and Linux (x86_64).pnpm test(549 files, 8147 tests),pnpm build, eslint and tsc.install.ps1: parsed by PowerShell 7.6, and its folder choice, PATH matching and process matching run against fake folders and processes (15 cases).install.shrun end to end in a sandbox: a fresh 0.32.4, an upgrade to 0.33.0-rc.1 with the helper, a repair.v0.33.0-rc.1andv0.33.0-rc.2, the second built from 598289a (each a commit on no branch that only sets the version and lets-rctags past the default-branch check): all six desktop and five server targets built, the Windows installers with the new NSIS hook; both macOS apps notarized and passed the signing and symbol gate; every server package carriescodeg-computer-helper. Both are published as prereleases, soreleases/latestand both updaters still see 0.32.4, and Docker got only the0.33.0-rc.Ntags.Before release
None of this can be done from a development build:
codeg-serverwithCODEG_COMPUTER_USE=1in a real desktop session, from a browser on another device.install.ps1on a Windows machine that has the desktop app: a server moved out of the desktop app's folder, and neither installer stopping the other's processes.🤖 Generated with Claude Code