Repository navigation
feat(browser): leave notes on page elements for the agent - #952
Merged
Merged
Conversation
A crosshair on the session browser's toolbar stops the pointer reaching the page and outlines the element under it, named in a tag. A click opens a note beside it, in the review's comment box; each note keeps a numbered tab on a dashed ring around its element, following it as the page scrolls. Notes wait in a tray floating over the page's bottom, so the page never resizes, and Send types them into the session as one message: label, page, a selector browser_click takes, the React component in a development build, the markup without field values, and the note. A send that fails keeps the notes. The page only answers what is where (browser.ElementAt, browser.Locate); the outline, tabs, popover and tray are drawn by the window, so nothing is injected into the page's DOM.
- A production build's one or two letter component names (the minifier's) no longer reach the tag or the agent; a development build's source path is cut at the project's src/ even under a folder named app/. - A refused send says why in the tray, not in a toast over the toolbar. - Ctrl+Shift+E and Ctrl+Enter are Cmd on macOS. - The wheel over the popover or the tray no longer scrolls the page under it. - The ring's number tab sits on its straight top edge, past the corner. - docs/ceilings.md: the pane copies the page's selection since #941, and a Chromium without an AppArmor profile does not start on Ubuntu 23.10+.
…the popover Every frame of the marching dashes redraws the whole window: with a note open and idle that held 31 to 42% of a core, and a lit tray row 38 to 62% (measured 2026-10-27). The ring under an open note is now still, and the march elsewhere repaints 20 times a second at the same speed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A session browser's toolbar gets a crosshair that turns the pane into a picker: you point at elements of the page, leave a note on each, and send them to the session's agent as one message. Inspired by Orca's grab and Open Design's comment mode, built for a page drawn from a CDP screencast.
relay.Submit, so it reaches every provider. Each note gives the element's label and page, a selectorbrowser_clickandbrowser_typetake, the React component and source when the page is a development build, its markup without any field's value, and the note. Notes leave the tray only once the message is in; a refused send (a shell session, say) keeps them.Backend
browser.ElementAtreturns the element under a point: label, a selector built to find it again (id, or up to two classes that are not CSS-in-JS hashes,:nth-of-typeonly where a level is not unique yet), component and source from the React fiber, markup with everyvalueremoved, and its box. URL drops query and fragment.browser.Locatereturns where each selector's element stands now, an empty box for one gone.Notes for review
docs/ceilings.md: iframes are picked as the iframe, closed shadow roots as their host, a selector inside an open shadow root only resolves from that root, React 19 dropped_debugSource, and unsent notes live in the window.browser_notes_ring.gowalks a rounded rectangle's outline and strokes each dash on its own, since one path with many subpaths was bridged by the stroker.layouts.browser.*, the agent message underprompts.browserNotes.*.Test plan
gofmt,go vet,go test ./...).