Skip to content

feat(browser): leave notes on page elements for the agent - #952

Merged
omartelo merged 3 commits into
mainfrom
feat/browser-page-notes
Oct 27, 2026
Merged

omartelo merged 3 commits into
mainfrom
feat/browser-page-notes

Conversation

@omartelo

Copy link
Copy Markdown
Collaborator

Summary

A session browser's toolbar gets a crosshair that turns the pane into a picker: you point at elements of the page, leave a note on each, and send them to the session's agent as one message. Inspired by Orca's grab and Open Design's comment mode, built for a page drawn from a CDP screencast.

  • Pick. With the crosshair on (or Ctrl+Shift+E), the pointer stops reaching the page. The element under it gets a marching dashed ring, offset from its own border, and a dark tag with its label, React component and size. Esc leaves the pick.
  • Note. A click opens the review's comment box beside the element, fading and zooming in from its side. Enter adds the note, Shift+Enter is a newline. A noted element keeps a dashed ring and a numbered tab at its top left; clicking the tab reopens the note with Save, Cancel and Delete.
  • Follow. Noted elements are looked up again every 400 ms, and every 30 ms for 0.7 s after the wheel turns, so the rings and an open note follow the page as it scrolls. A tab is hidden once its element leaves the viewport or the page navigates away.
  • Tray. Notes wait in a tray floating over the bottom of the page, inset by the island's gutter, so adding one never resizes the page (a docked tray relaid the page out and blinked on every note). Hovering a row lights its element, a click edits it, x removes it.
  • Send. Send (or Ctrl+Enter) types one message into the session through relay.Submit, so it reaches every provider. Each note gives the element's label and page, a selector browser_click and browser_type take, the React component and source when the page is a development build, its markup without any field's value, and the note. Notes leave the tray only once the message is in; a refused send (a shell session, say) keeps them.

Backend

  • browser.ElementAt returns the element under a point: label, a selector built to find it again (id, or up to two classes that are not CSS-in-JS hashes, :nth-of-type only where a level is not unique yet), component and source from the React fiber, markup with every value removed, and its box. URL drops query and fragment.
  • browser.Locate returns where each selector's element stands now, an empty box for one gone.
  • The page only reports what is where. The ring, tag, tabs, popover and tray are drawn by the window, so nothing is injected into the page and a page listener never sees the pick's clicks.

Notes for review

  • New ceiling in docs/ceilings.md: iframes are picked as the iframe, closed shadow roots as their host, a selector inside an open shadow root only resolves from that root, React 19 dropped _debugSource, and unsent notes live in the window.
  • Gio has no dashed stroke: browser_notes_ring.go walks a rounded rectangle's outline and strokes each dash on its own, since one path with many subpaths was bridged by the stroker.
  • Strings in all 7 locales: UI under layouts.browser.*, the agent message under prompts.browserNotes.*.

Test plan

  • Backend and native gates green (gofmt, go vet, go test ./...).
  • Live Chrome tests: the selector finds the picked element again, CSS-in-JS classes are skipped, field values never reach the markup, the page root answers nothing, a missing or invalid selector gives an empty box.
  • Unit tests: the message format, notes kept until sent, edits and drops, stale lookups ignored, an open note following its element, pins hidden off the page, the ring's outline walk, the queue merging lookups.
  • Driven in a rig window on a real site: pick, notes, scroll, tray, send to a Claude Code session.

A crosshair on the session browser's toolbar stops the pointer reaching
the page and outlines the element under it, named in a tag. A click
opens a note beside it, in the review's comment box; each note keeps a
numbered tab on a dashed ring around its element, following it as the
page scrolls. Notes wait in a tray floating over the page's bottom, so
the page never resizes, and Send types them into the session as one
message: label, page, a selector browser_click takes, the React
component in a development build, the markup without field values, and
the note. A send that fails keeps the notes.

The page only answers what is where (browser.ElementAt, browser.Locate);
the outline, tabs, popover and tray are drawn by the window, so nothing
is injected into the page's DOM.
- A production build's one or two letter component names (the minifier's)
  no longer reach the tag or the agent; a development build's source path is
  cut at the project's src/ even under a folder named app/.
- A refused send says why in the tray, not in a toast over the toolbar.
- Ctrl+Shift+E and Ctrl+Enter are Cmd on macOS.
- The wheel over the popover or the tray no longer scrolls the page under it.
- The ring's number tab sits on its straight top edge, past the corner.
- docs/ceilings.md: the pane copies the page's selection since #941, and a
  Chromium without an AppArmor profile does not start on Ubuntu 23.10+.
…the popover

Every frame of the marching dashes redraws the whole window: with a note
open and idle that held 31 to 42% of a core, and a lit tray row 38 to 62%
(measured 2026-10-27). The ring under an open note is now still, and the
march elsewhere repaints 20 times a second at the same speed.
@omartelo
omartelo merged commit 81e8b64 into main Oct 27, 2026
7 checks passed
@omartelo
omartelo deleted the feat/browser-page-notes branch October 27, 2026 20:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant