feat(extension): report editable element ids alongside the a11y snapshot

[miguelg719]

Stack

Stack: #2951 snapshot editable ids → #2952 library → #2953 act pipeline → #2954 observe + cache check → #2955 extract. This is 1/5 (#2951). Each PR targets its predecessor.

Why

The a11y outline tells you an element's role, not whether you can type into it. A contenteditable div shows up as generic, and rich-text editors inside iframes look like plain containers. The decisions act path (next PRs) needs to know which nodes accept text to build its "input" candidate view; Chrome already reports this on each AX node as the editable property.

What

• a11yForFrame collects the encoded ids of nodes whose AX editable property is plaintext or richtext and returns them as editableIds.
• captureHybridSnapshot / tryScopedSnapshot merge them into combinedEditableIds (and per-frame editableIds).
• It is a side channel on the private snapshot type: the outline text, xpath map and url map are byte-for-byte unchanged, so nothing the LLM sees (or any cache key) moves.
• No consumer yet; the embedded Go extension zip is regenerated because extension source changed.

Testing

• New a11y-editable-ids.test.ts: editable nodes are reported by encoded id, ignored nodes are skipped, the outline is untouched.
• Extension unit tests, protocol tests, root parity tests, typecheck, oxfmt --check, extensionpack --check pass locally.

Naming: this stack was renamed from "Jev" to experimentalDecisions (TypeSafe Jev is the default decision-model provider). Further up the stack, #3102 adds Cloudflare Clef, Perplexity Decider and the OpenAI Decisions API as providers, and #3103 moves the feature behind stagehand.experimentalDecisions.act / observe / extract with the config as a public create option.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 8c62351

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[cubic-dev-ai]

1 issue found across 5 files

Confidence score: 4/5

• The feature test doesn’t verify capture wiring in packages/extension/understudy/a11y/snapshot/capture.ts, so regressions in frame aggregation or the scoped snapshot path could slip through. Add capture-level regression tests for both paths.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/extension/understudy/a11y/snapshot/capture.ts">

<violation number="1" location="packages/extension/understudy/a11y/snapshot/capture.ts:836">
P2: Add capture-level regression tests for aggregation across frames and for the scoped snapshot path. The current feature test leaves this wiring unverified, so a future propagation regression can ship while the editable-node test remains green.

(Based on your team's feedback about unit tests for new behavior.)</violation>
</file>

Architecture diagram

sequenceDiagram
    participant Capture as Snapshot Capture
    participant CDP as CDP Session
    participant AX as Chrome AX Tree
    participant A11y as a11yForFrame
    participant Maps as Per-Frame Maps
    participant Merge as Snapshot Merger
    participant Snapshot as HybridSnapshot

    Note over Capture,Snapshot: A11y snapshot carries editable element IDs as a side channel

    Capture->>CDP: Accessibility.getFullAXTree(frameId)
    CDP->>AX: Fetch full accessibility tree
    AX-->>CDP: AX nodes with role, name, properties
    CDP-->>Capture: AX nodes

    Capture->>A11y: Build frame snapshot
    A11y->>A11y: Filter ignored backend DOM nodes
    A11y->>A11y: Decorate roles and build outline + urlMap
    loop Each unignored node
        A11y->>A11y: Inspect AX "editable" property
        alt editable is plaintext or richtext
            A11y->>A11y: Encode backendDOMNodeId and append to editableIds
        end
    end
    A11y-->>Capture: outline, urlMap, editableIds

    alt Full hybrid snapshot
        Capture->>Maps: Store frame maps including editableIds
        Capture->>Merge: Merge per-frame maps
        Merge->>Snapshot: combinedEditableIds from flattened per-frame editableIds
    else Scoped single-frame snapshot
        Capture->>Snapshot: combinedEditableIds from this frame's editableIds
    end

    Note over Snapshot: Outline text, xpath map, and url map are unchanged<br/>LLM-visible content and cache keys are not affected

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: Add capture-level regression tests for aggregation across frames and for the scoped snapshot path. The current feature test leaves this wiring unverified, so a future propagation regression can ship while the editable-node test remains green.

(Based on your team's feedback about unit tests for new behavior.)

View Feedback

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/extension/understudy/a11y/snapshot/capture.ts, line 836:

<comment>Add capture-level regression tests for aggregation across frames and for the scoped snapshot path. The current feature test leaves this wiring unverified, so a future propagation regression can ship while the editable-node test remains green.

(Based on your team's feedback about unit tests for new behavior.) </comment>

<file context>
@@ -828,6 +833,7 @@ export function mergeFramesIntoSnapshot(
     combinedTree,
     combinedXpathMap,
     combinedUrlMap,
+    combinedEditableIds: [...perFrameMaps.values()].flatMap((maps) => maps.editableIds ?? []),
     perFrame: perFrameOutlines.map(({ frameId, outline }) => {
       const maps = perFrameMaps.get(frameId);
</file context>

[miguelg719]

Not changing this — the cross-frame aggregation is a one-line spread of per-frame results, and the a11yForFrame boundary (where the ids are computed) has its own test; capture-level tests need full CDP session mocks that this stack does not introduce. The editable ids are exercised end to end by the pipeline tests in #2953 (contenteditable targets).