feat: 参考https://github.com/stablyai/orca实现浏览器审查元素 可以标注发送给AI对话

[huiliaoning]

Prior search

• I searched existing issues and discussions, and this isn't already proposed.

What's the problem

feat: 参考https://github.com/stablyai/orca实现浏览器审查元素 可以把相关审查的元素标注发送给AI对话，很方便调试

What would solve it

feat: 参考https://github.com/stablyai/orca实现浏览器审查元素 可以把相关审查的元素标注发送给AI对话，很方便调试

Alternatives you considered

No response

[chyendongnhanh338]

Thanks for this feature request! I've analyzed how Orca implements the browser inspect element feature. Here's a detailed breakdown:

How Orca's Browser Inspect Feature Works

Core Workflow

1. Grab Mode - Users activate "inspect element mode" via a toolbar button
2. Element Selection - Hover over any DOM element in the browser to highlight it, then click to select
3. Data Extraction - The system captures comprehensive information about the selected element
4. Format & Send - Extracted data is formatted as structured Markdown and sent to the AI chat

Extracted Information (BrowserGrabPayload)

For each selected element, Orca captures:

DOM Structure:

• Tag name, CSS selector, full DOM path
• CSS classes, element attributes
• HTML snippet (max 4KB)
• Nearby elements and ancestor path

Computed Styles:
16 key CSS properties including: display, position, dimensions, spacing (margin/padding), colors, borders, fonts, z-index

Accessibility:

• ARIA role, accessible name, aria-label, aria-labelledby

Context:

• Surrounding text content (up to 10 snippets)
• React component name (if detectable)
• Source file path (if available via React DevTools)

Visual:

• Element screenshot (PNG, max 2MB)
• Viewport position and dimensions

Page Metadata:

• URL, title, viewport size, scroll position, device pixel ratio

Technical Implementation

• Injection Script: Uses Electron's executeJavaScript to inject a selector overlay into the page
• Isolated Context: Runs in a separate JavaScript world (ID 1207) to avoid conflicts with page scripts
• Shadow DOM Overlay: Creates a closed shadow root with high z-index to display numbered markers
• Viewport Tracking: Listens to scroll/resize events to keep markers positioned correctly
• IPC Communication: Coordinates between main process, preload script, and renderer
• Budget Limits: All text fields have length constraints to prevent extracting oversized data

Output Format

Extracted data is formatted as structured Markdown:

## Design Feedback: /path/to/page

**URL:** https://example.com/page
**Viewport:** 1920x1080

### 1. Button "Submit"
**Intent:** fix
**Selector:** `button.submit-btn`
**React:** SubmitButton
**Bounds:** x=100, y=200, 300x50
**Text:** "Submit form"
**Computed styles:**
- display: flex
- padding: 12px 24px
- background: #3b82f6
**HTML:**
```html
<button class="submit-btn">Submit form</button>

Feedback: The button color doesn't match the design system

### Annotations System

- **Multi-element support**: Mark up to 20 elements per page
- **Intent types**: fix, change, question, approve
- **Priority levels**: blocking, important, suggestion
- **Persistence**: Annotations are saved (excluding screenshots for size)

### Implementation Considerations for Paseo

**Benefits:**
- Extremely valuable for frontend debugging and UI review
- Natural integration with Paseo's agent chat system
- Rich context helps AI provide better suggestions

**Challenges:**
- **Platform differences**: Orca is Electron-only; Paseo supports iOS/Android/Web/Electron
- **Browser integration**: Need to evaluate if Paseo has embedded browser capabilities or if this requires new infrastructure
- **Mobile adaptation**: Touch interactions differ from mouse hover (e.g., long-press to select)
- **Implementation scope**: This is a complete subsystem with injection scripts, IPC, UI overlays, etc.

### Suggested Approach

If implemented, consider phased rollout:
1. **Phase 1**: Basic element selection and extraction on Electron desktop
2. **Phase 2**: Mobile touch interaction adaptation
3. **Phase 3**: Annotation persistence and batch sending

---

👍 This would be a powerful feature for frontend developers! @huiliaoning, would you be interested in implementing this? The Orca codebase provides a solid reference implementation that could be adapted for Paseo's architecture.