Skip to content

CLAUDE.md

Latest commit

 

History

History
160 lines (123 loc) · 5.59 KB

CLAUDE.md

File metadata and controls

160 lines (123 loc) · 5.59 KB
Raw

OpenAI Agents SDK Website Screenshot Analysis Tool

This document outlines the detailed step-by-step process for building an OpenAI Agents SDK agent that captures website screenshots and analyzes them using GPT-5 vision capabilities.

Project Overview

An OpenAI Agents SDK agent that:

  1. Calls a custom tool to open a URL in headless Chromium and capture a screenshot
  2. Sends that image to gpt-5 to visually analyze it
  3. Replies in exactly two sentences about what's on the page

Step 1: Prerequisites & Environment Setup

Required Components:

  • Python 3.10+
  • openai-agents package
  • Headless browser stack (Chromium via Playwright)
  • OpenAI API key (set OPENAI_API_KEY environment variable)

Installation Commands:

pip install openai-agents
pip install playwright
playwright install chromium

Step 2: Tool Design - Screenshot Function

Function Specification:

  • Name: capture_screenshot
  • Purpose: Launch Chromium headless, navigate to URL, capture PNG screenshot

Input Parameters:

  • url (string, required) - Target website URL
  • full_page (bool, default true) - Capture full page or viewport only
  • viewport_width (int, default 1280) - Browser viewport width
  • viewport_height (int, default 800) - Browser viewport height
  • wait_until (enum, default domcontentloaded) - Navigation wait condition
  • wait_for_selector (optional string) - CSS selector to wait for before capture
  • consent_click_selector (optional string) - Selector for cookie consent buttons

Return Value:

  • Image payload suitable for vision analysis (file path or base64)
  • Prefer file upload/URL over raw base64 for large images to keep prompts light

Step 3: Wiring Tool into Agents SDK

Integration Steps:

  1. Register Python function as Function Tool in Agents SDK
  2. SDK auto-derives JSON schema from function signature & docstring
  3. Add tool to Agent with model="gpt-5"
  4. Use Responses-model path for image input handling

Agent Configuration:

  • Model: gpt-5 (or gpt-5-mini for testing)
  • Tools: [capture_screenshot function]
  • Response format: Support image inputs

Step 4: Agent Instructions (System Prompt)

Core Instructions:

When the `capture_screenshot` tool is available and the user asks about a website, **always** call it first.

Analyze the returned image and reply with **exactly two sentences** describing:
- The page's purpose and brand
- Prominent elements (hero, headline, primary CTA)  
- Any obvious trust signals

If the screenshot fails, acknowledge the failure in exactly two sentences.

Model Parameters:

  • verbosity = low - Keep responses concise
  • reasoning_effort = minimal - Fast, direct answers
  • max_turns = 2-3 - Prevent infinite loops

Step 5: End-to-End Flow

Execution Sequence:

  1. User input: "Summarize https://example.com in two sentences"
  2. Agent planning: Identifies need for screenshot tool
  3. Tool execution: Headless Chromium captures PNG screenshot
  4. Model analysis: GPT-5 analyzes image with vision capabilities
  5. Response generation: Exactly two sentences describing the page
  6. Output: Final answer delivered to user

Step 6: Implementation Guardrails

Default Settings:

  • Viewport: 1280×800 pixels
  • Wait condition: networkidle for full page load
  • Image format: Compressed PNG
  • Quality: Balance between detail and cost

Error Handling:

  • Navigation timeouts: Return structured error message
  • Cookie walls: Optional consent button clicking
  • Failed screenshots: GPT-5 acknowledges failure in two sentences

Performance Optimization:

  • Image compression for cost control
  • Timeout limits on browser operations
  • Structured error responses

Step 7: Technology Stack Rationale

OpenAI Agents SDK Benefits:

  • First-class Function Tools integration
  • Simple Runner loop management
  • Built-in tracing capabilities
  • Seamless OpenAI models & tools integration

GPT-5 Advantages:

  • Strong tool-use reliability
  • Precise control over verbosity and reasoning effort
  • Excellent vision analysis capabilities
  • Consistent two-sentence output formatting

Step 8: Testing Checklist

Validation Points:

  • Agent consistently calls screenshot tool for URL requests
  • Screenshots capture stable page content (no blocking overlays)
  • Model output never exceeds two sentences
  • Summaries focus on purpose + key CTAs, not boilerplate UI text

Test Cases:

  • News website homepage
  • E-commerce product page
  • SaaS company landing page
  • Site with cookie consent banners

Step 9: Optional Extensions

Alternative Approaches:

  • Hosted Computer Tool: Use SDK's Computer Tool instead of custom Playwright
  • WebSearch Fallback: Add WebSearchTool for blocked/failed screenshots
  • Sessions: Implement persistent context for follow-up questions

Enhancement Options:

  • Multiple viewport sizes for responsive testing
  • Element-specific screenshot targeting
  • Performance metrics collection
  • Batch URL processing

Implementation Priority

  1. Phase 1: Basic screenshot tool with Playwright
  2. Phase 2: Agents SDK integration and GPT-5 analysis
  3. Phase 3: Error handling and guardrails
  4. Phase 4: Testing and optimization
  5. Phase 5: Optional extensions and enhancements

Key Resources