Resolution Search Improvements

[ngoiyaeric]

Can you improve the prompting for the system such that the pointers are contextual to what the system can pin point through its vision capabilities, for example if the user specifies that they are looking for swimming pools then the pointer marks should point to them based on what the system's model vision has observed in context, enable the model to zoom into the image in the background if required for the resolution,
Example, locate solar panels in the satellite view, the system should look around, zoom, reason to identify these in the point of interest.

[ngoiyaeric]

@coderabbitai plan

[coderabbitai]

Coding Plan

Summary

• Fix the broken user query passthrough so the model knows what objects to find, then enhance the system prompt to prioritize intent-driven detection over generic analysis
• Extend the GeoJSON schema with category and confidence fields to enable contextual, classified pointer marks
• Implement server-side iterative zoom using sharp for image cropping, with a fixed 3-pass maximum and early exit via isComplete flag
• Use the same vision model for all passes, stream results incrementally for user feedback, and inject crop metadata into zoom-pass prompts for accurate coordinate mapping

Design Choices

Design Choice 1: How should the image cropping architecture work for zoom passes?

Options Considered:

1. Server-side with sharp — Use the sharp npm package in the server action to crop regions from already-compressed images. Single round-trip per zoom pass, no client coordination needed.
2. Client-side multi-turn — Send crop requests back to client, client crops from original full-resolution canvas capture, sends cropped region back. Preserves max resolution but requires complex client-server coordination and multiple round-trips.
3. Server-side with Node.js canvas — Use canvas npm package to mirror the existing browser-based compressImage() approach on the server.

Chosen Option: 1

Rationale: Option 1: Server-side cropping with sharp. This library is well-established, performant, and keeps the architecture simple with no client-side state management. While images are already compressed to 1024px before reaching the server, cropping still enables focused analysis on specific regions. The sharp dependency is lightweight and commonly used in Next.js projects.

Design Choice 2: What is the source of the user's search query text?

Options Considered:

1. Main chat/followup input — Use whatever text input triggered the resolution search (the message from followup-panel.tsx or any text accompanying the header search button action)
2. Dedicated resolution search input — Add a new, separate input field specifically for resolution search queries

Chosen Option: 1

Rationale: Option 1: Use existing input pathways. The followup-panel.tsx already appends an input field to FormData—this just needs to be read by the server action. For header-search-button.tsx, allow an optional query parameter. This avoids adding new UI components and leverages the natural conversational flow.

Design Choice 3: Should crop bounding box metadata be injected into zoom-pass prompts?

Options Considered:

1. Inject crop metadata — Include normalized bounding box coordinates (x, y, width, height as 0-1 values) in the zoom-pass system prompt, enabling the model to understand spatial context
2. No metadata injection — Let the model analyze crops in isolation, handle coordinate mapping entirely in post-processing

Chosen Option: 1

Rationale: Option 1: Inject crop metadata into zoom-pass prompts. The system prompt for zoom passes will include the crop region's normalized bounds relative to the original image. This allows the model to reason about spatial relationships and provide coordinates that can be accurately transformed back to the original image space.

Design Choice 4: How should UI feedback work during multi-pass analysis?

Options Considered:

1. Incremental streaming — Stream partial results after each pass, showing progress (e.g., "Analyzing region 1/3...") and intermediate findings
2. Silent completion — Accumulate all results silently, then stream the final merged response only after all passes complete

Chosen Option: 1

Rationale: Option 1: Incremental streaming feedback. The existing streamObject() pattern supports streaming partial results. Each zoom pass will stream its findings as they're generated, with UI updates indicating progress. This provides better UX for longer multi-pass analyses and follows the existing streaming patterns in app/actions.tsx.

Design Choice 5: Which model(s) should be used for initial vs. zoom passes?

Options Considered:

1. Same model for all passes — Use the vision model from getModel(true) for initial analysis and all zoom passes
2. Lighter auxiliary model for zoom passes — Use a faster/cheaper model for focused region analysis after the initial full-image pass

Chosen Option: 1

Rationale: Option 1: Same model for all passes. The codebase's getModel(hasImage) already handles model selection with a priority chain. Using the same model ensures consistent detection quality across passes. Zoom passes analyze smaller regions and should be relatively fast regardless. Adding model switching complexity is not justified for the marginal cost savings.

Design Choice 6: How should the iteration cap work?

Options Considered:

1. Fixed hardcoded cap — Maximum 3 iterations, not configurable
2. Environment variable configurable — Allow deployment-time configuration of max iterations
3. Dynamic logic-driven — Stop based solely on model's isComplete flag with no hard cap

Chosen Option: 1

Rationale: Option 1: Fixed cap of 3 iterations with early exit. The loop terminates when either: (a) the model sets isComplete: true, indicating analysis is complete, or (b) 3 total passes have been executed. A fixed cap is predictable, prevents runaway costs, and 3 passes (1 initial + 2 zoom) is sufficient for most use cases. Configuration can be added later if needed.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Contextual Prompt Pipeline

This phase establishes the foundation for intent-driven object detection by ensuring the user's query reaches the model, updating prompts to emphasize finding user-specified objects, and enhancing the schema to classify detected features.

Task 1: Pass User Query to the Vision Model

Currently, the user's search query is discarded and replaced with a hardcoded string. This task fixes that critical gap so the model knows what objects to look for.

• Update app/actions.tsx in the resolution_search handler to read the input field from FormData instead of using the hardcoded "Analyze this map view." string
• Ensure components/header-search-button.tsx appends the user's query text (if any) to the FormData alongside the image files
• Pass the user input through to the resolutionSearch() call so it appears in the message history sent to the model

Task 2: Revise System Prompt for Intent-Driven Detection

The current system prompt provides generic geospatial analysis instructions. This task refocuses it on identifying specific objects requested by the user.

• Modify the system prompt in lib/agents/resolution-search.tsx to explicitly instruct the model to prioritize finding objects mentioned in the user's query
• Add instructions that GeoJSON features should directly correspond to detected instances of the requested objects (e.g., each swimming pool gets its own Point feature)
• Include guidance on providing confidence levels and reasoning for identifications
• Retain existing contextual elements (timezone, coordinates, news) but restructure the prompt to lead with the user's search intent

Task 3: Enhance GeoJSON Schema with Feature Classification

The current schema's feature properties only include name and description. This task adds classification fields so features can be categorized by type.

• Extend the properties object in resolutionSearchSchema within lib/agents/resolution-search.tsx to include:
  • category: string field for the type of detected object (e.g., "swimming_pool", "solar_panel", "building")
  • confidence: optional number (0-1) indicating detection confidence
• Update the system prompt instructions to explain how these new fields should be populated
• This enables future rendering differentiation and filtering by feature type

🤖 Prompt for AI agents

This phase establishes the contextual prompt pipeline so the user's query drives
object detection. The goal is to pass the user's query to the vision model,
update the system prompt for intent-driven detection, and extend the GeoJSON
schema with classification fields.

**Task 1: Pass User Query to the Vision Model (`app/actions.tsx` and
`components/header-search-button.tsx`)**
- In `app/actions.tsx`, update the `resolution_search` handler to read the
`input` field from FormData instead of the hardcoded `"Analyze this map view."`
string
- In `components/header-search-button.tsx`, append the user's query text (if
any) to the FormData alongside the image files
- Ensure the user input is passed through to the `resolutionSearch()` call so it
appears in the message history sent to the model

**Task 2: Revise System Prompt for Intent-Driven Detection
(`lib/agents/resolution-search.tsx`)**
- Modify the system prompt to explicitly instruct the model to prioritize
finding objects mentioned in the user's query
- Add instructions that GeoJSON features should directly correspond to detected
instances of the requested objects (e.g., each swimming pool gets its own Point
feature)
- Include guidance on providing confidence levels and reasoning for
identifications
- Retain existing contextual elements (timezone, coordinates, news) but
restructure the prompt to lead with the user's search intent

**Task 3: Enhance GeoJSON Schema with Feature Classification
(`lib/agents/resolution-search.tsx`)**
- Extend the `properties` object in `resolutionSearchSchema` to add:
- `category`: string field for the type of detected object (e.g.,
`"swimming_pool"`, `"solar_panel"`, `"building"`)
  - `confidence`: optional number (0-1) indicating detection confidence
- Update the system prompt instructions to explain how these new fields should
be populated

Phase 2: Iterative Zoom Analysis

This phase enables the model to request focused analysis of specific image regions, implementing a multi-pass reasoning loop for identifying small or hard-to-see objects.

Task 1: Define Zoom Request Schema

Allow the model to indicate regions where closer inspection would help identification. This schema extension enables the iterative loop.

• Add a zoomRequests field to resolutionSearchSchema in lib/agents/resolution-search.tsx as an optional array of objects containing:
  • region: bounding box coordinates (normalized 0-1 values for x, y, width, height relative to image dimensions)
  • reason: string explaining why this region needs closer inspection
  • targetObject: what the model is trying to identify in this region
• Add an isComplete boolean field to indicate whether the model has finished analysis (enables early exit before hitting the 3-pass cap)
• Update system prompt to explain when and how to request zoom regions, emphasizing that zoom requests are optional and should only be made when additional detail would meaningfully improve detection

Task 2: Create Server-Side Image Region Extraction Utility

Implement a server-side utility using sharp to crop specific regions from images for focused re-analysis.

• Add sharp as a project dependency
• Create a new function in lib/utils/image-server-utils.ts (e.g., extractRegion()) that accepts a base64 image and normalized bounding box coordinates
• Use sharp to extract the specified region and resize it to maximize the 1024px dimension for best possible detail in the cropped view
• Return the cropped region as a base64 data URL ready for model consumption
• Keep this separate from the client-side image-utils.ts to maintain clear browser/server boundaries

Task 3: Implement Iterative Analysis Loop with Streaming Feedback

Transform the single-call resolution search into a multi-pass analysis that can process zoom requests with incremental UI updates.

• Refactor resolutionSearch() in lib/agents/resolution-search.tsx to support iteration:
  • Initial pass: analyze full image, stream results, check for zoom requests and isComplete flag
  • Subsequent passes: for each requested zoom region, extract the region using extractRegion(), include crop bounding box metadata in the prompt, and re-analyze
  • Stream findings from each pass as they complete for incremental UI feedback
  • Merge findings from all passes into a unified GeoJSON FeatureCollection
• Enforce fixed iteration limit of 3 total passes (1 initial + up to 2 zoom passes)
• Exit early if model sets isComplete: true
• Use the same model (via getModel(true)) for all passes to ensure consistent detection quality
• Track which regions have been analyzed to avoid redundant processing

Task 4: Coordinate Mapping for Zoomed Features

Ensure that features detected in zoomed regions are correctly positioned in the full image coordinate space, using the injected crop metadata.

• When the model returns coordinates from a zoomed region, transform them back to the original image's coordinate space using the crop bounding box metadata
• Update the zoom-pass system prompt to include the normalized crop bounds (x, y, width, height as 0-1 values) and instruct the model to provide coordinates relative to the cropped view
• Implement coordinate transformation logic in the iterative loop: absoluteCoord = cropOrigin + (relativeCoord * cropSize)
• Handle the lat/lng extraction appropriately when analyzing cropped regions by interpolating based on crop bounds and original viewport coordinates

🤖 Prompt for AI agents

This phase implements iterative zoom analysis, transforming the single-call
resolution search into a multi-pass loop where the model can request focused
analysis of specific image regions. The goal is to define the zoom request
schema, build a server-side image cropping utility using `sharp`, implement the
iteration loop with streaming feedback, and handle coordinate mapping for zoomed
features.

**Task 1: Define Zoom Request Schema (`lib/agents/resolution-search.tsx`)**
- Add a `zoomRequests` field to `resolutionSearchSchema` as an optional array of
objects, each containing:
- `region`: bounding box with normalized 0-1 values for x, y, width, height
relative to image dimensions
  - `reason`: string explaining why this region needs closer inspection
- `targetObject`: string describing what the model is trying to identify in this
region
- Add an `isComplete` boolean field to indicate whether the model has finished
analysis (enables early exit before hitting the 3-pass cap)
- Update the system prompt to explain when and how to request zoom regions,
emphasizing that zoom requests are optional and should only be made when
additional detail would meaningfully improve detection

**Task 2: Create Server-Side Image Region Extraction Utility
(`lib/utils/image-server-utils.ts`)**
- Add `sharp` as a project dependency
- Create a new file `lib/utils/image-server-utils.ts` with an exported
`extractRegion()` function that accepts a base64 image and normalized bounding
box coordinates
- Use `sharp` to extract the specified region and resize it to maximize the
1024px dimension for best possible detail in the cropped view
- Return the cropped region as a base64 data URL ready for model consumption
- Keep this file separate from the client-side `image-utils.ts` to maintain
clear browser/server boundaries

**Task 3: Implement Iterative Analysis Loop with Streaming Feedback
(`lib/agents/resolution-search.tsx`)**
- Refactor `resolutionSearch()` to support multi-pass iteration:
- **Initial pass**: analyze the full image, stream results, and check for zoom
requests and `isComplete` flag
- **Subsequent passes**: for each requested zoom region, extract the region
using `extractRegion()`, include crop bounding box metadata in the prompt, and
re-analyze
- **Stream** findings from each pass as they complete for incremental UI
feedback
  - **Merge** findings from all passes into a unified GeoJSON FeatureCollection
- Enforce a fixed iteration limit of 3 total passes (1 initial + up to 2 zoom
passes)
- Exit early if the model sets `isComplete: true`
- Use the same model via `getModel(true)` for all passes to ensure consistent
detection quality
- Track which regions have been analyzed to avoid redundant processing

**Task 4: Coordinate Mapping for Zoomed Features
(`lib/agents/resolution-search.tsx`)**
- When the model returns coordinates from a zoomed region, transform them back
to the original image's coordinate space using the crop bounding box metadata
- Update the zoom-pass system prompt to include the normalized crop bounds (x,
y, width, height as 0-1 values) and instruct the model to provide coordinates
relative to the cropped view
- Implement coordinate transformation logic in the iterative loop:
`absoluteCoord = cropOrigin + (relativeCoord * cropSize)`
- Handle lat/lng extraction appropriately when analyzing cropped regions by
interpolating based on crop bounds and original viewport coordinates

Research

The resolution search feature captures satellite imagery (Mapbox canvas screenshot and/or Google Static Maps), compresses it to 1024px max via compressImage(), and sends it to an AI vision model via the resolutionSearch agent in lib/agents/resolution-search.tsx. The agent uses streamObject() with a Zod schema to return a structured response including a GeoJSON FeatureCollection. These features are rendered as markers on the live Mapbox map via components/map/geojson-layer.tsx. Currently, the user's query text is not passed to the model (hardcoded as "Analyze this map view"), the system prompt provides only generic geospatial analysis instructions, the GeoJSON schema lacks feature classification fields, and there is no iterative zoom/reasoning capability—it's a single model call.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

This phase establishes the contextual prompt pipeline so the user's query drives
object detection. The goal is to pass the user's query to the vision model,
update the system prompt for intent-driven detection, and extend the GeoJSON
schema with classification fields.

**Task 1: Pass User Query to the Vision Model (`app/actions.tsx` and
`components/header-search-button.tsx`)**
- In `app/actions.tsx`, update the `resolution_search` handler to read the
`input` field from FormData instead of the hardcoded `"Analyze this map view."`
string
- In `components/header-search-button.tsx`, append the user's query text (if
any) to the FormData alongside the image files
- Ensure the user input is passed through to the `resolutionSearch()` call so it
appears in the message history sent to the model

**Task 2: Revise System Prompt for Intent-Driven Detection
(`lib/agents/resolution-search.tsx`)**
- Modify the system prompt to explicitly instruct the model to prioritize
finding objects mentioned in the user's query
- Add instructions that GeoJSON features should directly correspond to detected
instances of the requested objects (e.g., each swimming pool gets its own Point
feature)
- Include guidance on providing confidence levels and reasoning for
identifications
- Retain existing contextual elements (timezone, coordinates, news) but
restructure the prompt to lead with the user's search intent

**Task 3: Enhance GeoJSON Schema with Feature Classification
(`lib/agents/resolution-search.tsx`)**
- Extend the `properties` object in `resolutionSearchSchema` to add:
- `category`: string field for the type of detected object (e.g.,
`"swimming_pool"`, `"solar_panel"`, `"building"`)
  - `confidence`: optional number (0-1) indicating detection confidence
- Update the system prompt instructions to explain how these new fields should
be populated
===============================================================================

Task: 2

This phase implements iterative zoom analysis, transforming the single-call
resolution search into a multi-pass loop where the model can request focused
analysis of specific image regions. The goal is to define the zoom request
schema, build a server-side image cropping utility using `sharp`, implement the
iteration loop with streaming feedback, and handle coordinate mapping for zoomed
features.

**Task 1: Define Zoom Request Schema (`lib/agents/resolution-search.tsx`)**
- Add a `zoomRequests` field to `resolutionSearchSchema` as an optional array of
objects, each containing:
- `region`: bounding box with normalized 0-1 values for x, y, width, height
relative to image dimensions
  - `reason`: string explaining why this region needs closer inspection
- `targetObject`: string describing what the model is trying to identify in this
region
- Add an `isComplete` boolean field to indicate whether the model has finished
analysis (enables early exit before hitting the 3-pass cap)
- Update the system prompt to explain when and how to request zoom regions,
emphasizing that zoom requests are optional and should only be made when
additional detail would meaningfully improve detection

**Task 2: Create Server-Side Image Region Extraction Utility
(`lib/utils/image-server-utils.ts`)**
- Add `sharp` as a project dependency
- Create a new file `lib/utils/image-server-utils.ts` with an exported
`extractRegion()` function that accepts a base64 image and normalized bounding
box coordinates
- Use `sharp` to extract the specified region and resize it to maximize the
1024px dimension for best possible detail in the cropped view
- Return the cropped region as a base64 data URL ready for model consumption
- Keep this file separate from the client-side `image-utils.ts` to maintain
clear browser/server boundaries

**Task 3: Implement Iterative Analysis Loop with Streaming Feedback
(`lib/agents/resolution-search.tsx`)**
- Refactor `resolutionSearch()` to support multi-pass iteration:
- **Initial pass**: analyze the full image, stream results, and check for zoom
requests and `isComplete` flag
- **Subsequent passes**: for each requested zoom region, extract the region
using `extractRegion()`, include crop bounding box metadata in the prompt, and
re-analyze
- **Stream** findings from each pass as they complete for incremental UI
feedback
  - **Merge** findings from all passes into a unified GeoJSON FeatureCollection
- Enforce a fixed iteration limit of 3 total passes (1 initial + up to 2 zoom
passes)
- Exit early if the model sets `isComplete: true`
- Use the same model via `getModel(true)` for all passes to ensure consistent
detection quality
- Track which regions have been analyzed to avoid redundant processing

**Task 4: Coordinate Mapping for Zoomed Features
(`lib/agents/resolution-search.tsx`)**
- When the model returns coordinates from a zoomed region, transform them back
to the original image's coordinate space using the crop bounding box metadata
- Update the zoom-pass system prompt to include the normalized crop bounds (x,
y, width, height as 0-1 values) and instruct the model to provide coordinates
relative to the cropped view
- Implement coordinate transformation logic in the iterative loop:
`absoluteCoord = cropOrigin + (relativeCoord * cropSize)`
- Handle lat/lng extraction appropriately when analyzing cropped regions by
interpolating based on crop bounds and original viewport coordinates

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- @coderabbitai You can skip phase 3. Add a simple unit test case for phase 2.
- @coderabbitai For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[ngoiyaeric]

@coderabbitai I have reviewed the task and the codebase. I'm ready to start the deep planning mode. Before I proceed, I have a few clarifying questions to ensure I fully understand your expectations:

Iterative Zoom Implementation (Server vs. Client): The plan mentions "Server-side crop from compressed images." However, the existing compressImage utility uses the browser's Canvas API. Since resolutionSearch() runs on the server, I'll need to use a server-side library (like sharp) for cropping if I do it there. Alternatively, should the client perform the crops and send them back to the server in a multi-turn interaction? Which approach do you prefer?
User Query Source: In HeaderSearchButton, where should I retrieve the user's search query from? Is it the current value in the main chat input, or should I add a specific input field for the resolution search?
Coordinate Mapping: When the model detects an object in a zoomed/cropped region, it will provide coordinates relative to that crop. I'll need to map these back to the original image and then to Lat/Lng. To help the model, should I include the bounding box coordinates of the crop in the prompt for each zoom pass?
UI Feedback for Iterations: During the iterative analysis, should the UI provide feedback to the user (e.g., "Zooming into detected area...") or should it stay in the "Analyzing..." state until the entire process is complete and the final results are merged?
Model Selection: For the zoom passes, should I continue using the primary vision model (e.g., GPT-4o), or would you prefer a more lightweight "auxiliary" model (like GPT-4o-mini or Gemini Flash) for these focused analysis tasks to save on latency/cost?
Iteration Limit: The plan suggests a maximum of 3 passes. Is this a strict requirement, or would you like this to be dynamically determined based on the number of objects found or the model's confidence?
Once you've clarified these points, I'll be able to create a detailed execution plan.

[coderabbitai]

I've updated the plan above based on your feedback. Ready to proceed when you are!