feat(cli+web): inline image display for all agent flavors

[heavygee]

Summary

HAPI chat can render inline images via the generated-image message type (web fetches bytes from CLI via hub RPC). PR #700 added the display_image MCP tool and wired prompts for some flavors, but image display is not reliable for all agents today.

Parent: #697. Related: #508 (tool-result image blocks), #511 (composer thumbnails).

What PR #700 actually shipped (merged 2026-05-27)

Not Claude-only. Partial cross-flavor wiring:

• CLI MCP tool display_image in startHappyServer.ts — reads local file, registers in generatedImages, emits type: 'generated-image' agent message
• Codex: stdio MCP bridge forwards display_image; system prompt hint; plus native generated_image app-server events in codexRemoteLauncher.ts
• Claude: MCP via startHappyServer + system prompt hint
• OpenCode: system prompt hint + buildHapiMcpBridge MCP
• Web: flavor-agnostic — ToolMessage.tsx GeneratedImageCard, normalizeAgent.ts, hub route GET /sessions/:id/generated-images/:imageId

Gaps (why "all agents" still fails)

1. MCP wired but agents not told to use it

These flavors start buildHapiMcpBridge() (which uses startHappyServer — tool exists) but have no system prompt hint to call display_image:

• Cursor (ACP)
• Gemini (ACP)
• Kimi (ACP)
• Pi

buildHapiMcpBridge also only sets explicit tool approval for change_title; display_image should be auto-approved (yolo-safe read-only display).

2. ACP agent-native image blocks dropped

cli/src/agent/backends/acp/AcpMessageHandler.ts returns null for inner content { type: 'image', ... } — Cursor/Gemini/Kimi/OpenCode ACP responses that include image blocks never reach the hub/web.

3. Tool results still text-only (#508)

web/src/components/ToolCard/views/_results.tsx extracts text blocks only. Read/bash/chart outputs with embedded image content are invisible even when the agent did produce image data.

4. User attachment previews (#511)

MessageAttachments.tsx can render previewUrl thumbnails but many attachment paths still show filename chips only.

Proposed fix (v1 — cross-flavor parity)

CLI (all remote flavors using MCP bridge):

1. Add shared display_image system prompt snippet; inject into Cursor, Gemini, Kimi, Pi launchers (match Claude/Codex/OpenCode pattern)
2. Extend buildHapiMcpBridge tools config: display_image: { approval_mode: 'approve' } (or auto where flavor supports it)

ACP path:

3. Teach AcpMessageHandler to convert ACP image content blocks into generated-image session messages (register bytes/path via generatedImages helper)

Web (optional v1.1 or same PR if small):

4. Render base64/image blocks in tool results per [Enhancement] Render images inline in tool results and attachments with lightbox preview #508 (reuse ImagePreview / lightbox)

Acceptance criteria

• Cursor session: agent saves PNG locally, calls HAPI display_image (or ACP emits image block) → image visible inline in web chat
• Same for Gemini and Kimi (minimum); Pi if trivial
• Codex + Claude paths unchanged (regression)
• No flavor-specific web rendering forks — reuse existing generated-image pipeline
• Tests: MCP tool handler, ACP image block conversion, web normalize/render smoke

Non-goals (v1)

• Remote URL images without local file registration
• Markdown ![](path) auto-promotion from assistant text (follow-up)
• Composer attachment thumbnail polish (feat(web): show image thumbnails in composer attachment chips #511) unless zero-cost

References

• cli/src/claude/utils/startHappyServer.ts — display_image implementation
• cli/src/codex/utils/buildHapiMcpBridge.ts — shared MCP bridge (missing prompt + tool approval)
• cli/src/agent/backends/acp/AcpMessageHandler.ts:326 — image blocks dropped
• web/src/components/AssistantChat/messages/ToolMessage.tsx — existing inline renderer

[github-actions]

Verified. This is a valid enhancement request, and the existing generated-image pipeline is the right target for cross-flavor parity.

Current code supports inline generated images when an agent emits type: 'generated-image': the MCP display_image tool reads a local image, registers it, and sends that message type. The hub exposes a generated-image fetch route, and the web chat already normalizes/renders generated-image blocks.

Two notes from the current tree:

• Cursor/Gemini/Kimi ACP launchers do start the shared HAPI MCP bridge, but I did not find a display-image instruction injected into their prompts.
• OpenCode already has a hapi_display_image instruction. Pi does not appear to use buildHapiMcpBridge() in cli/src/pi, so its path needs separate verification.

The main missing pieces I found are:

• add display_image to the shared MCP tool config/approval path, since the bridge currently configures only change_title;
• add a shared display-image prompt snippet where ACP launchers can inject it consistently;
• teach ACP message normalization to preserve/convert image content instead of treating it as an unknown inner content type;
• optionally extend tool-result rendering beyond text extraction for image blocks.

Relevant code:

• cli/src/claude/utils/startHappyServer.ts:84 - registers the display_image MCP tool.
• cli/src/claude/utils/startHappyServer.ts:108 - registers image bytes in the generated image store.
• cli/src/claude/utils/startHappyServer.ts:116 - emits type: 'generated-image' to the session.
• cli/src/codex/utils/buildHapiMcpBridge.ts:74 - shared MCP bridge tool config currently only includes change_title approval.
• cli/src/cursor/cursorAcpRemoteLauncher.ts:57 - Cursor ACP starts the HAPI MCP bridge.
• cli/src/gemini/geminiRemoteLauncher.ts:49 - Gemini starts the HAPI MCP bridge.
• cli/src/kimi/kimiRemoteLauncher.ts:48 - Kimi starts the HAPI MCP bridge.
• cli/src/opencode/utils/systemPrompt.ts:15 - OpenCode already instructs use of hapi_display_image.
• cli/src/agent/backends/acp/AcpMessageHandler.ts:326 - ACP unknown inner content, including image/resource, returns null.
• web/src/chat/normalizeAgent.ts:558 - web normalizes generated-image agent messages.
• web/src/components/AssistantChat/messages/ToolMessage.tsx:52 - web generated-image card fetches and displays image blobs.
• hub/src/web/routes/git.ts:152 - hub route for generated-image bytes.
• web/src/components/ToolCard/views/_results.tsx:31 - tool result extraction currently handles text blocks only.
• web/src/components/AssistantChat/messages/MessageAttachments.tsx:53 - user attachment thumbnails render only when image attachments have previewUrl.

Need more info:

• For Pi support, which Pi agent path should expose/consume the HAPI MCP tool? I did not find a current buildHapiMcpBridge() hookup under cli/src/pi.

---

HAPI Bot

[heavygee]

Scope extension from peer-stack #959 proof handoff (fork worktree peer-stack-worktrees/0620-f7a4):

display_video MCP — display_video tool added alongside display_image in startHappyServer.ts (mp4/webm sniff via detectVideoMimeType). Same generated-image message + hub /generated-images/:id RPC path; web renders <video controls> when mimeType starts with video/.

Tooling: hapi-display-image.mjs auto-selects display_video for mp4/webm bytes.

Handoff pacing: peer-stack scratchlist handoff script now uses HAPI_PEER_HANDOFF_STEP_DELAY_MS (default 1200) + HAPI_PEER_HANDOFF_SLOW_MO_MS (default 400) so recorded proof is readable.

Tracks under #956 / PR #958 cross-flavor inline media work.

[heavygee]

Update — inline video scope (#956)

PR #958 now covers images + video on the same generated-media pipeline:

• MCP tools: display_image, display_video (mp4/webm)
• Web: GeneratedImageCard uses <video controls> when mimeType is video/*; bytes still fetched from GET /api/sessions/:id/generated-images/:imageId
• Shared prompts + HAPI_MCP_BRIDGE_PROMPT for Cursor/Gemini/Kimi/OpenCode (parity with images)
• Dogfood script: bun scripts/tooling/hapi-display-image.mjs <session-prefix> <absolute-path> [title] auto-picks display_video for mp4/webm

Not Cursor IDE composer — HAPI web chat only.

Live session must be restarted after CLI deploy so hapiMcpUrl MCP exposes display_video (tool-not-found on stale runner until soup/restart).

[Necmttn]

For cross-flavor image/video parity, I would model media display as an artifact event linked back to the producing tool call.

Fields: provider flavor, session id, tool_call_id, MCP tool name, artifact id, mime type, byte size, content hash, hub route, render status, and visibility outcome. That keeps display_image, display_video, ACP image blocks, and tool-result media comparable even though they enter through different paths.

Generated with ax - https://github.com/Necmttn/ax

[heavygee]

Agree with @Necmttn's artifact-event direction for cross-flavor inline media parity.

#956 / PR #958 v1 (shipping now): unified storage + fetch + web render on the existing generated-image wire type, plus optional source provenance on each inline media message:

• ingress: mcp | acp | tool_result
• optional flavor, toolCallId, toolName

MCP display_image / display_video and ACP image blocks now emit this metadata; Codex generated_image tool-result path emits tool_result. This is the cheap middle ground toward comparable ingress paths without a new message type yet.

Also in PR #958: fix ACP ordering regression (flush buffered text before async generated_image emit) per review bot Major finding.

Follow-up: opened a dedicated issue for the fuller artifact-event schema (session id, content hash, hub route, render status, visibility outcome, etc.) — @Necmttn please confirm or revise that spec there. v1 source is intentionally minimal so we can grow into artifact rows without breaking chat render.