Feature Idea: Remove Image from Context (Compress Image to text)

[henningsway]

Problem

When using Databot with model providers that do not support vision capabilities (e.g., due to GitHub licensing restrictions or when using text-only models via Custom Providers), the session can run into errors or break when plots are generated. Since the created plot images are added to the conversation context, the text-only model cannot process them, which disrupts the workflow.

As noted in the documentation, Databot currently works best with Anthropic models, but users may not always have access to vision-capable models.

Proposed Solution

Provide an option to convert generated plots to a text-based representation before they are added to the conversation context. This could include:

• Alt-text / caption generation: Automatically generate a brief textual description of the plot (e.g., "Bar chart showing sales by region, with North America highest at 45%")
• User-supplied annotations: Allow the user to manually add a text comment describing the plot, which replaces the image in the context
• Graceful fallback: When a non-vision model is detected, automatically skip embedding the image and instead insert a placeholder or summary

Benefits

• Prevents session-breaking errors when vision is unavailable
• Maintains conversational continuity even without image support
• Enables Databot usage with a wider range of models (including text-only and local models)
• Reduces token consumption from large image embeddings

[wch]

Thanks for the idea. To automatically get a description of the image, that would require a vision-capable LLM, right? I'm not sure it's worth it to implement UI specifically for users to add an annotation -- they could just tell the LLM in regular chat what's in the image.

The general direction is for models, even local ones, to gain capabilities, and I'm certain we'll see vision capabilities in more local models as time goes on, so this problem will sort of solve itself. Because of this I don't think implementing features like these will rise to the top of our priority list.

[henningsway]

I agree, a bigger solution (including UI) seems too expensive and probably not worth it in the long run!

Maybe the "graceful fallback"-option or some other option is feasible, though?

A little more context: I often work with models provided through GitHub copilot. Because the Vision-capabilities require Preview mode to be enabled and in preview their licensing model changes quite drastically (giving GitHub more rights concerning the created code afaik).

So I often run into "session" breaking plots. I sometimes can switch to api based anthropic models or compress the conversation so far, but it would be nicer if non-vision-models could still be used even if plots are produced. Telling the agent what's in the plot in the chat totally seems feasible from my point of view.

[Zessi-C]

I'm still hitting this in July 2026 (Positron 2026.06.1), and there's an additional severity issue beyond just "model can't see the plot": once an image enters the context and the API errors, the conversation is permanently broken. Every subsequent message fails because the image stays in history. The only recovery is abandoning the chat and starting over.

I did some digging into the source code to understand why:

1. ModelDefinition (modelDefinitions.ts:9-15) has no supportsVision field — only name, identifier, maxInputTokens, maxOutputTokens, promptCaching. There's no way to declare a model is text-only.

2. Images enter context through getPlot (tools.ts:322), notebook tools (notebookUtils.ts:348), and user attachments (participants.ts:538).

3. In toAIMessage() (utils.ts:50-52), image parts are added to messages without checking if the model supports them. Same for handleGetPlotToolResultForChatCompletions() (utils.ts:690) and convertToolResultPartToAiToolMessage() (utils.ts:261).

4. No recovery mechanism — the image remains in conversation history after an API error, so the session can't self-heal.

I understand the "models will gain vision over time" argument, but text-only models are still very common (DeepSeek, GLM, many local models), and the "permanently broken conversation" aspect is a worse UX than just "can't see the plot."

A couple of ideas that wouldn't require a vision LLM:

• Graceful fallback: Detect text-only models and replace image parts with a text placeholder before sending. The transformRequestBody() function in openai-fetch-utils.ts already does similar compatibility transforms (max_tokens → max_completion_tokens, developer → system), so it would be a natural place to strip image content.

• supportsVision field on ModelDefinition: Let users declare per-model whether vision is supported, so the fallback only applies where needed.

• Auto-recovery on error: If an API error occurs, automatically strip image content from the context so the conversation can continue.

In the meantime, switching to a vision-capable model is a workable workaround — but it requires the user to remember to switch back and forth, and not everyone has access to a vision model.

[wch]

We have merged a fix in Posit Assistant for this issue and it will be available in the next release of Posit Assistant.