Consolidate multimodal ContentChunk support for all message roles by juliendenize · Pull Request #241 · mistralai/mistral-common

juliendenize · 2026-06-05T14:50:02Z

Summary

Consolidates PRs #236, #237, #238 into a single PR. Adds multimodal ContentChunk support for all message roles with version-gated validation and comprehensive test refactoring.

Changes

Content type system

Add per-role content type aliases: AssistantContentChunk, SystemContentChunk, ToolContentChunk
Widen all message content fields to use per-role aliases (matching internal repo)
Add shared _content_to_openai/_content_from_openai helpers on BaseMessage
Refactor all to_openai/from_openai to use shared helpers (no role parsing)
Use ToolContentChunk type alias in _parse_tool_content signature

Normalizer validation

Add _narrow_assistant_content with version-aware validation:
- Pre-v11 (v1-v7): rejects ThinkChunk in assistant messages
- v11+ (v13, v15): accepts TextChunk and ThinkChunk
Add _narrow_tool_content validation:
- Pre-v15: rejects non-text chunks in tool messages
- V15: accepts all content chunk types, sorts by tool call order
Add _narrow_system_content validation:
- Pre-v7: rejects non-text chunks (AudioChunk, ThinkChunk)
- V7+: accepts TextChunk, AudioChunk, ThinkChunk
- V15: rejects ThinkChunk in system messages
Skip JSON normalization of tool content for V7+ normalizers

Encoding

Widen encode_tool/system_message return types to tuple[list[int], list[np.ndarray], list[Audio]]
Narrow encode_system_message return type to exclude images

Chat templates

Add TemplateConfig properties: tool_supports_multimodal, system_supports_audio, system_supports_thinking
Update template generation with dynamic supported_types_desc
Refactor test_parity to use _get_conversations with to_openai conversion
Fix AssistantMessage.to_openai reasoning format to handle multimodal content

Version Gating

Feature	Pre-v7	V7	V11/V13	V15
Assistant ThinkChunk	Rejected	Rejected	Accepted	Accepted
System AudioChunk	Rejected	Accepted	Accepted	Accepted
System ThinkChunk	Rejected	Accepted	Accepted	Rejected
Tool non-text chunks	Rejected	Rejected	Rejected	Accepted
Tool JSON normalization	Yes	No	No	No

Test refactoring

Refactored normalizer tests into version-specific classes (TestChatCompletionRequestNormalizationV7/V13/V15), eliminating standalone test classes
Every normalizer test now asserts the full InstructRequest[ChatMessage, Tool] output (messages, system_prompt, settings, etc.) instead of individual message fields
Added match strings to all pytest.raises calls for precise error verification
Added intra-message and inter-message ThinkChunk aggregation tests for V13
Fixed double backticks to single backticks in docstrings (Google-style)
Used walrus operator for reasoning_effort extraction in test_parity

Closes #236, closes #237, closes #238

Add per-role content type aliases: AssistantContentChunk, SystemContentChunk, ToolContentChunk. Widen all message content fields to use per-role aliases. Add shared _content_to_openai/_content_from_openai helpers on BaseMessage. Refactor all to_openai/from_openai to use shared helpers. Add normalizer _narrow_*_content methods with version-aware validation: - _narrow_assistant_content: pre-V15 allows TextChunk/ThinkChunk, V15 allows all - _narrow_tool_content: pre-V15 allows text only, V15 allows all - _narrow_system_content: V7+ allows text/think/audio, V15 rejects ThinkChunk Widen encode_tool/assistant/system_message return types to tuple[list[int], list[np.ndarray], list[Audio]]. Add TemplateConfig properties: tool_supports_multimodal, assistant_supports_multimodal, system_supports_audio, system_supports_thinking. Update template generation with dynamic supported_types_desc. Closes #236, closes #237, closes #238

joa-stdn · 2026-06-11T22:01:40Z

+
+        Returns:
+            String content as-is, list of chunks serialized via each chunk's
+            to_openai(), or None.


…layers

… validator-concern tests

joa-stdn · 2026-06-15T19:37:43Z

+        tool_messages: list[ToolMessageType] = []
+        for message in messages:
+            assert isinstance(message, self._tool_message_class), "Expected tool message"
+            content = self._aggregate_content_chunks([message])


why can't we run super here?

Because the super one asserts that the content is a string whereas the one in v15 doesn't

Resolve conflicts from mistralai#241 (consolidated ContentChunk, removed UserContentChunk) and mistralai#245 (audio data URL handling): - chunk.py: apply Responses-API content-type aliases on top of the consolidated ContentChunk; drop the removed UserContentChunk alias. - tests/test_converters.py: keep both ChunkTypes and ContentChunk imports. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This was referenced Jun 5, 2026

Add ToolMessage multimodal ContentChunk support #236

Closed

Add AssistantMessage multimodal ContentChunk support #237

Closed

Add SystemMessage audio ContentChunk support #238

Closed

juliendenize force-pushed the feat/contentchunk-multimodal branch 8 times, most recently from 212e214 to b71292f Compare June 5, 2026 16:22